org.apache.ignite.internal.processors.cache.persistence.pagemem

Class PageMemoryImpl

java.lang.Object

org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl

All Implemented Interfaces:

PageIdAllocator, PageMemory, PageSupport, PageMemoryEx

public class PageMemoryImpl
extends Object
implements PageMemoryEx

Page header structure is described by the following diagram. When page is not allocated (in a free list):

 +--------+------------------------------------------------------+
 |8 bytes |         PAGE_SIZE + PAGE_OVERHEAD - 8 bytes          |
 +--------+------------------------------------------------------+
 |Next ptr|                      Page data                       |
 +--------+------------------------------------------------------+
 

When page is allocated and is in use:

 +------------------+--------+--------+----+----+--------+--------+----------------------+
 |     8 bytes      |8 bytes |8 bytes |4 b |4 b |8 bytes |8 bytes |       PAGE_SIZE      |
 +------------------+--------+--------+----+----+--------+--------+----------------------+
 | Marker/Timestamp |Rel ptr |Page ID |C ID|PIN | LOCK   |TMP BUF |       Page data      |
 +------------------+--------+--------+----+----+--------+--------+----------------------+
 

Note that first 8 bytes of page header are used either for page marker or for next relative pointer depending on whether the page is in use or not.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	PageMemoryImpl.ThrottlingPolicy Throttling enabled and its type enum.

Field Summary

Fields

Modifier and Type	Field and Description
static String	CHECKPOINT_POOL_OVERFLOW_ERROR_MSG Checkpoint pool overflow error message.
static int	PAGE_CACHE_ID_OFFSET Page cache group ID offset.
static int	PAGE_ID_OFFSET Page ID offset
static int	PAGE_LOCK_OFFSET Page lock offset.
static long	PAGE_MARKER
static int	PAGE_OVERHEAD 8b Marker/timestamp 8b Relative pointer 8b Page ID 4b Cache group ID 4b Pin count 8b Lock 8b Temporary buffer
static int	PAGE_PIN_CNT_OFFSET Page pin counter offset.
static int	PAGE_TMP_BUF_OFFSET Page temp copy buffer relative pointer offset.
static int	RANDOM_PAGES_EVICT_NUM Number of random pages that will be picked for eviction.
static int	RELATIVE_PTR_OFFSET Page relative pointer.
static int	TRY_AGAIN_TAG Try again tag.

Fields inherited from interface org.apache.ignite.internal.pagemem.PageIdAllocator

FLAG_DATA, FLAG_IDX, INDEX_PARTITION, MAX_PARTITION_ID

Constructor Summary

Constructors

Constructor and Description
PageMemoryImpl(DirectMemoryProvider directMemoryProvider, long[] sizes, GridCacheSharedContext<?,?> ctx, int pageSize, ReplacedPageWriter flushDirtyPage, GridInClosure3X<Long,FullPageId,PageMemoryEx> changeTracker, CheckpointLockStateChecker stateChecker, DataRegionMetricsImpl memMetrics, PageMemoryImpl.ThrottlingPolicy throttlingPlc, CheckpointWriteProgressSupplier cpProgressProvider)

Method Summary

Modifier and Type	Method and Description
long	acquiredPages()
long	acquirePage(int grpId, long pageId) Gets the page absolute pointer associated with the given page ID.
long	acquirePage(int grpId, long pageId, boolean restore)
int	activePagesCount() Gets the number of active pages across all segments.
long	allocatePage(int grpId, int partId, byte flags) Allocates a page from the space for the given partition ID and the given flags.
GridMultiCollectionWrapper<FullPageId>	beginCheckpoint() Gets a collection of dirty page IDs since the last checkpoint.
int	checkpointBufferPagesCount() Number of pages used in checkpoint buffer.
int	checkpointBufferPagesSize() Number of used pages in checkpoint buffer.
IgniteInternalFuture<Void>	clearAsync(LoadedPagesMap.KeyPredicate pred, boolean cleanDirty) Asynchronously clears pages satisfying the given predicate.
void	finishCheckpoint() Finishes checkpoint operation.
boolean	freePage(int grpId, long pageId) The given page is free now.
Integer	getForCheckpoint(FullPageId fullId, ByteBuffer outBuf, CheckpointMetricsTracker tracker) Gets page byte buffer for the checkpoint procedure.
boolean	hasTempCopy(long absPtr) Checks if a page has temp copy buffer.
int	invalidate(int grpId, int partId) Marks partition as invalid / outdated.
boolean	isDirty(int grpId, long pageId, long page)
long	loadedPages()
long	metaPageId(int grpId) Gets or allocates metadata page for specified grpId.
void	onCacheGroupDestroyed(int grpId) Clears internal metadata of destroyed cache group.
ByteBuffer	pageBuffer(long pageAddr)
int	pageSize()
long	partitionMetaPageId(int grpId, int partId) Gets or allocates partition metadata page for specified grpId and partId.
long	readLock(int grpId, long pageId, long page)
long	readLockForce(int grpId, long pageId, long page) Obtains read lock without checking page tag.
void	readUnlock(int grpId, long pageId, long page) Releases locked page.
int	realPageSize(int grpId)
void	releasePage(int grpId, long pageId, long page)
boolean	safeToUpdate() Heuristic method which allows a thread to check if it safe to start memory struture modifications in regard with checkpointing.
static int	segmentIndex(int grpId, long pageId, int segments)
void	start() Start page memory.
void	stop(boolean deallocate) Stop page memory.
int	systemPageSize()
long	totalPages()
long	tryWriteLock(int grpId, long pageId, long page)
long	writeLock(int grpId, long pageId, long page)
long	writeLock(int grpId, long pageId, long page, boolean restore)
void	writeUnlock(int grpId, long pageId, long page, Boolean walPlc, boolean dirtyFlag) Releases locked page.
void	writeUnlock(int grpId, long pageId, long page, Boolean walPlc, boolean dirtyFlag, boolean restore)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PAGE_MARKER

public static final long PAGE_MARKER

See Also:

Constant Field Values

RELATIVE_PTR_OFFSET

public static final int RELATIVE_PTR_OFFSET

Page relative pointer. Does not change once a page is allocated.

See Also:

Constant Field Values

PAGE_ID_OFFSET

public static final int PAGE_ID_OFFSET

Page ID offset

See Also:

Constant Field Values

PAGE_CACHE_ID_OFFSET

public static final int PAGE_CACHE_ID_OFFSET

Page cache group ID offset.

See Also:

Constant Field Values

PAGE_PIN_CNT_OFFSET

public static final int PAGE_PIN_CNT_OFFSET

Page pin counter offset.

See Also:

Constant Field Values

PAGE_LOCK_OFFSET

public static final int PAGE_LOCK_OFFSET

Page lock offset.

See Also:

Constant Field Values

PAGE_TMP_BUF_OFFSET

public static final int PAGE_TMP_BUF_OFFSET

Page temp copy buffer relative pointer offset.

See Also:

Constant Field Values

PAGE_OVERHEAD

public static final int PAGE_OVERHEAD

8b Marker/timestamp 8b Relative pointer 8b Page ID 4b Cache group ID 4b Pin count 8b Lock 8b Temporary buffer

See Also:

Constant Field Values

RANDOM_PAGES_EVICT_NUM

public static final int RANDOM_PAGES_EVICT_NUM

Number of random pages that will be picked for eviction.

See Also:

Constant Field Values

TRY_AGAIN_TAG

public static final int TRY_AGAIN_TAG

Try again tag.

See Also:

Constant Field Values

CHECKPOINT_POOL_OVERFLOW_ERROR_MSG

public static final String CHECKPOINT_POOL_OVERFLOW_ERROR_MSG

Checkpoint pool overflow error message.

See Also:

Constant Field Values

Constructor Detail

PageMemoryImpl

public PageMemoryImpl(DirectMemoryProvider directMemoryProvider,
                      long[] sizes,
                      GridCacheSharedContext<?,?> ctx,
                      int pageSize,
                      ReplacedPageWriter flushDirtyPage,
                      @Nullable
                      GridInClosure3X<Long,FullPageId,PageMemoryEx> changeTracker,
                      CheckpointLockStateChecker stateChecker,
                      DataRegionMetricsImpl memMetrics,
                      @Nullable
                      PageMemoryImpl.ThrottlingPolicy throttlingPlc,
                      @NotNull
                      CheckpointWriteProgressSupplier cpProgressProvider)

Parameters:

directMemoryProvider - Memory allocator to use.

sizes - segments sizes, last is checkpoint pool size.

ctx - Cache shared context.

pageSize - Page size.

flushDirtyPage - write callback invoked when a dirty page is removed for replacement.

changeTracker - Callback invoked to track changes in pages.

stateChecker - Checkpoint lock state provider. Used to ensure lock is held by thread, which modify pages.

memMetrics - Memory metrics to track dirty pages count and page replace rate.

throttlingPlc - Write throttle enabled and its type. Null equal to none.

cpProgressProvider - checkpoint progress, base for throttling. Null disables throttling.

Method Detail

start

public void start()
           throws IgniteException

Start page memory.

Specified by:

start in interface PageMemory

Throws:

IgniteException

stop

public void stop(boolean deallocate)
          throws IgniteException

Stop page memory.

Specified by:

stop in interface PageMemory

Parameters:

deallocate - True to deallocate memory, false to allow memory reuse on subsequent PageMemory.start()

Throws:

IgniteException

releasePage

public void releasePage(int grpId,
                        long pageId,
                        long page)

Specified by:

releasePage in interface PageSupport

Parameters:

grpId - Cache group ID.

pageId - Page ID to release.

page - Page pointer.

readLock

public long readLock(int grpId,
                     long pageId,
                     long page)

Specified by:

readLock in interface PageSupport

Parameters:

grpId - Cache group ID.

pageId - Page ID.

page - Page pointer.

Returns:

Pointer for reading the page.

readUnlock

public void readUnlock(int grpId,
                       long pageId,
                       long page)

Releases locked page.

Specified by:

readUnlock in interface PageSupport

Parameters:

grpId - Cache group ID.

pageId - Page ID.

page - Page pointer.

writeLock

public long writeLock(int grpId,
                      long pageId,
                      long page)

Specified by:

writeLock in interface PageSupport

Parameters:

grpId - Cache group ID.

pageId - Page ID.

page - Page pointer.

Returns:

Address of a buffer with contents of the given page or 0L if attempt to take the write lock failed.

writeLock

public long writeLock(int grpId,
                      long pageId,
                      long page,
                      boolean restore)

Specified by:

writeLock in interface PageMemoryEx

Parameters:

grpId - Group ID.

pageId - Page ID.

page - Page pointer.

restore - Determines if the page is locked for restore.

Returns:

ByteBuffer for modifying the page.

tryWriteLock

public long tryWriteLock(int grpId,
                         long pageId,
                         long page)

Specified by:

tryWriteLock in interface PageSupport

Parameters:

grpId - Cache group ID.

pageId - Page ID.

page - Page pointer.

Returns:

Address of a buffer with contents of the given page or 0L if attempt to take the write lock failed.

writeUnlock

public void writeUnlock(int grpId,
                        long pageId,
                        long page,
                        Boolean walPlc,
                        boolean dirtyFlag)

Releases locked page.

Specified by:

writeUnlock in interface PageSupport

Parameters:

grpId - Cache group ID.

pageId - Page ID.

page - Page pointer.

walPlc - True if page should be recorded to WAL, false if the page must not be recorded and null for the default behavior.

dirtyFlag - Determines whether the page was modified since the last checkpoint.

writeUnlock

public void writeUnlock(int grpId,
                        long pageId,
                        long page,
                        Boolean walPlc,
                        boolean dirtyFlag,
                        boolean restore)

Specified by:

writeUnlock in interface PageMemoryEx

Parameters:

grpId - Group ID.

pageId - Page ID.

page - Page pointer.

walPlc - True if page should be recorded to WAL, false if the page must not be recorded and null for the default behavior.

dirtyFlag - Determines whether the page was modified since the last checkpoint.

restore - Determines if the page is locked for restore.

isDirty

public boolean isDirty(int grpId,
                       long pageId,
                       long page)

Specified by:

isDirty in interface PageSupport

Parameters:

grpId - Cache group ID.

pageId - Page ID.

page - Page pointer.

Returns:

True if the page is dirty.

allocatePage

public long allocatePage(int grpId,
                         int partId,
                         byte flags)
                  throws IgniteCheckedException

Allocates a page from the space for the given partition ID and the given flags.

Specified by:

allocatePage in interface PageIdAllocator

Parameters:

grpId - Cache Group ID.

partId - Partition ID.

Returns:

Allocated page ID.

Throws:

IgniteCheckedException

pageBuffer

public ByteBuffer pageBuffer(long pageAddr)

Specified by:

pageBuffer in interface PageMemory

Parameters:

pageAddr - Page address.

Returns:

Page byte buffer.

freePage

public boolean freePage(int grpId,
                        long pageId)
                 throws IgniteCheckedException

The given page is free now.

Specified by:

freePage in interface PageIdAllocator

Parameters:

grpId - Cache Group ID.

pageId - Page ID.

Throws:

IgniteCheckedException

metaPageId

public long metaPageId(int grpId)
                throws IgniteCheckedException

Gets or allocates metadata page for specified grpId.

Specified by:

metaPageId in interface PageMemoryEx

Parameters:

grpId - Group ID.

Returns:

Meta page for grpId.

Throws:

IgniteCheckedException - If failed.

partitionMetaPageId

public long partitionMetaPageId(int grpId,
                                int partId)
                         throws IgniteCheckedException

Gets or allocates partition metadata page for specified grpId and partId.

Specified by:

partitionMetaPageId in interface PageMemoryEx

Parameters:

grpId - Group ID.

partId - Partition ID.

Returns:

Meta page for grpId and partId.

Throws:

IgniteCheckedException - If failed.

acquirePage

public long acquirePage(int grpId,
                        long pageId)
                 throws IgniteCheckedException

Gets the page absolute pointer associated with the given page ID. Each page obtained with this method must be released by calling PageSupport.releasePage(int, long, long). This method will allocate page with given ID if it doesn't exist.

Specified by:

acquirePage in interface PageSupport

Parameters:

grpId - Cache group ID.

pageId - Page ID.

Returns:

Page pointer.

Throws:

IgniteCheckedException - If failed.

acquirePage

public long acquirePage(int grpId,
                        long pageId,
                        boolean restore)
                 throws IgniteCheckedException

Specified by:

acquirePage in interface PageMemoryEx

Parameters:

grpId - Cache group ID.

pageId - Page id.

restore - Get page for restore

Returns:

Page.

Throws:

IgniteCheckedException - If failed.

StorageException - If page reading failed from storage.

See Also:

Will not read page from file if it is not present in memory

pageSize

public int pageSize()

Specified by:

pageSize in interface PageMemory

Returns:

Page size in bytes.

systemPageSize

public int systemPageSize()

Specified by:

systemPageSize in interface PageMemory

Returns:

Page size with system overhead, in bytes.

realPageSize

public int realPageSize(int grpId)

Specified by:

realPageSize in interface PageMemory

Parameters:

grpId - Group id.

Returns:

Page size without encryption overhead.

safeToUpdate

public boolean safeToUpdate()

Heuristic method which allows a thread to check if it safe to start memory struture modifications in regard with checkpointing.

Specified by:

safeToUpdate in interface PageMemoryEx

Returns:

False if there are too many dirty pages and a thread should wait for a checkpoint to begin.

totalPages

public long totalPages()

Returns:

Total pages can be placed in all segments.

beginCheckpoint

public GridMultiCollectionWrapper<FullPageId> beginCheckpoint()
                                                       throws IgniteException

Gets a collection of dirty page IDs since the last checkpoint. If a dirty page is being written after the checkpointing operation begun, the modifications will be written to a temporary buffer which will be flushed to the main memory after the checkpointing finished. This method must be called when no concurrent operations on pages are performed.

Specified by:

beginCheckpoint in interface PageMemoryEx

Returns:

Collection of dirty page IDs.

Throws:

IgniteException - If checkpoint has been already started and was not finished.

finishCheckpoint

public void finishCheckpoint()

Finishes checkpoint operation.

Specified by:

finishCheckpoint in interface PageMemoryEx

getForCheckpoint

public Integer getForCheckpoint(FullPageId fullId,
                                ByteBuffer outBuf,
                                CheckpointMetricsTracker tracker)

Gets page byte buffer for the checkpoint procedure.

Specified by:

getForCheckpoint in interface PageMemoryEx

Parameters:

fullId - Page ID to get byte buffer for. The page ID must be present in the collection returned by the PageMemoryEx.beginCheckpoint() method call.

outBuf - Temporary buffer to write changes into.

tracker - Checkpoint metrics tracker.

Returns:

Partition generation if data was read, null otherwise (data already saved to storage).

invalidate

public int invalidate(int grpId,
                      int partId)

Marks partition as invalid / outdated.

Specified by:

invalidate in interface PageMemoryEx

Parameters:

grpId - Group ID.

partId - Partition ID.

Returns:

New partition generation (growing 1-based partition file version).

onCacheGroupDestroyed

public void onCacheGroupDestroyed(int grpId)

Clears internal metadata of destroyed cache group.

Specified by:

onCacheGroupDestroyed in interface PageMemoryEx

Parameters:

grpId - Cache group ID.

clearAsync

public IgniteInternalFuture<Void> clearAsync(LoadedPagesMap.KeyPredicate pred,
                                             boolean cleanDirty)

Asynchronously clears pages satisfying the given predicate.

Specified by:

clearAsync in interface PageMemoryEx

Parameters:

pred - Predicate for cache group id, pageId.

cleanDirty - Flag indicating that dirty pages collection should be cleaned.

Returns:

Future that will be completed when all pages are cleared.

loadedPages

public long loadedPages()

Specified by:

loadedPages in interface PageMemory

Returns:

Total number of loaded pages in memory.

acquiredPages

public long acquiredPages()

Returns:

Total number of acquired pages.

readLockForce

public long readLockForce(int grpId,
                          long pageId,
                          long page)

Obtains read lock without checking page tag.

Specified by:

readLockForce in interface PageSupport

Parameters:

grpId - Cache group ID.

pageId - Page ID.

page - Page pointer.

Returns:

Pointer for reading the page.

hasTempCopy

public boolean hasTempCopy(long absPtr)

Checks if a page has temp copy buffer.

Parameters:

absPtr - Absolute pointer.

Returns:

True if a page has temp buffer.

activePagesCount

public int activePagesCount()

Gets the number of active pages across all segments. Used for test purposes only.

Returns:

Number of active pages.

checkpointBufferPagesCount

public int checkpointBufferPagesCount()

Number of pages used in checkpoint buffer.

Specified by:

checkpointBufferPagesCount in interface PageMemory

checkpointBufferPagesSize

public int checkpointBufferPagesSize()

Number of used pages in checkpoint buffer.

segmentIndex

public static int segmentIndex(int grpId,
                               long pageId,
                               int segments)

Parameters:

pageId - Page ID.

Returns:

Segment index.