Class FileWriteAheadLogManager
- java.lang.Object
-
- org.apache.ignite.internal.processors.cache.GridCacheSharedManagerAdapter
-
- org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager
-
- All Implemented Interfaces:
IgniteWriteAheadLogManager,GridCacheSharedManager,IgniteChangeGlobalStateSupport
public class FileWriteAheadLogManager extends GridCacheSharedManagerAdapter implements IgniteWriteAheadLogManager
File WAL manager.
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description static classFileWriteAheadLogManager.ReadFileHandle
-
Field Summary
Fields Modifier and Type Field Description static StringCDC_DISABLEDCDC disabled distributed property name.static doubleDFLT_CHECKPOINT_TRIGGER_ARCHIVE_SIZE_PERCENTAGEstatic longDFLT_THRESHOLD_WAIT_TIME_NEXT_WAL_SEGMENTstatic intDFLT_WAL_COMPRESSOR_WORKER_THREAD_CNTstatic booleanDFLT_WAL_MMAPstatic PatternWAL_NAME_PATTERNPattern for segment file names.static FileFilterWAL_SEGMENT_COMPACTED_OR_RAW_FILE_FILTERWAL segment file filter, seeWAL_NAME_PATTERNstatic PatternWAL_SEGMENT_FILE_COMPACTED_PATTERNstatic FileFilterWAL_SEGMENT_FILE_FILTERWAL segment file filter, seeWAL_NAME_PATTERNstatic PatternWAL_TEMP_NAME_PATTERNPattern for WAL temp files - these files will be cleared at startup.-
Fields inherited from class org.apache.ignite.internal.processors.cache.GridCacheSharedManagerAdapter
cctx, diagnosticLog, log
-
-
Constructor Summary
Constructors Constructor Description FileWriteAheadLogManager(GridKernalContext ctx)Constructor.
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description @Nullable FilearchiveDir()Archive directory if any.static FilearchiveSegment(File walArchiveDir, long idx, String ext)voidawaitCompacted(long idx)Blocks current thread while segment with theidxnot compressed.FilecompactedSegment(long idx)longcurrentSegment()booleandisabled(int grpId, long pageId)Checks WAL disabled for cache group.WALPointerflush(WALPointer ptr, boolean explicitFsync)Makes sure that all log entries written to the log up until the specified pointer are actually written to the underlying storage.SegmentRoutergetSegmentRouter()Collection<File>getWalFilesFromArchive(WALPointer low, WALPointer high)Collects WAL segments from the archive only if they are all present.booleanisAlwaysWriteFullPages()booleanisFullSync()static booleanisSegmentFileName(@Nullable String name)Check that file name matches segment name.longlastArchivedSegment()longlastCompactedSegment()longlastTruncatedSegment()Getting last truncated segment.WALPointerlastWritePointer()Get last written pointer.static FileDescriptor[]loadFileDescriptors(File walFilesDir)Scans provided folder for a WAL segment filesWALPointerlog(WALRecord rec)Appends the given log entry to the write-ahead log.WALPointerlog(WALRecord rec, RolloverType rolloverType)Appends the given log entry to the write-ahead log.longmaxWalSegmentSize()voidnotchLastCheckpointPtr(WALPointer ptr)Notifiesthisabout latest checkpoint pointer.voidonActivate(GridKernalContext kctx)Called when cluster performing activation.voidonDeActivate(GridKernalContext kctx)Called when cluster performing deactivation.static ByteBufferprepareSerializerVersionBuffer(long idx, int ver, boolean compacted, ByteBuffer buf)Needs only for WAL compaction.WALRecordread(WALPointer ptr)Reads WAL record by the specified pointer.voidrelease(WALPointer start)Invoke this method to release WAL history since provided pointer that was previously reserved.WALIteratorreplay(WALPointer start)Invoke this method to iterate over the written log entries.WALIteratorreplay(WALPointer start, @Nullable IgniteBiPredicate<WALRecord.RecordType,WALPointer> recordDeserializeFilter)Invoke this method to iterate over the written log entries.booleanreserve(WALPointer start)Invoke this method to reserve WAL history since provided pointer and prevent it's deletion.booleanreserved(WALPointer ptr)Checks if WAL segment is under lock or reservedintreserved(WALPointer low, WALPointer high)Checks if WAL segments is under lock or reserved.voidresumeLogging(WALPointer filePtr)Resumes logging after start.static FileDescriptor[]scan(@Nullable File[] allFiles)static longsegmentIndex(Path segment)longsegmentSize(long idx)Getting local WAL segment size.intserializerVersion()voidsetCreateWalFileListener(@Nullable IgniteInClosure<FileIO> createWalFileListener)Setup listener for WAL segment write File IO creation.voidsetFileIOFactory(FileIOFactory ioFactory)For test purposes only.voidstart0()voidstartAutoReleaseSegments()Start automatically releasing segments when reachingDataStorageConfiguration.getMaxWalArchiveSize().protected voidstop0(boolean cancel)Method is called twice on deactivate and stop.static longtotalSize(FileDescriptor... fileDescriptors)Total size of the segments in bytes.inttruncate(@Nullable WALPointer high)Gives a hint to WAL manager to clear entries logged before the given pointer.FileDescriptor[]walArchiveFiles()Files fromwalArchiveDir.intwalArchiveSegments()@Nullable FilewalCdcDirectory()-
Methods inherited from class org.apache.ignite.internal.processors.cache.GridCacheSharedManagerAdapter
context, isStopping, kernalStartInfo, kernalStopInfo, onDisconnected, onKernalStart, onKernalStart0, onKernalStop, onKernalStop0, onReconnected, printMemoryStats, start, startInfo, stop, stopInfo, toString
-
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
-
Methods inherited from interface org.apache.ignite.internal.processors.cache.GridCacheSharedManager
onDisconnected, onKernalStart, onKernalStop, onReconnected, printMemoryStats, start, stop
-
-
-
-
Field Detail
-
WAL_NAME_PATTERN
public static final Pattern WAL_NAME_PATTERN
Pattern for segment file names.
-
WAL_TEMP_NAME_PATTERN
public static final Pattern WAL_TEMP_NAME_PATTERN
Pattern for WAL temp files - these files will be cleared at startup.
-
WAL_SEGMENT_FILE_FILTER
public static final FileFilter WAL_SEGMENT_FILE_FILTER
WAL segment file filter, seeWAL_NAME_PATTERN
-
WAL_SEGMENT_FILE_COMPACTED_PATTERN
public static final Pattern WAL_SEGMENT_FILE_COMPACTED_PATTERN
-
WAL_SEGMENT_COMPACTED_OR_RAW_FILE_FILTER
public static final FileFilter WAL_SEGMENT_COMPACTED_OR_RAW_FILE_FILTER
WAL segment file filter, seeWAL_NAME_PATTERN
-
DFLT_WAL_MMAP
public static final boolean DFLT_WAL_MMAP
-
DFLT_WAL_COMPRESSOR_WORKER_THREAD_CNT
public static final int DFLT_WAL_COMPRESSOR_WORKER_THREAD_CNT
-
DFLT_CHECKPOINT_TRIGGER_ARCHIVE_SIZE_PERCENTAGE
public static final double DFLT_CHECKPOINT_TRIGGER_ARCHIVE_SIZE_PERCENTAGE
-
DFLT_THRESHOLD_WAIT_TIME_NEXT_WAL_SEGMENT
public static final long DFLT_THRESHOLD_WAIT_TIME_NEXT_WAL_SEGMENT
-
CDC_DISABLED
public static final String CDC_DISABLED
CDC disabled distributed property name.- See Also:
- Constant Field Values
-
-
Constructor Detail
-
FileWriteAheadLogManager
public FileWriteAheadLogManager(GridKernalContext ctx)
Constructor.- Parameters:
ctx- Kernal context.
-
-
Method Detail
-
setFileIOFactory
public void setFileIOFactory(FileIOFactory ioFactory)
For test purposes only.- Parameters:
ioFactory- IO factory.
-
start0
public void start0() throws IgniteCheckedException- Overrides:
start0in classGridCacheSharedManagerAdapter- Throws:
IgniteCheckedException- If failed.
-
getSegmentRouter
public SegmentRouter getSegmentRouter()
- Returns:
- Info about of WAL paths.
-
archiveDir
@Nullable public @Nullable File archiveDir()
Archive directory if any.- Specified by:
archiveDirin interfaceIgniteWriteAheadLogManager- Returns:
- Archive directory.
-
getWalFilesFromArchive
public Collection<File> getWalFilesFromArchive(WALPointer low, WALPointer high) throws IgniteCheckedException
Collects WAL segments from the archive only if they are all present. Will wait for the last segment to be archived if it is not. If there are missing segments an empty collection is returned.- Parameters:
low- Low bound (include).high- High bound (not include).- Returns:
- WAL segments from the archive, or an empty collection if at
least a segment between
lowandhighis missing. - Throws:
IgniteCheckedException- If failed.
-
stop0
protected void stop0(boolean cancel)
Method is called twice on deactivate and stop. It shutdown workers but do not deallocate them to avoid duplication.- Overrides:
stop0in classGridCacheSharedManagerAdapter- Parameters:
cancel- Cancel flag.
-
onActivate
public void onActivate(GridKernalContext kctx) throws IgniteCheckedException
Called when cluster performing activation.- Specified by:
onActivatein interfaceIgniteChangeGlobalStateSupport- Parameters:
kctx- Kernal context.- Throws:
IgniteCheckedException- If failed.
-
onDeActivate
public void onDeActivate(GridKernalContext kctx)
Called when cluster performing deactivation.- Specified by:
onDeActivatein interfaceIgniteChangeGlobalStateSupport- Parameters:
kctx- Kernal context.
-
isAlwaysWriteFullPages
public boolean isAlwaysWriteFullPages()
- Specified by:
isAlwaysWriteFullPagesin interfaceIgniteWriteAheadLogManager- Returns:
trueIf we have to always write full pages.
-
isFullSync
public boolean isFullSync()
- Specified by:
isFullSyncin interfaceIgniteWriteAheadLogManager- Returns:
trueif WAL will perform fair syncs on fsync call.
-
resumeLogging
public void resumeLogging(WALPointer filePtr) throws IgniteCheckedException
Resumes logging after start. When WAL manager is started, it will skip logging any updates until this method is called to avoid logging changes induced by the state restore procedure.- Specified by:
resumeLoggingin interfaceIgniteWriteAheadLogManager- Throws:
IgniteCheckedException- If fails.
-
serializerVersion
public int serializerVersion()
- Specified by:
serializerVersionin interfaceIgniteWriteAheadLogManager- Returns:
- Current serializer version.
-
log
public WALPointer log(WALRecord rec) throws IgniteCheckedException
Appends the given log entry to the write-ahead log.- Specified by:
login interfaceIgniteWriteAheadLogManager- Parameters:
rec- Entry to log.- Returns:
- WALPointer that may be passed to
IgniteWriteAheadLogManager.flush(WALPointer, boolean)method to make sure the record is written to the log. - Throws:
IgniteCheckedException- If failed to construct log entry.StorageException- If IO error occurred while writing log entry.
-
log
public WALPointer log(WALRecord rec, RolloverType rolloverType) throws IgniteCheckedException
Appends the given log entry to the write-ahead log. If entry logging leads to rollover, caller can specify whether to write the entry to the current segment or to th next one.- Specified by:
login interfaceIgniteWriteAheadLogManager- Parameters:
rec- Entry to log.rolloverType- Rollover type.- Returns:
- WALPointer that may be passed to
IgniteWriteAheadLogManager.flush(WALPointer, boolean)method to make sure the record is written to the log. - Throws:
IgniteCheckedException- If failed to construct log entry.StorageException- If IO error occurred while writing log entry.- See Also:
RolloverType
-
flush
public WALPointer flush(WALPointer ptr, boolean explicitFsync) throws IgniteCheckedException, StorageException
Makes sure that all log entries written to the log up until the specified pointer are actually written to the underlying storage.- Specified by:
flushin interfaceIgniteWriteAheadLogManager- Parameters:
ptr- Optional pointer to write. Ifnull, will sync up to the latest record.explicitFsync- If true, data will be synced to the storage device on hardware level.- Returns:
- Last WAL position which was flushed to WAL segment file. May be greater than or equal to a
ptr. May benull, it means nothing has been flushed. - Throws:
IgniteCheckedException- If failed to write.StorageException- If IO exception occurred during the write. If an exception is thrown from this method, the WAL will be invalidated and the node will be stopped.
-
read
public WALRecord read(WALPointer ptr) throws IgniteCheckedException, StorageException
Reads WAL record by the specified pointer.- Specified by:
readin interfaceIgniteWriteAheadLogManager- Parameters:
ptr- WAL pointer.- Returns:
- WAL record.
- Throws:
IgniteCheckedException- If failed to read.StorageException- If IO error occurred while reading WAL entries.
-
replay
public WALIterator replay(WALPointer start) throws IgniteCheckedException, StorageException
Invoke this method to iterate over the written log entries.- Specified by:
replayin interfaceIgniteWriteAheadLogManager- Parameters:
start- Optional WAL pointer from which to start iteration.- Returns:
- Records iterator.
- Throws:
StorageException- If IO error occurred while reading WAL entries.IgniteCheckedException
-
replay
public WALIterator replay(WALPointer start, @Nullable @Nullable IgniteBiPredicate<WALRecord.RecordType,WALPointer> recordDeserializeFilter) throws IgniteCheckedException, StorageException
Invoke this method to iterate over the written log entries.- Specified by:
replayin interfaceIgniteWriteAheadLogManager- Parameters:
start- Optional WAL pointer from which to start iteration.recordDeserializeFilter- Specify a filter to skip WAL records. Those records will not be explicitly deserialized.- Returns:
- Records iterator.
- Throws:
StorageException- If IO error occurred while reading WAL entries.IgniteCheckedException
-
reserve
public boolean reserve(WALPointer start)
Invoke this method to reserve WAL history since provided pointer and prevent it's deletion. NOTE: If theDataStorageConfiguration.getMaxWalArchiveSize()is exceeded, the segment will be released.- Specified by:
reservein interfaceIgniteWriteAheadLogManager- Parameters:
start- WAL pointer.- Returns:
Trueif the reservation was successful.
-
release
public void release(WALPointer start)
Invoke this method to release WAL history since provided pointer that was previously reserved.- Specified by:
releasein interfaceIgniteWriteAheadLogManager- Parameters:
start- WAL pointer.
-
truncate
public int truncate(@Nullable @Nullable WALPointer high)Gives a hint to WAL manager to clear entries logged before the given pointer. If entries are needed for binary recovery, they will not be affected. Some entries may be reserved eg for historical rebalance and they also will not be affected.- Specified by:
truncatein interfaceIgniteWriteAheadLogManager- Parameters:
high- Upper border to which WAL segments will be deleted.- Returns:
- Number of deleted WAL segments.
-
notchLastCheckpointPtr
public void notchLastCheckpointPtr(WALPointer ptr)
Notifiesthisabout latest checkpoint pointer.Current implementations, in fact, react by keeping all WAL segments uncompacted starting from index prior to the index of
ptr. Compaction implies filtering out physical records and ZIP compression.- Specified by:
notchLastCheckpointPtrin interfaceIgniteWriteAheadLogManager- Parameters:
ptr- Pointer for which it is safe to compact the log.
-
currentSegment
public long currentSegment()
- Specified by:
currentSegmentin interfaceIgniteWriteAheadLogManager- Returns:
- Current segment index.
-
walArchiveSegments
public int walArchiveSegments()
- Specified by:
walArchiveSegmentsin interfaceIgniteWriteAheadLogManager- Returns:
- Total number of segments in the WAL archive.
-
lastArchivedSegment
public long lastArchivedSegment()
- Specified by:
lastArchivedSegmentin interfaceIgniteWriteAheadLogManager- Returns:
- Last archived segment index.
-
lastCompactedSegment
public long lastCompactedSegment()
- Specified by:
lastCompactedSegmentin interfaceIgniteWriteAheadLogManager- Returns:
- Last compacted segment index.
-
reserved
public boolean reserved(WALPointer ptr)
Checks if WAL segment is under lock or reserved- Specified by:
reservedin interfaceIgniteWriteAheadLogManager- Parameters:
ptr- Pointer to check.- Returns:
- True if given pointer is located in reserved segment.
-
reserved
public int reserved(WALPointer low, WALPointer high)
Checks if WAL segments is under lock or reserved.- Specified by:
reservedin interfaceIgniteWriteAheadLogManager- Parameters:
low- Pointer since which WAL is locked or reserved. Ifnull, checks from the oldest segment.high- Pointer for which WAL is locked or reserved.- Returns:
- Number of reserved WAL segments.
-
disabled
public boolean disabled(int grpId, long pageId)Checks WAL disabled for cache group.- Specified by:
disabledin interfaceIgniteWriteAheadLogManager- Parameters:
grpId- Group id.pageId- Page id.
-
segmentIndex
public static long segmentIndex(Path segment)
- Parameters:
segment- WAL segment file.- Returns:
- Segment index.
-
walArchiveFiles
public FileDescriptor[] walArchiveFiles()
Files fromwalArchiveDir.- Returns:
- Raw or compressed WAL segments from archive.
-
scan
public static FileDescriptor[] scan(@Nullable @Nullable File[] allFiles)
- Returns:
- Sorted WAL files descriptors.
-
setCreateWalFileListener
public void setCreateWalFileListener(@Nullable @Nullable IgniteInClosure<FileIO> createWalFileListener)Setup listener for WAL segment write File IO creation.- Parameters:
createWalFileListener- Listener to be invoked for new segment file IO creation.
-
maxWalSegmentSize
public long maxWalSegmentSize()
- Returns:
maxWalSegmentSize.
-
compactedSegment
public File compactedSegment(long idx)
- Specified by:
compactedSegmentin interfaceIgniteWriteAheadLogManager- Parameters:
idx- Segment index.- Returns:
- Compressed archive segment.
-
awaitCompacted
public void awaitCompacted(long idx) throws IgniteInterruptedCheckedExceptionBlocks current thread while segment with theidxnot compressed. If segment compressed, already, returns immediately.- Specified by:
awaitCompactedin interfaceIgniteWriteAheadLogManager- Parameters:
idx- Segment index.- Throws:
IgniteInterruptedCheckedException
-
archiveSegment
public static File archiveSegment(File walArchiveDir, long idx, String ext)
- Parameters:
walArchiveDir- WAL archive directory.idx- Segment index.ext- Optional extension- Returns:
- Path to archive segment.
-
prepareSerializerVersionBuffer
public static ByteBuffer prepareSerializerVersionBuffer(long idx, int ver, boolean compacted, ByteBuffer buf)
Needs only for WAL compaction.- Parameters:
idx- Index.ver- Version.compacted- Compacted flag.
-
loadFileDescriptors
public static FileDescriptor[] loadFileDescriptors(File walFilesDir) throws IgniteCheckedException
Scans provided folder for a WAL segment files- Parameters:
walFilesDir- directory to scan- Returns:
- found WAL file descriptors
- Throws:
IgniteCheckedException
-
segmentSize
public long segmentSize(long idx)
Getting local WAL segment size.- Specified by:
segmentSizein interfaceIgniteWriteAheadLogManager- Parameters:
idx- Absolute segment index.- Returns:
- Segment size,
0if size is unknown.
-
lastWritePointer
public WALPointer lastWritePointer()
Get last written pointer.- Specified by:
lastWritePointerin interfaceIgniteWriteAheadLogManager- Returns:
- Last written pointer.
-
isSegmentFileName
public static boolean isSegmentFileName(@Nullable @Nullable String name)Check that file name matches segment name.- Parameters:
name- File name.- Returns:
Trueif file name matches segment name.
-
lastTruncatedSegment
public long lastTruncatedSegment()
Getting last truncated segment.- Returns:
- Absolut segment index.
-
totalSize
public static long totalSize(FileDescriptor... fileDescriptors)
Total size of the segments in bytes.- Returns:
- Size in bytes.
-
walCdcDirectory
@Nullable public @Nullable File walCdcDirectory()
- Returns:
- WAL cdc directory (including consistent ID as subfolder)
-
startAutoReleaseSegments
public void startAutoReleaseSegments()
Start automatically releasing segments when reachingDataStorageConfiguration.getMaxWalArchiveSize().- Specified by:
startAutoReleaseSegmentsin interfaceIgniteWriteAheadLogManager
-
-