Package org.apache.ignite.internal.processors.cache.persistence.wal

Class FileWriteAheadLogManager

java.lang.Object

org.apache.ignite.internal.processors.cache.GridCacheSharedManagerAdapter

org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager

All Implemented Interfaces:

IgniteWriteAheadLogManager, GridCacheSharedManager, IgniteChangeGlobalStateSupport

public class FileWriteAheadLogManager
extends GridCacheSharedManagerAdapter
implements IgniteWriteAheadLogManager

File WAL manager.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	FileWriteAheadLogManager.ReadFileHandle

Field Summary

Fields

Modifier and Type	Field	Description
static String	CDC_DISABLED	CDC disabled distributed property name.
static double	DFLT_CHECKPOINT_TRIGGER_ARCHIVE_SIZE_PERCENTAGE
static long	DFLT_THRESHOLD_WAIT_TIME_NEXT_WAL_SEGMENT
static int	DFLT_WAL_COMPRESSOR_WORKER_THREAD_CNT
static boolean	DFLT_WAL_MMAP
static Pattern	WAL_NAME_PATTERN	Pattern for segment file names.
static FileFilter	WAL_SEGMENT_COMPACTED_OR_RAW_FILE_FILTER	WAL segment file filter, see WAL_NAME_PATTERN
static Pattern	WAL_SEGMENT_FILE_COMPACTED_PATTERN
static FileFilter	WAL_SEGMENT_FILE_FILTER	WAL segment file filter, see WAL_NAME_PATTERN
static Pattern	WAL_TEMP_NAME_PATTERN	Pattern for WAL temp files - these files will be cleared at startup.

Fields inherited from class org.apache.ignite.internal.processors.cache.GridCacheSharedManagerAdapter

cctx, diagnosticLog, log

Constructor Summary

Constructors

Constructor	Description
FileWriteAheadLogManager(GridKernalContext ctx)	Constructor.

Method Summary

Modifier and Type	Method	Description
@Nullable File	archiveDir()	Archive directory if any.
static File	archiveSegment(File walArchiveDir, long idx, String ext)
void	awaitCompacted(long idx)	Blocks current thread while segment with the idx not compressed.
File	compactedSegment(long idx)
long	currentSegment()
boolean	disabled(int grpId, long pageId)	Checks WAL disabled for cache group.
WALPointer	flush(WALPointer ptr, boolean explicitFsync)	Makes sure that all log entries written to the log up until the specified pointer are actually written to the underlying storage.
SegmentRouter	getSegmentRouter()
Collection<File>	getWalFilesFromArchive(WALPointer low, WALPointer high)	Collects WAL segments from the archive only if they are all present.
boolean	isAlwaysWriteFullPages()
boolean	isFullSync()
static boolean	isSegmentFileName(@Nullable String name)	Check that file name matches segment name.
long	lastArchivedSegment()
long	lastCompactedSegment()
long	lastTruncatedSegment()	Getting last truncated segment.
WALPointer	lastWritePointer()	Get last written pointer.
static FileDescriptor[]	loadFileDescriptors(File walFilesDir)	Scans provided folder for a WAL segment files
WALPointer	log(WALRecord rec)	Appends the given log entry to the write-ahead log.
WALPointer	log(WALRecord rec, RolloverType rolloverType)	Appends the given log entry to the write-ahead log.
long	maxWalSegmentSize()
void	notchLastCheckpointPtr(WALPointer ptr)	Notifies this about latest checkpoint pointer.
void	onActivate(GridKernalContext kctx)	Called when cluster performing activation.
void	onDeActivate(GridKernalContext kctx)	Called when cluster performing deactivation.
static ByteBuffer	prepareSerializerVersionBuffer(long idx, int ver, boolean compacted, ByteBuffer buf)	Needs only for WAL compaction.
WALRecord	read(WALPointer ptr)	Reads WAL record by the specified pointer.
void	release(WALPointer start)	Invoke this method to release WAL history since provided pointer that was previously reserved.
WALIterator	replay(WALPointer start)	Invoke this method to iterate over the written log entries.
WALIterator	replay(WALPointer start, @Nullable IgniteBiPredicate<WALRecord.RecordType,WALPointer> recordDeserializeFilter)	Invoke this method to iterate over the written log entries.
boolean	reserve(WALPointer start)	Invoke this method to reserve WAL history since provided pointer and prevent it's deletion.
boolean	reserved(WALPointer ptr)	Checks if WAL segment is under lock or reserved
int	reserved(WALPointer low, WALPointer high)	Checks if WAL segments is under lock or reserved.
void	resumeLogging(WALPointer filePtr)	Resumes logging after start.
static FileDescriptor[]	scan(@Nullable File[] allFiles)
static long	segmentIndex(Path segment)
long	segmentSize(long idx)	Getting local WAL segment size.
int	serializerVersion()
void	setCreateWalFileListener(@Nullable IgniteInClosure<FileIO> createWalFileListener)	Setup listener for WAL segment write File IO creation.
void	setFileIOFactory(FileIOFactory ioFactory)	For test purposes only.
void	start0()
void	startAutoReleaseSegments()	Start automatically releasing segments when reaching DataStorageConfiguration.getMaxWalArchiveSize().
protected void	stop0(boolean cancel)	Method is called twice on deactivate and stop.
static long	totalSize(FileDescriptor... fileDescriptors)	Total size of the segments in bytes.
int	truncate(@Nullable WALPointer high)	Gives a hint to WAL manager to clear entries logged before the given pointer.
FileDescriptor[]	walArchiveFiles()	Files from walArchiveDir.
int	walArchiveSegments()
@Nullable File	walCdcDirectory()

Methods inherited from class org.apache.ignite.internal.processors.cache.GridCacheSharedManagerAdapter

context, isStopping, kernalStartInfo, kernalStopInfo, onDisconnected, onKernalStart, onKernalStart0, onKernalStop, onKernalStop0, onReconnected, printMemoryStats, start, startInfo, stop, stopInfo, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.ignite.internal.processors.cache.GridCacheSharedManager

onDisconnected, onKernalStart, onKernalStop, onReconnected, printMemoryStats, start, stop

Field Detail

WAL_NAME_PATTERN

public static final Pattern WAL_NAME_PATTERN

Pattern for segment file names.

WAL_TEMP_NAME_PATTERN

public static final Pattern WAL_TEMP_NAME_PATTERN

Pattern for WAL temp files - these files will be cleared at startup.

WAL_SEGMENT_FILE_FILTER

public static final FileFilter WAL_SEGMENT_FILE_FILTER

WAL segment file filter, see WAL_NAME_PATTERN

WAL_SEGMENT_FILE_COMPACTED_PATTERN

public static final Pattern WAL_SEGMENT_FILE_COMPACTED_PATTERN

WAL_SEGMENT_COMPACTED_OR_RAW_FILE_FILTER

public static final FileFilter WAL_SEGMENT_COMPACTED_OR_RAW_FILE_FILTER

WAL segment file filter, see WAL_NAME_PATTERN

DFLT_WAL_MMAP

public static final boolean DFLT_WAL_MMAP

See Also:

IgniteSystemProperties.IGNITE_WAL_MMAP, Constant Field Values

DFLT_WAL_COMPRESSOR_WORKER_THREAD_CNT

public static final int DFLT_WAL_COMPRESSOR_WORKER_THREAD_CNT

See Also:

IgniteSystemProperties.IGNITE_WAL_COMPRESSOR_WORKER_THREAD_CNT, Constant Field Values

DFLT_CHECKPOINT_TRIGGER_ARCHIVE_SIZE_PERCENTAGE

public static final double DFLT_CHECKPOINT_TRIGGER_ARCHIVE_SIZE_PERCENTAGE

See Also:

IgniteSystemProperties.IGNITE_CHECKPOINT_TRIGGER_ARCHIVE_SIZE_PERCENTAGE, Constant Field Values

DFLT_THRESHOLD_WAIT_TIME_NEXT_WAL_SEGMENT

public static final long DFLT_THRESHOLD_WAIT_TIME_NEXT_WAL_SEGMENT

See Also:

IgniteSystemProperties.IGNITE_THRESHOLD_WAIT_TIME_NEXT_WAL_SEGMENT, Constant Field Values

CDC_DISABLED

public static final String CDC_DISABLED

CDC disabled distributed property name.

See Also:

Constant Field Values

Constructor Detail

FileWriteAheadLogManager

public FileWriteAheadLogManager(GridKernalContext ctx)

Constructor.

Parameters:

ctx - Kernal context.

Method Detail

setFileIOFactory

public void setFileIOFactory(FileIOFactory ioFactory)

For test purposes only.

Parameters:

ioFactory - IO factory.

start0

public void start0()
            throws IgniteCheckedException

Overrides:

start0 in class GridCacheSharedManagerAdapter

Throws:

IgniteCheckedException - If failed.

getSegmentRouter

public SegmentRouter getSegmentRouter()

Returns:

Info about of WAL paths.

archiveDir

@Nullable
public @Nullable File archiveDir()

Archive directory if any.

Specified by:

archiveDir in interface IgniteWriteAheadLogManager

Returns:

Archive directory.

getWalFilesFromArchive

public Collection<File> getWalFilesFromArchive(WALPointer low,
                                               WALPointer high)
                                        throws IgniteCheckedException

Collects WAL segments from the archive only if they are all present. Will wait for the last segment to be archived if it is not. If there are missing segments an empty collection is returned.

Parameters:

low - Low bound (include).

high - High bound (not include).

Returns:

WAL segments from the archive, or an empty collection if at least a segment between low and high is missing.

Throws:

IgniteCheckedException - If failed.

stop0

protected void stop0(boolean cancel)

Method is called twice on deactivate and stop. It shutdown workers but do not deallocate them to avoid duplication.

Overrides:

stop0 in class GridCacheSharedManagerAdapter

Parameters:

cancel - Cancel flag.

onActivate

public void onActivate(GridKernalContext kctx)
                throws IgniteCheckedException

Called when cluster performing activation.

Specified by:

onActivate in interface IgniteChangeGlobalStateSupport

Parameters:

kctx - Kernal context.

Throws:

IgniteCheckedException - If failed.

onDeActivate

public void onDeActivate(GridKernalContext kctx)

Called when cluster performing deactivation.

Specified by:

onDeActivate in interface IgniteChangeGlobalStateSupport

Parameters:

kctx - Kernal context.

isAlwaysWriteFullPages

public boolean isAlwaysWriteFullPages()

Specified by:

isAlwaysWriteFullPages in interface IgniteWriteAheadLogManager

Returns:

true If we have to always write full pages.

isFullSync

public boolean isFullSync()

Specified by:

isFullSync in interface IgniteWriteAheadLogManager

Returns:

true if WAL will perform fair syncs on fsync call.

resumeLogging

public void resumeLogging(WALPointer filePtr)
                   throws IgniteCheckedException

Resumes logging after start. When WAL manager is started, it will skip logging any updates until this method is called to avoid logging changes induced by the state restore procedure.

Specified by:

resumeLogging in interface IgniteWriteAheadLogManager

Throws:

IgniteCheckedException - If fails.

serializerVersion

public int serializerVersion()

Specified by:

serializerVersion in interface IgniteWriteAheadLogManager

Returns:

Current serializer version.

log

public WALPointer log(WALRecord rec)
               throws IgniteCheckedException

Appends the given log entry to the write-ahead log.

Specified by:

log in interface IgniteWriteAheadLogManager

Parameters:

rec - Entry to log.

Returns:

WALPointer that may be passed to IgniteWriteAheadLogManager.flush(WALPointer, boolean) method to make sure the record is written to the log.

Throws:

IgniteCheckedException - If failed to construct log entry.

StorageException - If IO error occurred while writing log entry.

log

public WALPointer log(WALRecord rec,
                      RolloverType rolloverType)
               throws IgniteCheckedException

Appends the given log entry to the write-ahead log. If entry logging leads to rollover, caller can specify whether to write the entry to the current segment or to th next one.

Specified by:

log in interface IgniteWriteAheadLogManager

Parameters:

rec - Entry to log.

rolloverType - Rollover type.

Returns:

WALPointer that may be passed to IgniteWriteAheadLogManager.flush(WALPointer, boolean) method to make sure the record is written to the log.

Throws:

IgniteCheckedException - If failed to construct log entry.

StorageException - If IO error occurred while writing log entry.

See Also:

RolloverType

flush

public WALPointer flush(WALPointer ptr,
                        boolean explicitFsync)
                 throws IgniteCheckedException,
                        StorageException

Makes sure that all log entries written to the log up until the specified pointer are actually written to the underlying storage.

Specified by:

flush in interface IgniteWriteAheadLogManager

Parameters:

ptr - Optional pointer to write. If null, will sync up to the latest record.

explicitFsync - If true, data will be synced to the storage device on hardware level.

Returns:

Last WAL position which was flushed to WAL segment file. May be greater than or equal to a ptr. May be null, it means nothing has been flushed.

Throws:

IgniteCheckedException - If failed to write.

StorageException - If IO exception occurred during the write. If an exception is thrown from this method, the WAL will be invalidated and the node will be stopped.

read

public WALRecord read(WALPointer ptr)
               throws IgniteCheckedException,
                      StorageException

Reads WAL record by the specified pointer.

Specified by:

read in interface IgniteWriteAheadLogManager

Parameters:

ptr - WAL pointer.

Returns:

WAL record.

Throws:

IgniteCheckedException - If failed to read.

StorageException - If IO error occurred while reading WAL entries.

replay

public WALIterator replay(WALPointer start)
                   throws IgniteCheckedException,
                          StorageException

Invoke this method to iterate over the written log entries.

Specified by:

replay in interface IgniteWriteAheadLogManager

Parameters:

start - Optional WAL pointer from which to start iteration.

Returns:

Records iterator.

Throws:

StorageException - If IO error occurred while reading WAL entries.

IgniteCheckedException

replay

public WALIterator replay(WALPointer start,
                          @Nullable
                          @Nullable IgniteBiPredicate<WALRecord.RecordType,WALPointer> recordDeserializeFilter)
                   throws IgniteCheckedException,
                          StorageException

Invoke this method to iterate over the written log entries.

Specified by:

replay in interface IgniteWriteAheadLogManager

Parameters:

start - Optional WAL pointer from which to start iteration.

recordDeserializeFilter - Specify a filter to skip WAL records. Those records will not be explicitly deserialized.

Returns:

Records iterator.

Throws:

StorageException - If IO error occurred while reading WAL entries.

IgniteCheckedException

reserve

public boolean reserve(WALPointer start)

Invoke this method to reserve WAL history since provided pointer and prevent it's deletion. NOTE: If the DataStorageConfiguration.getMaxWalArchiveSize() is exceeded, the segment will be released.

Specified by:

reserve in interface IgniteWriteAheadLogManager

Parameters:

start - WAL pointer.

Returns:

True if the reservation was successful.

release

public void release(WALPointer start)

Invoke this method to release WAL history since provided pointer that was previously reserved.

Specified by:

release in interface IgniteWriteAheadLogManager

Parameters:

start - WAL pointer.

truncate

public int truncate(@Nullable
                    @Nullable WALPointer high)

Gives a hint to WAL manager to clear entries logged before the given pointer. If entries are needed for binary recovery, they will not be affected. Some entries may be reserved eg for historical rebalance and they also will not be affected.

Specified by:

truncate in interface IgniteWriteAheadLogManager

Parameters:

high - Upper border to which WAL segments will be deleted.

Returns:

Number of deleted WAL segments.

notchLastCheckpointPtr

public void notchLastCheckpointPtr(WALPointer ptr)

Notifies this about latest checkpoint pointer.

Current implementations, in fact, react by keeping all WAL segments uncompacted starting from index prior to the index of ptr. Compaction implies filtering out physical records and ZIP compression.

Specified by:

notchLastCheckpointPtr in interface IgniteWriteAheadLogManager

Parameters:

ptr - Pointer for which it is safe to compact the log.

currentSegment

public long currentSegment()

Specified by:

currentSegment in interface IgniteWriteAheadLogManager

Returns:

Current segment index.

walArchiveSegments

public int walArchiveSegments()

Specified by:

walArchiveSegments in interface IgniteWriteAheadLogManager

Returns:

Total number of segments in the WAL archive.

lastArchivedSegment

public long lastArchivedSegment()

Specified by:

lastArchivedSegment in interface IgniteWriteAheadLogManager

Returns:

Last archived segment index.

lastCompactedSegment

public long lastCompactedSegment()

Specified by:

lastCompactedSegment in interface IgniteWriteAheadLogManager

Returns:

Last compacted segment index.

reserved

public boolean reserved(WALPointer ptr)

Checks if WAL segment is under lock or reserved

Specified by:

reserved in interface IgniteWriteAheadLogManager

Parameters:

ptr - Pointer to check.

Returns:

True if given pointer is located in reserved segment.

reserved

public int reserved(WALPointer low,
                    WALPointer high)

Checks if WAL segments is under lock or reserved.

Specified by:

reserved in interface IgniteWriteAheadLogManager

Parameters:

low - Pointer since which WAL is locked or reserved. If null, checks from the oldest segment.

high - Pointer for which WAL is locked or reserved.

Returns:

Number of reserved WAL segments.

disabled

public boolean disabled(int grpId,
                        long pageId)

Checks WAL disabled for cache group.

Specified by:

disabled in interface IgniteWriteAheadLogManager

Parameters:

grpId - Group id.

pageId - Page id.

segmentIndex

public static long segmentIndex(Path segment)

Parameters:

segment - WAL segment file.

Returns:

Segment index.

walArchiveFiles

public FileDescriptor[] walArchiveFiles()

Files from walArchiveDir.

Returns:

Raw or compressed WAL segments from archive.

scan

public static FileDescriptor[] scan(@Nullable
                                    @Nullable File[] allFiles)

Returns:

Sorted WAL files descriptors.

setCreateWalFileListener

public void setCreateWalFileListener(@Nullable
                                     @Nullable IgniteInClosure<FileIO> createWalFileListener)

Setup listener for WAL segment write File IO creation.

Parameters:

createWalFileListener - Listener to be invoked for new segment file IO creation.

maxWalSegmentSize

public long maxWalSegmentSize()

Returns:

maxWalSegmentSize.

compactedSegment

public File compactedSegment(long idx)

Specified by:

compactedSegment in interface IgniteWriteAheadLogManager

Parameters:

idx - Segment index.

Returns:

Compressed archive segment.

awaitCompacted

public void awaitCompacted(long idx)
                    throws IgniteInterruptedCheckedException

Blocks current thread while segment with the idx not compressed. If segment compressed, already, returns immediately.

Specified by:

awaitCompacted in interface IgniteWriteAheadLogManager

Parameters:

idx - Segment index.

Throws:

IgniteInterruptedCheckedException

archiveSegment

public static File archiveSegment(File walArchiveDir,
                                  long idx,
                                  String ext)

Parameters:

walArchiveDir - WAL archive directory.

idx - Segment index.

ext - Optional extension

Returns:

Path to archive segment.

prepareSerializerVersionBuffer

public static ByteBuffer prepareSerializerVersionBuffer(long idx,
                                                        int ver,
                                                        boolean compacted,
                                                        ByteBuffer buf)

Needs only for WAL compaction.

Parameters:

idx - Index.

ver - Version.

compacted - Compacted flag.

loadFileDescriptors

public static FileDescriptor[] loadFileDescriptors(File walFilesDir)
                                            throws IgniteCheckedException

Scans provided folder for a WAL segment files

Parameters:

walFilesDir - directory to scan

Returns:

found WAL file descriptors

Throws:

IgniteCheckedException

segmentSize

public long segmentSize(long idx)

Getting local WAL segment size.

Specified by:

segmentSize in interface IgniteWriteAheadLogManager

Parameters:

idx - Absolute segment index.

Returns:

Segment size, 0 if size is unknown.

lastWritePointer

public WALPointer lastWritePointer()

Get last written pointer.

Specified by:

lastWritePointer in interface IgniteWriteAheadLogManager

Returns:

Last written pointer.

isSegmentFileName

public static boolean isSegmentFileName(@Nullable
                                        @Nullable String name)

Check that file name matches segment name.

Parameters:

name - File name.

Returns:

True if file name matches segment name.

lastTruncatedSegment

public long lastTruncatedSegment()

Getting last truncated segment.

Returns:

Absolut segment index.

totalSize

public static long totalSize(FileDescriptor... fileDescriptors)

Total size of the segments in bytes.

Returns:

Size in bytes.

walCdcDirectory

@Nullable
public @Nullable File walCdcDirectory()

Returns:

WAL cdc directory (including consistent ID as subfolder)

startAutoReleaseSegments

public void startAutoReleaseSegments()

Start automatically releasing segments when reaching DataStorageConfiguration.getMaxWalArchiveSize().

Specified by:

startAutoReleaseSegments in interface IgniteWriteAheadLogManager