Package org.apache.ignite.internal.processors.cache.persistence.snapshot

Class IgniteSnapshotManager

    • Field Detail

      • PART_DELTA_TEMPLATE

        public static final String PART_DELTA_TEMPLATE
        File name template consists of delta pages.
        See Also:
        Constant Field Values

      • INDEX_DELTA_NAME

        public static final String INDEX_DELTA_NAME
        File name template for index delta pages.
        See Also:
        Constant Field Values

      • CP_SNAPSHOT_REASON

        public static final String CP_SNAPSHOT_REASON
        Text Reason for checkpoint to start snapshot operation.
        See Also:
        Constant Field Values

      • RMT_SNAPSHOT_PREFIX

        public static final String RMT_SNAPSHOT_PREFIX
        Name prefix for each remote snapshot operation.
        See Also:
        Constant Field Values

      • DFLT_SNAPSHOT_TMP_DIR

        public static final String DFLT_SNAPSHOT_TMP_DIR
        Default snapshot directory for loading remote snapshots.
        See Also:
        Constant Field Values

      • SNP_IN_PROGRESS_ERR_MSG

        public static final String SNP_IN_PROGRESS_ERR_MSG
        Snapshot in progress error message.
        See Also:
        Constant Field Values

      • SNP_NODE_STOPPING_ERR_MSG

        public static final String SNP_NODE_STOPPING_ERR_MSG
        Error message to finalize snapshot tasks.
        See Also:
        Constant Field Values

      • INCREMENTAL_SNAPSHOT_METRICS

        public static final String INCREMENTAL_SNAPSHOT_METRICS
        Incremental snapshot metrics prefix.

      • SNAPSHOT_METAFILE_TMP_EXT

        public static final String SNAPSHOT_METAFILE_TMP_EXT
        Snapshot temporary metafile extension.
        See Also:
        Constant Field Values

      • SNAPSHOT_RUNNER_THREAD_PREFIX

        public static final String SNAPSHOT_RUNNER_THREAD_PREFIX
        Prefix for snapshot threads.
        See Also:
        Constant Field Values

      • SNAPSHOT_TRANSFER_RATE_DMS_KEY

        public static final String SNAPSHOT_TRANSFER_RATE_DMS_KEY
        Snapshot transfer rate distributed configuration key
        See Also:
        Constant Field Values

      • DFLT_SNAPSHOT_TRANSFER_RATE_BYTES

        public static final long DFLT_SNAPSHOT_TRANSFER_RATE_BYTES
        Snapshot transfer rate is unlimited by default.
        See Also:
        Constant Field Values

      • SNAPSHOT_LIMITED_TRANSFER_BLOCK_SIZE_BYTES

        public static final int SNAPSHOT_LIMITED_TRANSFER_BLOCK_SIZE_BYTES
        Maximum block size for limited snapshot transfer (64KB by default).
        See Also:
        Constant Field Values

      • SNP_RUNNING_DIR_KEY

        public static final String SNP_RUNNING_DIR_KEY
        Metastorage key to save currently running snapshot directory path.
        See Also:
        Constant Field Values

      • DFLT_CHECK_ON_RESTORE

        public static final boolean DFLT_CHECK_ON_RESTORE
        Default value of check flag.
        See Also:
        Constant Field Values

      • SNAPSHOT_FINISHED_WRN_MSG

        public static final String SNAPSHOT_FINISHED_WRN_MSG
        Snapshot operation finish with warnings log message.
        See Also:
        Constant Field Values

      • INC_SNP_NAME_PATTERN

        public static final Pattern INC_SNP_NAME_PATTERN
        Pattern for incremental snapshot directory names.

    • Constructor Detail

      • IgniteSnapshotManager

        public IgniteSnapshotManager​(GridKernalContext ctx)
        Parameters:
        ctx - Kernal context.

    • Method Detail

      • partDeltaFile

        public static File partDeltaFile​(File snapshotCacheDir,
                                 int partId)
        Parameters:
        snapshotCacheDir - Snapshot directory to store files.
        partId - Cache partition identifier.
        Returns:
        A file representation.

      • partDeltaIndexFile

        public static File partDeltaIndexFile​(File delta)
        Partition delta index file. Represents a sequence of page indexes that written to a delta.
        Parameters:
        delta - File with delta pages.
        Returns:
        File with delta pages index.

      • partDeltaFileName

        public static String partDeltaFileName​(int partId)
        Parameters:
        partId - Partition id.
        Returns:
        File name of delta partition pages.

      • deleteSnapshot

        public void deleteSnapshot​(File snpDir,
                           PdsFolderSettings<?> pdsSettings)
        Parameters:
        snpDir - Snapshot dir.
        pdsSettings - PDS settings.

      • snapshotLocalDir

        public File snapshotLocalDir​(String snpName)
        Parameters:
        snpName - Snapshot name.
        Returns:
        Local snapshot directory for snapshot with given name.

      • snapshotLocalDir

        public File snapshotLocalDir​(String snpName,
                             @Nullable
                             @Nullable String snpPath)
        Parameters:
        snpName - Snapshot name.
        snpPath - Snapshot directory path.
        Returns:
        Local snapshot directory where snapshot files are located.

      • incrementalSnapshotLocalDir

        public File incrementalSnapshotLocalDir​(String snpName,
                                        @Nullable
                                        @Nullable String snpPath,
                                        int incIdx)
        Returns path to specific incremental snapshot. For example, "work/snapshots/mybackup/increments/0000000000000001".
        Parameters:
        snpName - Snapshot name.
        snpPath - Snapshot directory path.
        incIdx - Increment index.
        Returns:
        Local snapshot directory where snapshot files are located.

      • incrementalSnapshotsLocalRootDir

        public File incrementalSnapshotsLocalRootDir​(String snpName,
                                             @Nullable
                                             @Nullable String snpPath)
        Returns root folder for incremental snapshot. For example, "work/snapshots/mybackup/increments/".
        Parameters:
        snpName - Snapshot name.
        snpPath - Snapshot directory path.
        Returns:
        Local snapshot directory where snapshot files are located.

      • incrementalSnapshotWalsDir

        public static File incrementalSnapshotWalsDir​(File incSnpDir,
                                              String consId)
        Parameters:
        incSnpDir - Incremental snapshot directory.
        consId - Consistent ID.
        Returns:
        WALs directory for specified incremental snapshot.

      • snapshotTmpDir

        public File snapshotTmpDir()
        Returns:
        Node snapshot working directory.

      • handleIncrementalSnapshotId

        public void handleIncrementalSnapshotId​(UUID id,
                                        long topVer)
        Handles received incremental snapshot ID from remote node.
        Parameters:
        id - Incremental snapshot ID.
        topVer - Incremental snapshot topology version.

      • storeSnapshotMeta

        public <M extends Serializable> void storeSnapshotMeta​(M meta,
                                                       File smf)
        Stores snapshot metadata.
        Parameters:
        meta - Metadata to store.
        smf - File to store.

      • isSnapshotCreating

        public boolean isSnapshotCreating()
        Returns:
        True if snapshot operation is in progress.

      • streamerWarning

        public void streamerWarning()
        Sets the streamer warning flag to current snapshot process if it is active.

      • currentCreateRequest

        @Nullable
public @Nullable SnapshotOperationRequest currentCreateRequest()
        Returns:
        Current create snapshot request. Null if there is no create snapshot operation in progress.

      • isRestoring

        public boolean isRestoring()
        Check if snapshot restore process is currently running.
        Returns:
        True if the snapshot restore operation is in progress.

      • isRestoring

        public boolean isRestoring​(String snpName)
        Check if snapshot restore process is currently running.
        Parameters:
        snpName - Snapshot name.
        Returns:
        True if the snapshot restore operation from the specified snapshot is in progress locally.

      • isRestoring

        public boolean isRestoring​(CacheConfiguration<?,​?> ccfg)
        Check if the cache or group with the specified name is currently being restored from the snapshot.
        Parameters:
        ccfg - Cache configuration.
        Returns:
        True if the cache or group with the specified name is being restored.

      • restoreStatus

        public IgniteFuture<Boolean> restoreStatus​(String snpName)
        Status of the restore operation cluster-wide.
        Parameters:
        snpName - Snapshot name.
        Returns:
        Future that will be completed when the status of the restore operation is received from all the server nodes. The result of this future will be false if the restore process with the specified snapshot name is not running on all nodes.

      • sequentialWrite

        public boolean sequentialWrite()
        Returns:
        True if disk writes during snapshot process should be in a sequential manner when possible.

      • cacheStartRequiredAliveNodes

        public Set<UUID> cacheStartRequiredAliveNodes​(@Nullable
                                              @Nullable IgniteUuid restoreId)
        Parameters:
        restoreId - Restore process ID.
        Returns:
        Server nodes on which a successful start of the cache(s) is required, if any of these nodes fails when starting the cache(s), the whole procedure is rolled back.

      • localSnapshotNames

        public List<String> localSnapshotNames​(@Nullable
                                       @Nullable String snpPath)
        Returns:
        List of all known snapshots on the local node.

      • cancelSnapshot

        public IgniteFuture<Void> cancelSnapshot​(String name)
        Cancel running snapshot operation. All intermediate results of cancelled snapshot operation will be deleted. If snapshot already created this command will have no effect.
        Specified by:
        cancelSnapshot in interface IgniteSnapshot
        Parameters:
        name - Snapshot name to cancel.
        Returns:
        Future which will be completed when cancel operation finished.

      • cancelSnapshotOperation

        public IgniteFuture<Boolean> cancelSnapshotOperation​(UUID reqId)
        Parameters:
        reqId - Snapshot operation request ID.
        Returns:
        Future which will be completed when cancel operation finished.

      • cancelLocalSnapshotTask

        public boolean cancelLocalSnapshotTask​(String name)
        Parameters:
        name - Snapshot name to cancel operation on local node.
        Returns:
        True if the snapshot operation was canceled.

      • cancelSnapshotRestore

        public IgniteFuture<Boolean> cancelSnapshotRestore​(String name)
        Cancel snapshot restore operation.
        Specified by:
        cancelSnapshotRestore in interface IgniteSnapshot
        Parameters:
        name - Snapshot name.
        Returns:
        Future that will be finished when the process is complete on all nodes. The result of this future will be false if the restore process with the specified snapshot name is not running at all.

      • createDump

        public IgniteFuture<Void> createDump​(String name,
                                     @Nullable
                                     @Nullable Collection<String> cacheGrpNames)
        Creates cache groups dump. Dump is consistent entry by entry backup of cache group content. In-memory cache groups also supported.
        Specified by:
        createDump in interface IgniteSnapshot
        Parameters:
        name - Dump name.
        cacheGrpNames - Cache groups to include in snapshot or null to include all.
        Returns:
        Future which will be completed when dump ends.

      • cancelLocalRestoreTask

        @Deprecated
public IgniteFuture<Boolean> cancelLocalRestoreTask​(String name)
        Deprecated.
        Use cancelLocalSnapshotOperations(UUID) instead.
        Parameters:
        name - Snapshot name.
        Returns:
        Future that will be finished when process the process is complete. The result of this future will be false if the restore process with the specified snapshot name is not running at all.

      • checkSnapshot

        public IgniteInternalFuture<SnapshotPartitionsVerifyTaskResult> checkSnapshot​(String name,
                                                                              @Nullable
                                                                              @Nullable String snpPath)
        Checks snapshot.
        Parameters:
        name - Snapshot name.
        snpPath - Snapshot directory path.
        Returns:
        Future with the result of execution snapshot partitions verify task, which besides calculating partition hashes of IdleVerifyResultV2 also contains the snapshot metadata distribution across the cluster.

      • checkSnapshot

        public IgniteInternalFuture<SnapshotPartitionsVerifyTaskResult> checkSnapshot​(String name,
                                                                              @Nullable
                                                                              @Nullable String snpPath,
                                                                              int incIdx)
        Checks snapshot and its increments.
        Parameters:
        name - Snapshot name.
        snpPath - Snapshot directory path.
        incIdx - Incremental snapshot index.
        Returns:
        Future with the result of execution snapshot partitions verify task, which besides calculating partition hashes of IdleVerifyResultV2 also contains the snapshot metadata distribution across the cluster.

      • checkSnapshot

        public IgniteInternalFuture<SnapshotPartitionsVerifyTaskResult> checkSnapshot​(String name,
                                                                              @Nullable
                                                                              @Nullable String snpPath,
                                                                              @Nullable
                                                                              @Nullable Collection<String> grps,
                                                                              boolean includeCustomHandlers,
                                                                              int incIdx,
                                                                              boolean check)
        The check snapshot procedure performs compute operation over the whole cluster to verify the snapshot entirety and partitions consistency. The result future will be completed with an exception if this exception is not related to the check procedure, and will be completed normally with the IdleVerifyResult.
        Parameters:
        name - Snapshot name.
        snpPath - Snapshot directory path.
        grps - Collection of cache group names to check.
        includeCustomHandlers - True to invoke all user-defined SnapshotHandlerType.RESTORE handlers, otherwise only system consistency check will be performed.
        incIdx - Incremental snapshot index.
        check - If true check snapshot integrity.
        Returns:
        Future with the result of execution snapshot partitions verify task, which besides calculating partition hashes of IdleVerifyResultV2 also contains the snapshot metadata distribution across the cluster.

      • snapshotCacheDirectories

        public List<File> snapshotCacheDirectories​(String snpName,
                                           @Nullable
                                           @Nullable String snpPath,
                                           String folderName,
                                           Predicate<String> names)
        Parameters:
        snpName - Snapshot name.
        folderName - The name of a directory for the cache group.
        names - Cache group names to filter.
        Returns:
        The list of cache or cache group names in given snapshot on local node.

      • readSnapshotMetadatas

        public List<SnapshotMetadata> readSnapshotMetadatas​(String snpName,
                                                    @Nullable
                                                    @Nullable String snpPath)
        Parameters:
        snpName - Snapshot name.
        snpPath - Snapshot directory path.
        Returns:
        List of snapshot metadata for the given snapshot name on local node. If snapshot has been taken from local node the snapshot metadata for given local node will be placed on the first place.

      • readIncrementalSnapshotMetadatas

        public Collection<IncrementalSnapshotMetadata> readIncrementalSnapshotMetadatas​(String snpName)
        Parameters:
        snpName - Snapshot name.
        Returns:
        Collection of incremental snapshots metafiles.

      • createSnapshot

        public IgniteFuture<Void> createSnapshot​(String name)
        Create a consistent copy of all persistence cache groups from the whole cluster.
        Specified by:
        createSnapshot in interface IgniteSnapshot
        Parameters:
        name - Snapshot unique name which satisfies the following name pattern [a-zA-Z0-9_].
        Returns:
        Future which will be completed when the process ends.

      • createIncrementalSnapshot

        public IgniteFuture<Void> createIncrementalSnapshot​(String name)
        Create an incremental snapshot for an existing snapshot.
        Specified by:
        createIncrementalSnapshot in interface IgniteSnapshot
        Parameters:
        name - Snapshot name.
        Returns:
        Future which will be completed when the process ends.

      • createSnapshot

        public IgniteFutureImpl<Void> createSnapshot​(String name,
                                             @Nullable
                                             @Nullable String snpPath,
                                             boolean incremental,
                                             boolean onlyPrimary)
        Create a consistent copy of all persistence cache groups from the whole cluster.
        Parameters:
        name - Snapshot unique name which satisfies the following name pattern [a-zA-Z0-9_].
        snpPath - Snapshot directory path.
        incremental - Incremental snapshot flag.
        onlyPrimary - If true snapshot only primary copies of partitions.
        Returns:
        Future which will be completed when a process ends.

      • createSnapshot

        public IgniteFutureImpl<Void> createSnapshot​(String name,
                                             @Nullable
                                             @Nullable String snpPath,
                                             @Nullable
                                             @Nullable Collection<String> cacheGrpNames,
                                             boolean incremental,
                                             boolean onlyPrimary,
                                             boolean dump,
                                             boolean compress,
                                             boolean encrypt)
        Create a consistent copy of all persistence cache groups from the whole cluster. Note, encrypt flag can be used only for cache dump. Full snapshots store partition files itself. So if cache is encrypted (CacheConfiguration.isEncryptionEnabled()= true) then snapshot files will be encrypted. On the other hand, dumps stores only entry data and can be used fo in-memory caches. So we provide an ability to encrypt dump content to protect data on the disk.
        Parameters:
        name - Snapshot unique name which satisfies the following name pattern [a-zA-Z0-9_].
        snpPath - Snapshot directory path.
        cacheGrpNames - Cache groups to include in snapshot or null to include all.
        incremental - Incremental snapshot flag.
        onlyPrimary - If true snapshot only primary copies of partitions.
        dump - If true cache dump must be created.
        compress - If true then compress partition files.
        encrypt - If true then content of dump encrypted.
        Returns:
        Future which will be completed when a process ends.

      • restoreSnapshot

        public IgniteFuture<Void> restoreSnapshot​(String name,
                                          @Nullable
                                          @Nullable Collection<String> grpNames)
        Restore cache group(s) from the snapshot.

        NOTE: Cache groups to be restored from the snapshot must not present in the cluster, if they present, they must be destroyed by the user (eg with IgniteCache.destroy()) before starting this operation.

        Specified by:
        restoreSnapshot in interface IgniteSnapshot
        Parameters:
        name - Snapshot name.
        grpNames - Cache groups to be restored or null to restore all cache groups from the snapshot.
        Returns:
        Future which will be completed when restore operation finished.

      • restoreSnapshot

        public IgniteFuture<Void> restoreSnapshot​(String name,
                                          @Nullable
                                          @Nullable Collection<String> grpNames,
                                          int incIdx)
        Restore cache group(s) from the snapshot and its increments. Snapshot is restored first and after that all increments are restored sequentially from the 1 to the specified incIdx.

        NOTE: Cache groups to be restored from the snapshot must not present in the cluster, if they present, they must be destroyed by the user (eg with IgniteCache.destroy()) before starting this operation.

        Specified by:
        restoreSnapshot in interface IgniteSnapshot
        Parameters:
        name - Snapshot name.
        grpNames - Cache groups to be restored or null to restore all cache groups from the snapshot.
        incIdx - Index of incremental snapshot.
        Returns:
        Future which will be completed when restore operation finished.

      • restoreSnapshot

        public IgniteFutureImpl<Void> restoreSnapshot​(String name,
                                              @Nullable
                                              @Nullable String snpPath,
                                              @Nullable
                                              @Nullable Collection<String> grpNames)
        Restore cache group(s) from the snapshot.
        Parameters:
        name - Snapshot name.
        snpPath - Snapshot directory path.
        grpNames - Cache groups to be restored or null to restore all cache groups from the snapshot.
        Returns:
        Future which will be completed when restore operation finished.

      • restoreSnapshot

        public IgniteFutureImpl<Void> restoreSnapshot​(String name,
                                              @Nullable
                                              @Nullable String snpPath,
                                              @Nullable
                                              @Nullable Collection<String> grpNames,
                                              int incIdx,
                                              boolean check)
        Restore cache group(s) from the snapshot.
        Parameters:
        name - Snapshot name.
        snpPath - Snapshot directory path.
        grpNames - Cache groups to be restored or null to restore all cache groups from the snapshot.
        incIdx - Index of incremental snapshot.
        check - If true check snapshot before restore.
        Returns:
        Future which will be completed when restore operation finished.

      • isSnapshotOperation

        public static boolean isSnapshotOperation​(DiscoveryEvent evt)
        Parameters:
        evt - Discovery event to check.
        Returns:
        true if exchange started by snapshot operation.

      • onDoneBeforeTopologyUnlock

        public void onDoneBeforeTopologyUnlock​(GridDhtPartitionsExchangeFuture fut)
        Callback from exchange process completion; called before topology is unlocked. Guarantees that no updates were performed on local node since exchange process started.
        Specified by:
        onDoneBeforeTopologyUnlock in interface PartitionsExchangeAware
        Parameters:
        fut - Partition map exchange future.

      • requestRemoteSnapshotFiles

        public IgniteInternalFuture<Void> requestRemoteSnapshotFiles​(UUID rmtNodeId,
                                                             UUID reqId,
                                                             String snpName,
                                                             @Nullable
                                                             @Nullable String rmtSnpPath,
                                                             Map<Integer,​Set<Integer>> parts,
                                                             BooleanSupplier stopChecker,
                                                             BiConsumer<@Nullable File,​@Nullable Throwable> partHnd)
                                                      throws IgniteCheckedException
        Parameters:
        rmtNodeId - The remote node to connect to.
        reqId - Snapshot operation request ID.
        snpName - Snapshot name to request.
        rmtSnpPath - Snapshot directory path on the remote node.
        parts - Collection of pairs group and appropriate cache partition to be snapshot.
        stopChecker - Node stop or process interrupt checker.
        partHnd - Received partition handler.
        Throws:
        IgniteCheckedException

      • onCacheGroupsStopped

        public void onCacheGroupsStopped​(List<Integer> grps)
        Parameters:
        grps - List of cache groups which will be destroyed.

      • snapshotMetaFileName

        public static String snapshotMetaFileName​(String consId)
        Parameters:
        consId - Consistent node id.
        Returns:
        Snapshot metadata file name.

      • partitionRowIterator

        public GridCloseableIterator<CacheDataRow> partitionRowIterator​(String snpName,
                                                                String folderName,
                                                                String grpName,
                                                                int partId,
                                                                @Nullable
                                                                @Nullable EncryptionCacheKeyProvider encrKeyProvider)
                                                         throws IgniteCheckedException
        Parameters:
        snpName - Snapshot name.
        folderName - The node folder name, usually it's the same as the U.maskForFileName(consistentId).
        grpName - Cache group name.
        partId - Partition id.
        encrKeyProvider - Encryption keys provider to create encrypted IO. If null, no encrypted IO is used.
        Returns:
        Iterator over partition.
        Throws:
        IgniteCheckedException - If and error occurs.

      • currentSnapshotTask

        public <T extends org.apache.ignite.internal.processors.cache.persistence.snapshot.AbstractSnapshotFutureTask<?>> T currentSnapshotTask​(Class<T> snpTaskCls)
        Returns:
        Current snapshot task.

      • writeSnapshotDirectoryToMetastorage

        public void writeSnapshotDirectoryToMetastorage​(File snpLocDir)
        Parameters:
        snpLocDir - Snapshot local directory.

      • disableIncrementalSnapshotsCreation

        public void disableIncrementalSnapshotsCreation​(MetaStorage metaStorage,
                                                int grpId)
        Disables creation of incremental snapshots for the given cache group.
        Parameters:
        metaStorage - External metastorage, useful if the flag is set before cluster activation.
        grpId - Group ID.

      • incrementalSnapshotCreationDisabledKey

        public static String incrementalSnapshotCreationDisabledKey​(int grpId)
        Convert cache group ID to key for INC_SNP_DISABLED_KEY_PREFIX metastorage records.
        Parameters:
        grpId - Group ID.
        Returns:
        Key.

      • ioFactory

        public void ioFactory​(FileIOFactory ioFactory)
        Parameters:
        ioFactory - Factory to create IO interface over a page stores.

      • ioFactory

        public FileIOFactory ioFactory()
        Returns:
        Factory to create IO interface over a page stores.

      • resolveSnapshotWorkDirectory

        public static File resolveSnapshotWorkDirectory​(IgniteConfiguration cfg)
        Parameters:
        cfg - Ignite configuration.
        Returns:
        Snapshot directory resolved through given configuration.

      • resolveSnapshotWorkDirectory

        public static File resolveSnapshotWorkDirectory​(IgniteConfiguration cfg,
                                                boolean create)
        Parameters:
        cfg - Ignite configuration.
        create - If true then create resolved directory if not exists.
        Returns:
        Snapshot directory resolved through given configuration.

      • cacheChangedException

        public static String cacheChangedException​(int cacheId,
                                           String name)
        Throw cache changed exception.
        Parameters:
        cacheId - Cache id.
        name - Cache name.

      • incrementalSnapshotId

        @Nullable
public @Nullable UUID incrementalSnapshotId()
        Returns:
        Current incremental snapshot ID.