net.snowflake.client.jdbc

Class SnowflakeFileTransferAgent

java.lang.Object

net.snowflake.client.jdbc.SnowflakeFileTransferAgent

All Implemented Interfaces:

SnowflakeFixedView

public class SnowflakeFileTransferAgent
extends Object
implements SnowflakeFixedView

Class for uploading/downloading files

Author:

jhuang

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	SnowflakeFileTransferAgent.CommandType
class	SnowflakeFileTransferAgent.DownloadCommandEncryptionFacade
class	SnowflakeFileTransferAgent.DownloadCommandFacade A class for encapsulating the columns to return for the download command
static class	SnowflakeFileTransferAgent.FileCompressionType
static class	SnowflakeFileTransferAgent.ResultStatus Result status enum
static class	SnowflakeFileTransferAgent.UploadColumns A class for encapsulating the columns to return for the upload command
class	SnowflakeFileTransferAgent.UploadCommandEncryptionFacade
class	SnowflakeFileTransferAgent.UploadCommandFacade

Field Summary

Fields

Modifier and Type	Field and Description
static String	SRC_FILE_NAME_FOR_STREAM

Constructor Summary

Constructors

Constructor and Description
SnowflakeFileTransferAgent(String command, SFSession connection, SFStatement statement)

Method Summary

Modifier and Type	Method and Description
void	cancel()
List<SnowflakeColumnMetadata>	describeColumns() Describe the metadata of a fixed view.
boolean	execute()
static Set<String>	expandFileNames(String[] filePathList) process a list of file paths separated by "," and expand the wildcards if any to generate the list of paths for all files matched by the wildcards
static net.snowflake.client.jdbc.SnowflakeFileTransferAgent.remoteLocation	extractLocationAndPath(String stageLocationPath) A small helper for extracting location name and path from full location path
SnowflakeFileTransferAgent.CommandType	getCommandType()
static Callable<Void>	getDownloadFileCallable(String stageLocationType, String stageLocation, Map credentials, String srcFilePath, String localLocation, Map<String,net.snowflake.client.jdbc.SnowflakeFileTransferAgent.FileMetadata> fileMetadataMap, SnowflakeStorageClient client, SFSession connection, String command, int parallel, net.snowflake.common.core.RemoteStoreFileEncryptionMaterial encMat, String stageRegion) A callable that can be executed in a separate thread using executor service.
List<Object>	getNextRow()
Object	getResultSet()
static Callable<Void>	getUploadFileCallable(String stageLocationType, String stageLocation, String srcFilePath, boolean requireCompress, Map<String,net.snowflake.client.jdbc.SnowflakeFileTransferAgent.FileMetadata> fileMetadataMap, SnowflakeStorageClient client, SFSession connection, String command, InputStream inputStream, boolean sourceFromStream, long sourceDataSize, int parallel, File srcFile, net.snowflake.common.core.RemoteStoreFileEncryptionMaterial encMat, String stageRegion) A callable that can be executed in a separate thread using exeuctor service.
static void	renewExpiredToken(SFSession connection, String command, SnowflakeStorageClient client) This static method is called when we are handling an expired token exception It retrieves a fresh token from GS and then calls .renew() on the storage client to refresh itself with the new token
void	setCompressSourceFromStream(boolean compressSourceFromStream)
void	setDestFileNameForStreamSource(String destFileNameForStreamSource)
void	setOverwrite(boolean overwrite)
void	setSourceStream(InputStream sourceStream)
void	setSourceStreamSize(long sourceStreamSize)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

SRC_FILE_NAME_FOR_STREAM

public static final String SRC_FILE_NAME_FOR_STREAM

See Also:

Constant Field Values

Constructor Detail

SnowflakeFileTransferAgent

public SnowflakeFileTransferAgent(String command,
                                  SFSession connection,
                                  SFStatement statement)
                           throws SnowflakeSQLException

Throws:

SnowflakeSQLException

Method Detail

getUploadFileCallable

public static Callable<Void> getUploadFileCallable(String stageLocationType,
                                                   String stageLocation,
                                                   String srcFilePath,
                                                   boolean requireCompress,
                                                   Map<String,net.snowflake.client.jdbc.SnowflakeFileTransferAgent.FileMetadata> fileMetadataMap,
                                                   SnowflakeStorageClient client,
                                                   SFSession connection,
                                                   String command,
                                                   InputStream inputStream,
                                                   boolean sourceFromStream,
                                                   long sourceDataSize,
                                                   int parallel,
                                                   File srcFile,
                                                   net.snowflake.common.core.RemoteStoreFileEncryptionMaterial encMat,
                                                   String stageRegion)

A callable that can be executed in a separate thread using exeuctor service. The callable does compression if needed and upload the result to the table's staging area.

Parameters:

stageLocationType - stage location type

stageLocation - stage location

srcFilePath - source file path

requireCompress - boolean indicating if requiring compression

fileMetadataMap - map where key is file name and value is metadata

client - client object used to communicate with c3

connection - connection object

command - command string

inputStream - null if upload source is file

sourceFromStream - whether upload source is file or stream

sourceDataSize - upload data size

parallel - number of threads for parallel uploading

srcFile - source file name

encMat - not null if encryption is required

stageRegion - region name where stage persists

Returns:

a callable that uploading file to the remote store

getDownloadFileCallable

public static Callable<Void> getDownloadFileCallable(String stageLocationType,
                                                     String stageLocation,
                                                     Map credentials,
                                                     String srcFilePath,
                                                     String localLocation,
                                                     Map<String,net.snowflake.client.jdbc.SnowflakeFileTransferAgent.FileMetadata> fileMetadataMap,
                                                     SnowflakeStorageClient client,
                                                     SFSession connection,
                                                     String command,
                                                     int parallel,
                                                     net.snowflake.common.core.RemoteStoreFileEncryptionMaterial encMat,
                                                     String stageRegion)

A callable that can be executed in a separate thread using executor service. The callable download files from a stage location to a local location

Parameters:

stageLocationType - stage location type

stageLocation - stage location

credentials - credentials

srcFilePath - path that stores the downloaded file

localLocation - local location

fileMetadataMap - file metadata map

client - remote store client

connection - connection object

command - command string

encMat - remote store encryption material

parallel - number of parallel threads for downloading

stageRegion - region name where the stage persists

Returns:

a callable responsible for downloading files

execute

public boolean execute()
                throws SQLException

Throws:

SQLException

cancel

public void cancel()

expandFileNames

public static Set<String> expandFileNames(String[] filePathList)
                                   throws SnowflakeSQLException

process a list of file paths separated by "," and expand the wildcards if any to generate the list of paths for all files matched by the wildcards

Parameters:

filePathList - file path list

Returns:

a set of file names that is matched

Throws:

SnowflakeSQLException - if cannot find the file

renewExpiredToken

public static void renewExpiredToken(SFSession connection,
                                     String command,
                                     SnowflakeStorageClient client)
                              throws SnowflakeSQLException

This static method is called when we are handling an expired token exception It retrieves a fresh token from GS and then calls .renew() on the storage client to refresh itself with the new token

Parameters:

connection - a connection object

command - a command to be retried

client - a Snowflake Storage client object

Throws:

SnowflakeSQLException - if any error occurs

extractLocationAndPath

public static net.snowflake.client.jdbc.SnowflakeFileTransferAgent.remoteLocation extractLocationAndPath(String stageLocationPath)

A small helper for extracting location name and path from full location path

Parameters:

stageLocationPath - stage location

Returns:

remoteLocation object

describeColumns

public List<SnowflakeColumnMetadata> describeColumns()
                                              throws Exception

Describe the metadata of a fixed view.

Specified by:

describeColumns in interface SnowflakeFixedView

Returns:

list of column meta data

Throws:

Exception - failed to construct list

getNextRow

public List<Object> getNextRow()
                        throws Exception

Specified by:

getNextRow in interface SnowflakeFixedView

Throws:

Exception

getResultSet

public Object getResultSet()
                    throws SnowflakeSQLException

Throws:

SnowflakeSQLException

getCommandType

public SnowflakeFileTransferAgent.CommandType getCommandType()

setSourceStream

public void setSourceStream(InputStream sourceStream)

setDestFileNameForStreamSource

public void setDestFileNameForStreamSource(String destFileNameForStreamSource)

setSourceStreamSize

public void setSourceStreamSize(long sourceStreamSize)

setCompressSourceFromStream

public void setCompressSourceFromStream(boolean compressSourceFromStream)

setOverwrite

public void setOverwrite(boolean overwrite)