public class SnowflakeChunkDownloader extends Object
| Constructor and Description |
|---|
SnowflakeChunkDownloader(int colCount,
com.fasterxml.jackson.databind.JsonNode chunksData,
Integer prefetchSlots,
Integer prefetchThreads,
String qrmk,
com.fasterxml.jackson.databind.JsonNode chunkHeaders,
int networkTimeoutInMilli,
Boolean useJsonParser)
Constructor to initialize downloader
|
| Modifier and Type | Method and Description |
|---|---|
void |
addDownloadTime(long downloadTime)
add download time
|
void |
addParsingTime(long parsingTime)
add parsing time
|
static ThreadPoolExecutor |
createChunkDownloaderExecutorService(String threadNamePrefix,
int parallel)
Create a pool of downloader threads.
|
static Callable<Void> |
getDownloadChunkCallable(SnowflakeChunkDownloader downloader,
SnowflakeResultChunk resultChunk,
String qrmk,
int chunkIndex,
Map<String,String> chunkHeadersMap,
int networkTimeoutInMilli)
Create a download callable that will be run in download thread
|
SnowflakeResultChunk |
getNextChunkToConsume()
The method does the following:
1.
|
void |
terminate()
terminate the downloader
|
public SnowflakeChunkDownloader(int colCount,
com.fasterxml.jackson.databind.JsonNode chunksData,
Integer prefetchSlots,
Integer prefetchThreads,
String qrmk,
com.fasterxml.jackson.databind.JsonNode chunkHeaders,
int networkTimeoutInMilli,
Boolean useJsonParser)
colCount - number of columns to expectchunksData - JSON object contains all the chunk informationprefetchSlots - number of prefetch chunksprefetchThreads - number of prefetch threadsqrmk - Query Result Master KeychunkHeaders - JSON object contains information about chunk headersnetworkTimeoutInMilli - network timeoutuseJsonParser - should JsonParser be used instead of object
deserializationpublic static ThreadPoolExecutor createChunkDownloaderExecutorService(String threadNamePrefix, int parallel)
threadNamePrefix - parallel - public SnowflakeResultChunk getNextChunkToConsume() throws InterruptedException, SnowflakeSQLException
InterruptedException - if downloading thread was interruptedSnowflakeSQLException - if downloader encountered an errorpublic void terminate()
public void addDownloadTime(long downloadTime)
downloadTime - Time for downloading a single chunkpublic void addParsingTime(long parsingTime)
parsingTime - Time for parsing a single chunkpublic static Callable<Void> getDownloadChunkCallable(SnowflakeChunkDownloader downloader, SnowflakeResultChunk resultChunk, String qrmk, int chunkIndex, Map<String,String> chunkHeadersMap, int networkTimeoutInMilli)
downloader - object to download the chunkresultChunk - object contains information about the chunk will
be downloadedqrmk - Query Result Master KeychunkIndex - the index of the chunk which will be downloaded in array
chunks. This is mainly for logging purposechunkHeadersMap - contains headers needed to be added when downloading from s3networkTimeoutInMilli - network timeoutCopyright © 2016. All rights reserved.