com.snowflake.client.jdbc

Class SnowflakeChunkDownloader

java.lang.Object

com.snowflake.client.jdbc.SnowflakeChunkDownloader

public class SnowflakeChunkDownloader
extends Object

Class for managing async download of offline result chunks Created by jhuang on 11/12/14.

Constructor Summary

Constructors

Constructor and Description
SnowflakeChunkDownloader(int colCount, com.fasterxml.jackson.databind.JsonNode chunksData, Integer prefetchSlots, Integer prefetchThreads, String qrmk, com.fasterxml.jackson.databind.JsonNode chunkHeaders, int networkTimeoutInMilli, Boolean useJsonParser) Constructor to initialize downloader

Method Summary

Modifier and Type	Method and Description
void	addDownloadTime(long downloadTime) add download time
void	addParsingTime(long parsingTime) add parsing time
static ThreadPoolExecutor	createChunkDownloaderExecutorService(String threadNamePrefix, int parallel) Create a pool of downloader threads.
static Callable<Void>	getDownloadChunkCallable(SnowflakeChunkDownloader downloader, SnowflakeResultChunk resultChunk, String qrmk, int chunkIndex, Map<String,String> chunkHeadersMap, int networkTimeoutInMilli) Create a download callable that will be run in download thread
SnowflakeResultChunk	getNextChunkToConsume() The method does the following: 1.
void	terminate() terminate the downloader

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SnowflakeChunkDownloader

public SnowflakeChunkDownloader(int colCount,
                                com.fasterxml.jackson.databind.JsonNode chunksData,
                                Integer prefetchSlots,
                                Integer prefetchThreads,
                                String qrmk,
                                com.fasterxml.jackson.databind.JsonNode chunkHeaders,
                                int networkTimeoutInMilli,
                                Boolean useJsonParser)

Constructor to initialize downloader

Parameters:

colCount - number of columns to expect

chunksData - JSON object contains all the chunk information

prefetchSlots - number of prefetch chunks

prefetchThreads - number of prefetch threads

qrmk - Query Result Master Key

chunkHeaders - JSON object contains information about chunk headers

networkTimeoutInMilli - network timeout

useJsonParser - should JsonParser be used instead of object deserialization

Method Detail

createChunkDownloaderExecutorService

public static ThreadPoolExecutor createChunkDownloaderExecutorService(String threadNamePrefix,
                                                                      int parallel)

Create a pool of downloader threads.

Parameters:

threadNamePrefix -

parallel -

Returns:

getNextChunkToConsume

public SnowflakeResultChunk getNextChunkToConsume()
                                           throws InterruptedException,
                                                  SnowflakeSQLException

The method does the following: 1. free the previous chunk data and submit a new chunk to be downloaded 2. get next chunk to consume, if it is not ready for consumption, it waits until it is ready

Returns:

next SnowflakeResultChunk to be consumed

Throws:

InterruptedException - if downloading thread was interrupted

SnowflakeSQLException - if downloader encountered an error

terminate

public void terminate()

terminate the downloader

addDownloadTime

public void addDownloadTime(long downloadTime)

add download time

Parameters:

downloadTime - Time for downloading a single chunk

addParsingTime

public void addParsingTime(long parsingTime)

add parsing time

Parameters:

parsingTime - Time for parsing a single chunk

getDownloadChunkCallable

public static Callable<Void> getDownloadChunkCallable(SnowflakeChunkDownloader downloader,
                                                      SnowflakeResultChunk resultChunk,
                                                      String qrmk,
                                                      int chunkIndex,
                                                      Map<String,String> chunkHeadersMap,
                                                      int networkTimeoutInMilli)

Create a download callable that will be run in download thread

Parameters:

downloader - object to download the chunk

resultChunk - object contains information about the chunk will be downloaded

qrmk - Query Result Master Key

chunkIndex - the index of the chunk which will be downloaded in array chunks. This is mainly for logging purpose

chunkHeadersMap - contains headers needed to be added when downloading from s3

networkTimeoutInMilli - network timeout

Returns:

A callable responsible for downloading chunk