Package com.databricks.jdbc.api.impl.arrow

Class StreamingChunkProvider

java.lang.Object

com.databricks.jdbc.api.impl.arrow.StreamingChunkProvider

All Implemented Interfaces:

ChunkProvider

public class StreamingChunkProvider
extends Object
implements ChunkProvider

A streaming chunk provider that fetches chunk links proactively and downloads chunks in parallel.

Key features:

• No dependency on total chunk count - streams until end of data
• Proactive link prefetching with configurable window
• Memory-bounded parallel downloads
• Automatic link refresh on expiration

This provider uses two key windows:

• Link prefetch window: How many links to fetch ahead of consumption
• Download window: How many chunks to keep in memory (downloading or ready)

Constructor Summary

Constructors

Constructor	Description
StreamingChunkProvider(ChunkLinkFetcher linkFetcher, IDatabricksHttpClient httpClient, CompressionCodec compressionCodec, StatementId statementId, int maxChunksInMemory, int linkPrefetchWindow, int chunkReadyTimeoutSeconds, double cloudFetchSpeedThreshold, ChunkLinkFetchResult initialLinks)	Creates a new StreamingChunkProvider.

Method Summary

Modifier and Type	Method	Description
void	close()	Closes the chunk provider and releases any resources associated with it.
AbstractArrowResultChunk	getChunk()	Retrieves the current chunk of data after a successful call to ChunkProvider.next().
long	getChunkCount()	Returns the total chunk count only when all chunks have been discovered.
long	getRowCount()
boolean	hasNextChunk()	Checks if there are more chunks available to iterate over.
boolean	isClosed()
boolean	next()	Advances to the next available chunk.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

StreamingChunkProvider

public StreamingChunkProvider(ChunkLinkFetcher linkFetcher,
                              IDatabricksHttpClient httpClient,
                              CompressionCodec compressionCodec,
                              StatementId statementId,
                              int maxChunksInMemory,
                              int linkPrefetchWindow,
                              int chunkReadyTimeoutSeconds,
                              double cloudFetchSpeedThreshold,
                              ChunkLinkFetchResult initialLinks)
                       throws DatabricksParsingException

Creates a new StreamingChunkProvider.

Parameters:

linkFetcher - Fetcher for chunk links

httpClient - HTTP client for downloads

compressionCodec - Codec for decompressing chunk data

statementId - Statement ID for logging and chunk creation

maxChunksInMemory - Maximum chunks to keep in memory (download window)

linkPrefetchWindow - How many links to fetch ahead

chunkReadyTimeoutSeconds - Timeout waiting for chunk to be ready

cloudFetchSpeedThreshold - Speed threshold for logging warnings

initialLinks - Initial links provided with result data (avoids extra fetch), may be null

Throws:

DatabricksParsingException

Method Detail

hasNextChunk

public boolean hasNextChunk()

Description copied from interface: ChunkProvider

Checks if there are more chunks available to iterate over.

Specified by:

hasNextChunk in interface ChunkProvider

Returns:

true if there are additional chunks to be retrieved; false otherwise.

next

public boolean next()
             throws DatabricksSQLException

Description copied from interface: ChunkProvider

Advances to the next available chunk. This method should be called before calling ChunkProvider.getChunk() to retrieve the data from the next chunk.

Specified by:

next in interface ChunkProvider

Returns:

true if the next chunk was successfully moved to; false if there are no more chunks.

Throws:

DatabricksSQLException

getChunk

public AbstractArrowResultChunk getChunk()
                                  throws DatabricksSQLException

Description copied from interface: ChunkProvider

Retrieves the current chunk of data after a successful call to ChunkProvider.next().

Specified by:

getChunk in interface ChunkProvider

Returns:

The current AbstractArrowResultChunk containing the data.

Throws:

DatabricksSQLException - if an error occurs while fetching the chunk.

close

public void close()

Description copied from interface: ChunkProvider

Closes the chunk provider and releases any resources associated with it. After calling this method, the chunk provider should not be used again.

Specified by:

close in interface ChunkProvider

getRowCount

public long getRowCount()

Specified by:

getRowCount in interface ChunkProvider

getChunkCount

public long getChunkCount()

Returns the total chunk count only when all chunks have been discovered.

In streaming mode, the total chunk count is unknown until we reach the end of the stream. This method returns -1 if chunks are still being discovered, and the actual count once all chunks have been fetched.

Specified by:

getChunkCount in interface ChunkProvider

Returns:

the total chunk count if all chunks have been discovered, or -1 if still streaming

isClosed

public boolean isClosed()

Specified by:

isClosed in interface ChunkProvider