[BUG] [Arrow cloud Fetch] Race condition between CHUNK_RELEASED and DOWNLOAD_FAILED causes unrecoverable state

[aliakseistruh-dr]

Describe the bug
Arrow Cloud Fetch result streaming fails with a DatabricksParsingException: Invalid state transition when the IdleConnectionEvictor background thread closes the HTTP connection pool while a chunk download is in progress.

The exception surfaces as Premature end of Content-Length delimited message body on the download, followed by an invalid state machine transition attempt (CHUNK_RELEASED -> DOWNLOAD_FAILED) in the retry path, which is a terminal state with no valid outgoing transitions.

This causes the entire result stream to fail non-recoverably even though the driver attempts an internal retry.

To Reproduce

1. Use the Databricks JDBC driver with Arrow Cloud Fetch enabled (EnableArrow=1) and IdleHttpConnectionExpiry not set (driver default applies)
2. Execute a query returning a large result set split into multiple Arrow chunks (e.g. a wide table with 100+ float columns, 100k+ rows)
3. Apply resource constraints that slow down Arrow chunk downloads (e.g. 0.5 CPU, 1GB RAM container)
4. Observe that a chunk download is still in progress when the IdleConnectionEvictor fires (thrift connection idle since Arrow Cloud Fetch resolves URLs upfront and then downloads data via separate connections)

The evictor fires because:

• After the driver receives Arrow chunk URLs from the Databricks warehouse, the thrift connection becomes idle
• The evictor's idle threshold (driver default) expires during the background downloads
• connectionManager.closeIdleConnections() closes the shared HTTP pool mid-download

Expected behavior
The driver should either:

1. Not close the HTTP pool while Arrow chunk downloads are in progress, or
2. Handle the CHUNK_RELEASED -> DOWNLOAD_FAILED transition gracefully -> this is a valid race condition where the consumer thread releases a chunk concurrently with the download thread attempting to mark it failed. The state machine should allow this transition rather than throwing an unrecoverable exception.

Client side logs

  Retry attempt 1 for chunk index: 7, Error: Data parsing failed for chunk index [7] and statement [<id>].
  Exception [com.databricks.internal.apache.http.ConnectionClosedException:
    Premature end of Content-Length delimited message body (expected: 5,439,615; received: 5,439,488)]

  Failed to transition to state [DOWNLOAD_FAILED] from state [CHUNK_RELEASED] for chunk [7]
  and statement [<id>]. Stack trace:
  com.databricks.jdbc.exception.DatabricksParsingException: Invalid state transition for chunk [7]
  and statement [<id>]: CHUNK_RELEASED -> DOWNLOAD_FAILED. Valid transitions from CHUNK_RELEASED are: []
      at com.databricks.jdbc.api.impl.arrow.ArrowResultChunkStateMachine.transition(ArrowResultChunkStateMachine.java:37)
      at com.databricks.jdbc.api.impl.arrow.AbstractArrowResultChunk.setStatus(AbstractArrowResultChunk.java:245)
      at com.databricks.jdbc.api.impl.arrow.ArrowResultChunk.handleFailure(ArrowResultChunk.java:157)
      at com.databricks.jdbc.api.impl.arrow.ArrowResultChunk.downloadData(ArrowResultChunk.java:130)
      at com.databricks.jdbc.api.impl.arrow.ChunkDownloadTask.call(ChunkDownloadTask.java:71)

Root cause (from source analysis):

In ChunkDownloadTask.java, the finally block unconditionally calls chunk.setStatus(ChunkStatus.DOWNLOAD_FAILED) without checking whether the chunk has already been released by the consumer thread.

The CHUNK_RELEASED state has no valid outgoing transitions in ChunkStatus.VALID_TRANSITIONS, making CHUNK_RELEASED -> DOWNLOAD_FAILED an illegal transition even though it is a legitimate race condition in concurrent execution.

The specific race window:

Consumer thread: PROCESSING_SUCCEEDED -> releaseChunk() -> CHUNK_RELEASED
Download thread: ConnectionClosedException -> finally: setStatus(DOWNLOAD_FAILED) <- INVALID

Workaround:

Set IdleHttpConnectionExpiry to a value larger than the expected maximum chunk download time (e.g. 600 seconds) to prevent the evictor from firing during active Arrow downloads:

jdbc:databricks://<host>:443;EnableArrow=1;IdleHttpConnectionExpiry=600

Client Environment (please complete the following information):

• OS: Linux (container, linux/amd64)
• Java version: Java 17.0.18
• Java vendor: OpenJDK (FIPS - 17.0.18-internal with BouncyCastle FIPS)
• Driver Version: 3.3.1 (com.databricks:databricks-jdbc:3.3.1)
• BI Tool: Custom gRPC service (Spring Boot + jOOQ fetchLazy() + Reactor Flux.fromIterable(cursor))

Additional context

• The bug is Arrow Cloud Fetch specific. JSON result format (EnableArrow=0) is not affected as it streams synchronously through the thrift connection with no background download threads and no state machine.
• The ChunkDownloadTask.finally block should guard the state transition: only attempt setStatus(DOWNLOAD_FAILED) if the chunk is not already in CHUNK_RELEASED state, or alternatively CHUNK_RELEASED -> DOWNLOAD_FAILED should be added as a valid transition in ChunkStatus.VALID_TRANSITIONS.
• The internal retry mechanism (DOWNLOAD_RETRY) is also rendered ineffective by this bug — after the invalid transition attempt fails, the chunk remains in CHUNK_RELEASED with no path to retry.

[sreekanth-db]

Hi @aliakseistruh-dr
Quick question: Are you trying to close the ResultSet / Statement / Connection or calling statement.getMoreResults() before consuming all available data from resultSet (i.e, before rs.next() returns false) ? Or does the framework / any mechanism in your codebase closes the resultSet in case of any exception during result iteration ?

[aliakseistruh-dr]

Hi @aliakseistruh-dr Quick question: Are you trying to close the ResultSet / Statement / Connection or calling statement.getMoreResults() before consuming all available data from resultSet (i.e, before rs.next() returns false) ? Or does the framework / any mechanism in your codebase closes the resultSet in case of any exception during result iteration ?

Hi @sreekanth-db,

Yes, we use jOOQ's fetchLazy() wrapped in Flux.using inside a Reactor pipeline. The cursor is closed reactively on any terminal signal (including cancel/error), which can happen before all rows are consumed. Combined with EnableArrow=1, this creates exactly the race I described: the consumer releases the chunk while the Arrow background download thread is still active.
We also have IdleHttpConnectionExpiry=600 set, but the premature cursor close from the reactive pipeline triggers the same CHUNK_RELEASED -> DOWNLOAD_FAILED invalid transition independently of the evictor.

Disabling Arrow (EnableArrow=0) is our current workaround candidate.

[sreekanth-db]

I've raised a PR to fix the invalid state transition warning: #1407

However, I want to clarify that the CHUNK_RELEASED -> DOWNLOAD_FAILED invalid state transition you're seeing in the logs is a consequence, not the root cause of your result result set failure. Here's what's actually happening:

1. The IdleConnectionEvictor kills download connections (due to CPU/memory starvation making connections appear idle at the socket level)
2. The download fails with ConnectionClosedException: Premature end of Content-Length delimited message body
3. This failure propagates to the consumer thread via rs.next() as a DatabricksSQLException
4. When your framework/code calls rs.close() in response (or via try-with-resources), the cleanup path races with download threads still processing errors — producing the invalid state transition warning

The invalid transition warning is silently caught and logged — it doesn't cause the result set failure. The result set failure is caused by the download error at step 2. The PR fixes the invalid state transition (CHUNK_RELEASED -> DOWNLOAD_FAILED) by ensuring doClose() waits for download threads to finish before releasing chunks.

For the underlying download failures caused by the evictor, your workaround of setting IdleHttpConnectionExpiry to higher number remains the right approach — it gives slow downloads more time before the evictor considers connections idle.

[sreekanth-db]

Since you are using a resource constrained environment, you may also reduce cloudFetchThreadPoolSize instead of default value 16 - fewer concurrent downloads means less CPU and memory contention, reducing the chance of connections appearing idle to the evictor.

[aliakseistruh-dr]

Since you are using a resource constrained environment, you may also reduce cloudFetchThreadPoolSize instead of default value 16 - fewer concurrent downloads means less CPU and memory contention, reducing the chance of connections appearing idle to the evictor.

Hi,

Reducing cloudFetchThreadPoolSize would lower concurrency but wouldn't eliminate the race it only changes the probability of hitting it.

The underlying issue is that ChunkDownloadTask.finally unconditionally calls chunk.setStatus(DOWNLOAD_FAILED) without checking current state, and CHUNK_RELEASED -> DOWNLOAD_FAILED has no valid outgoing transitions in ChunkStatus.VALID_TRANSITIONS.

Even with cloudFetchThreadPoolSize=1, the window exists, if the consumer thread reaches PROCESSING_SUCCEEDED-> releaseChunk()-> CHUNK_RELEASED while the single download thread is in its finally block, you get the same invalid transition. The fix needs to be at the state machine level, not at the concurrency level.

So i think your opened PR #1407 should fix that problem.

[samikshya-db]

The fix is now merged in #1407
And will be released in the next version release. Thanks.