Fix streaming pause handoff race + workaround Julia 1.10 streaming issues - #129
Merged
Conversation
read_callback pauses libcurl's send direction whenever the request buffer runs dry, which for a streaming request is every time the caller has nothing queued, and the one-shot curl_easy_pause(CURLPAUSE_CONT) the request pump issues is what resumes it. The pump staged its buffer, reset curl_done_reading, and un-paused as three separate steps, and at end of stream it un-paused and only then closed the request stream. A read_callback landing in one of those gaps saw a stream that was still open, paused again, and left the transfer paused with nobody to resume it: all data exchanged, both peers done, call hung until its deadline. Reproduced against the Go test server on 1.12.6 (libcurl 8.15.0) with -t 8: a bidirectional round of 100 messages that closes its request channel after draining every response wedges about once in 2000 rounds. The wedged state is diagnostic, curl_done_reading set with the request stream already closed and the buffer empty, and it was unrecoverable even by un-pausing, because read_callback returned CURL_READFUNC_PAUSE on a set curl_done_reading before it ever checked whether the stream had ended. read_callback only ever runs while the gRPCCURL handle's lock is held: libcurl invokes it from curl_multi_socket_action, curl_multi_add_handle and curl_easy_pause, and this package calls all three under that lock, which is the same lock req.lock refers to. So both handoffs now take it and do their whole job inside it. No callback can interleave with a handoff, which leaves a pause only two places to land: before the section, where the un-pause clears it, or after it, where it is a fresh pause against an empty buffer that the next batch resumes. That makes a crossed pause impossible rather than unlikely, and needs no polling to recover from one. - the mid-stream handoff writes the buffer, resets curl_done_reading and un-pauses as one critical section, rather than as three separate steps - request_eof replaces "request_c is closed and empty" as the end-of-stream signal. Nothing is ever put into that channel, and a field write can go inside the critical section where close(::Channel) cannot: close acquires the channel's own lock, and this section runs holding the lock that serializes every transfer on the handle, so anything in it that can block stalls them all. It also keeps read_callback, which runs inside libcurl, from taking a Channel lock at all - the curl_done_reading early return in read_callback is gone. It guarded the request buffer against a concurrent write that the lock now prevents, and it was what made the wedge permanent. The extra callback curl sometimes makes after a pause needs no special case: the buffer is empty then, so the streaming branch pauses again, or ends the request if the stream has since ended Verified on 1.12.6 with the Go test server: zero wedges in 20000 end-of-stream rounds against the 1-in-2000 baseline, zero in 5000 strict ping-pong rounds (500k round trips), full suite green at -t 1, -t 2, -t 4 and -t 8, Runic clean. No performance change. utils/gRPCClientUtils.jl benchmarks at -t 8, six samples per commit alternating between commits, per-message allocations identical on every workload: streaming_request 6.50 both, streaming_response 28.70 vs 28.68, streaming_bidirectional 26.55 vs 26.57, smol 82.4 vs 82.5. Throughput is within noise, bidirectional +0.2% and client streaming -1.4% on the mean against a main-only spread of 4.5% on that workload. Co-Authored-By: s-celles <s.celles@gmail.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
utils/gRPCClientUtils.jl pinned gRPCClient = "~1.0.0", which stopped resolving when the repository moved to 1.1.0-rc1, so the benchmark and stress tooling could not be instantiated against main at all. A plain "1" bound survives minor bumps. Separable from the streaming fix on this branch; drop or cherry-pick it as you prefer. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #129 +/- ##
==========================================
+ Coverage 92.09% 92.20% +0.11%
==========================================
Files 7 7
Lines 645 667 +22
==========================================
+ Hits 594 615 +21
- Misses 51 52 +1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
src/Streaming.jl was included only under `@static if VERSION >= v"1.12"`, with a load-time warning below that, so the streaming methods did not exist on Julia 1.10 and 1.11 and their testsets were skipped. Remove the guard, the matching guard around the streaming testsets, and the docs that describe streaming as 1.12 only. test/runtests.jl is almost entirely a de-indent: `git diff -w` shows the real change is the two removed guard lines plus the "Don't Stick User Tasks" edit below. "Don't Stick User Tasks" is now `broken = VERSION < v"1.12"`. It is a unary test that only happened to sit inside the guarded block, and its failure is not something this package can fix: arming the deadline watchdog with `Timer(cb, delay)` runs its callback loop in an `@async`, and on those versions scheduling a sticky task marks the scheduling task sticky too (JuliaLang/julia#41324, fixed by the 1.12 scheduler). Known gap, deliberately not hidden by this commit. On Julia 1.10 (libcurl 8.4.0) the suite is one test from green: "Bidirectional Streaming", the 1000 message stress, wedges and burns its full 300s deadline. That is the mid-stream wedge of issue #68 and it is pre-existing, not a consequence of the handoff fix in the previous commit: main's streaming code, with only this include guard removed, fails the same way at the same sizes and slightly more often. Small streams are reliable there, the failures start around 200 messages. 1.12 is unaffected and green at -t 1 and -t 8. Co-Authored-By: s-celles <s.celles@gmail.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
libcurl 8.4.0 and 8.5.0 stop servicing a transfer's receive direction while its send direction is paused. lib/http2.c drain_stream() sets dselect_bits to CSELECT_IN|CSELECT_OUT for an HTTP/2 stream whose upload is still open, and this package sends an empty Content-Length:, so CSELECT_OUT stays set for the life of the call. Curl_readwrite then consults select_bits_paused(), which in those versions ORs the two directions, so the KEEP_SEND_PAUSE that read_callback sets whenever the upload buffer runs dry makes it report "paused". Curl_readwrite returns CURLE_OK having read nothing and deliberately leaves the bits set, so every later curl_multi_socket_action re-takes the same early return: response bytes pile up unread in the socket receive queue and the caller blocks until its deadline. That early return arrived in 8.4.0 as the fix for curl/curl#11982 and over-corrected. Reported at https://curl.se/mail/lib-2024-01/0049.html and fixed in 8.6.0 by "transfer: make the select_bits_paused condition check both directions". Clearing KEEP_SEND_PAUSE is the only thing that makes the predicate false, which is why driving the multi handle does not help and curl_easy_pause(CURLPAUSE_CONT) does. Julia bundles libcurl as a stdlib so an affected version cannot be upgraded, and Julia 1.10 ships 8.4.0, so on those versions a request-streaming call now re-issues the un-pause every 50ms while it is in flight. The timer is armed only when the loaded libcurl is affected and only for request-streaming calls, and is closed in cleanup_request alongside the deadline watchdog. The gate resolves in __init__ from the running libcurl rather than from the CURL_VERSION const, which is evaluated during precompilation and baked into the cache. The two normally agree, but they diverge when a different libcurl is loaded under an existing cache, which is how this workaround gets tested. Measured against the Go test server, bidirectional, 200 messages of 1KB responses, burst-send then drain, 30 rounds, -t 4: Julia 1.10 / libcurl 8.4.0 30/30 wedged -> 0/30 Julia 1.12 / libcurl 8.15.0 0/30 -> 0/30, no timer armed full suite on 1.10 5m12s hang and one error -> passes in 12.2s full suite on 1.12 passes, unchanged Julia 1.11 (libcurl 8.6.0) is NOT fixed by this and is deliberately left ungated: it fails 63% of the same rounds with a different signature, stalling after every response has arrived rather than mid-stream, and forcing the workaround on there changes nothing (83% vs 80%). That is a separate libcurl bug fixed in 8.7.1. Co-Authored-By: s-celles <s.celles@gmail.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Collaborator
Author
|
@s-celles root cause is a bug that was introduced in libcurl 8.4.0 and fixed in 8.6.0. Julia 1.10 has libcurl 8.4.0. There is also a separate issue impacting 8.6.0, and then the issue impacting all versions caused by a race condition in our request streaming teardown. All three are fixed in this PR. |
The select_bits_paused early return does not exist before libcurl 8.4.0, so those versions never had the send-pause-blocks-receive bug and gain nothing from a 20Hz un-pause timer per in-flight streaming call. Gate on the closed range 8.4.0 <= libcurl < 8.6.0 instead of everything below 8.6.0. This is reachable in practice rather than hypothetical: Julia bundles libcurl, but several distributions build Julia against a system libcurl instead, which can pair a supported Julia with a pre-8.4 library. Verified rather than inferred from the source history. Julia 1.10 against libcurl 7.84.0 (taken from the Julia 1.8.5 distribution) with the workaround disabled, on the most sensitive shape, 2 messages with 256KB responses, 30 rounds: libcurl 8.4.0 29/30 wedged <- positive control, the detector works libcurl 7.84.0 0/30 The gate resolves correctly at every boundary: 7.84.0 off, 8.4.0 on, 8.5.0 on, 8.6.0 off, 8.7.1 off, 8.15.0 off. Full suite passes on Julia 1.10 with the bundled 8.4.0 (workaround armed, 12.1s), on Julia 1.10 with 7.84.0 (unarmed, 12.1s), and on Julia 1.12 (unarmed). Co-Authored-By: s-celles <s.celles@gmail.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
libcurl 8.6.0 strands a streaming call at the end of the stream: every response has arrived, but the call never completes and the watcher spins on the socket until the deadline. lib/http2.c drain_stream() sets data->state.select_bits to CURL_CSELECT_IN|CURL_CSELECT_OUT to mean "I still hold buffered input, run me and read it". multi_socket() then records the event the application reports: /* 8.6.0, lib/multi.c:3245 */ data->state.select_bits = (unsigned char)ev_bitmask; /* 8.7.0, lib/multi.c:3170 */ data->state.select_bits |= (unsigned char)ev_bitmask; By that point libcurl has already drained the socket, so the kernel receive queue is empty and the socket is only ever writable. The watcher reports CSELECT_OUT alone, and on 8.6.0 that plain assignment overwrites the CSELECT_IN libcurl staked for itself. Curl_readwrite runs the send half only, the buffered HTTP/2 input is never surfaced, and every later writable wake-up overwrites the bit again. Since a TCP socket with room in its send buffer is always writable, the watcher then spins on it at ~155k wake-ups/s. Fixed upstream in 8.7.0 by the |= above, "multi: fix multi_sock handling of select_bits", https://curl.se/bug/?i=12971. The workaround claims readability when the socket is writable and not readable, so the socket action ORs in the bit libcurl is waiting for rather than erasing it. It costs one extra recv attempt per writable wake-up on an affected libcurl, which returns EAGAIN when there is genuinely nothing to read. The affected range is 8.6.0 alone, closed at both ends and verified two ways. By source: 8.4.0 and 8.5.0 keep the drain signal in a separate field (state.dselect_bits, with the event going to conn->cselect_bits), so no assignment can clobber it, and 8.6.0 is the release that merged the two. By experiment, on Julia 1.11 with the shape that fails 30/30 on 8.6.0: libcurl 7.84.0, 8.4.0, 8.5.0 and 8.7.1 are each 0/30. The two libcurl workarounds this package now carries are mutually exclusive by construction: 8.4.0 and 8.5.0 get the un-pause pump, 8.6.0 gets this one, everything else gets neither. Measured against the Go test server, bidirectional, burst-send then drain: Julia 1.11 / 8.6.0, N=200 SZ=0 -t 1 30/30 wedged -> 0/30, and 0/100 on a longer soak Julia 1.11 / 8.6.0, N=200 SZ=0 -t 4 -> 0/30 Julia 1.11 / 8.6.0, N=200 SZ=1024 -t 4 21/30 wedged -> 0/30 Julia 1.12 / 8.15.0 0/50, gate off, code path not taken Full suite now passes on all three: Julia 1.10 12.1s, Julia 1.11 13.1s, Julia 1.12 13.2s. Co-Authored-By: s-celles <s.celles@gmail.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
csvance
force-pushed
the
fix-streaming-pause-handoff-race
branch
from
August 1, 2026 17:09
ea2eb14 to
c83b9e7
Compare
Contributor
|
Nice finding! Thanks @csvance |
Benchmarking main against this branch showed the only measurable cost of the fix was memory per request: sizeof(gRPCRequest) went from 192 to 208 bytes, about +39 bytes per message on workload_smol, from the two fields the fix added. `response_length::UInt32` needs 4-byte alignment, so the two Bools ahead of it already leave two padding bytes. Moving request_eof there costs nothing, taking the struct to 200 bytes. The remaining 8 is the `unpause` pointer field, which cannot be packed away. Both constructors use positional new(), so the value moves with the field. Full suite green on Julia 1.10, 1.11 and 1.12. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
csvance
marked this pull request as ready for review
August 1, 2026 17:40
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Streaming hangs below Julia 1.12: root cause is two libcurl bugs, not a Julia one
Closes #68.
Summary
src/Streaming.jlwas compiled out below Julia 1.12 because streaming calls hung there. Thecause is two separate libcurl bugs, each in the libcurl that a given Julia happens to
bundle, which is why the failure tracked the Julia version and looked like a Julia problem:
direction is paused. This is what hits Julia 1.10, and it is the bulk of this document
stream. This is what hits Julia 1.11, covered in its own section below
Verified by swapping only libcurl underneath an unchanged Julia 1.10 and an unchanged
checkout:
Bidirectional Streamingerrors, suite takes 5m12sLD_LIBRARY_PATHMechanism
lib/http2.c drain_stream()setsdselect_bits = CURL_CSELECT_IN | CURL_CSELECT_OUTforany HTTP/2 stream whose upload is still open. gRPCClient sends an empty
Content-Length:, soCSELECT_OUTstays set for the life of the call.read_callbackreturnsCURL_READFUNC_PAUSEwhenever the request buffer runs dry, whichfor a streaming call is every time the caller has nothing queued. That sets
KEEP_SEND_PAUSE.Curl_readwriteconsultsselect_bits_paused(), which in 8.4.0 and 8.5.0 ORs the twodirections:
With bits
IN|OUTand onlyKEEP_SEND_PAUSEset, the second clause is true, so thefunction reports "paused",
Curl_readwritereturnsCURLE_OKhaving read nothing, andit deliberately leaves the bits set. Every subsequent
curl_multi_socket_actionre-takesthe same early return, forever.
Response bytes then pile up unread in the kernel receive queue while the caller blocks until
its deadline. This matches the reported symptoms exactly:
curl_multi_socket_actionreturning
CURLM_OKwhile consuming nothing, tens of KB unread in the socket receive queue,and a socket watcher spinning at ~170k wake-ups per second on a
POLLINlibcurl refuses toconsume.
It also explains why only
curl_easy_pause(CURLPAUSE_CONT)recovers a wedged call.Clearing
KEEP_SEND_PAUSEis the only thing that makes the predicate false. A probe thatdelivers the identical wake-up without clearing the pause bit (
CURLPAUSE_ALLthenCURLPAUSE_SEND, which forcesEXPIRE_RUN_NOWand setsconn->cselect_bits = IN|OUT)recovers nothing: 0/13, versus 13/13 for a real un-pause.
Upstream
curl/curl#11982 and over-corrected
Unpaused connection, HTTP/2 only, with
CURL_READFUNC_PAUSEfollowed by an ineffectiveCURLPAUSE_SEND_CONTdirections", listed in the 8.6.0 changelog, which now
returns false as soon as any wanted direction is unpaused
Which versions are affected
The affected range is closed at both ends,
>= 8.4.0and< 8.6.0: the early return doesnot exist before 8.4.0, so those versions never had the bug.
Julia held at 1.10.11, libcurl swapped by
LD_LIBRARY_PATH(LibCURL_jlldlopenslibcurl.so.4by soname, so no artifact override is needed). Bidirectional, 200 messages,40 rounds:
The lower bound is verified, not just inferred from the source history. On the most
sensitive shape (2 messages, 256 KB responses) with the workaround disabled, libcurl 7.84.0
is 0/30 where 8.4.0 is 29/30. It matters because a Julia built against a system
libcurl, as several distributions do, can pair a supported Julia with a pre-8.4 library, and
that combination needs no workaround.
Cross-check in the other direction: Julia 1.12 forced onto 8.4.0 wedges at 32.5%, and Julia
1.10 on 8.15.0 is 0/200. The wedge also reproduces at
-t 1, so it is not a schedulerissue.
The shape that matters is not message count but whether the caller stops sending while
responses are still outstanding. Burst-then-drain wedges; ping-pong and windowed sending
never do, because every new send issues an un-pause that incidentally clears the block;
server streaming, which never pauses the upload, is clean.
The fix in this PR
A request-streaming call re-issues
curl_easy_pause(CURLPAUSE_CONT)every 50 ms while it isin flight, which is the only thing that clears the block. The timer is armed only when
the loaded libcurl is affected and only for request-streaming calls, and is closed in
cleanup_requestalongside the deadline watchdog. On a healthy call an un-pause with anempty request buffer just draws one
read_callbackthat pauses straight back.The gate is
8.4.0 <= libcurl < 8.6.0, verified to resolve correctly at each boundary(7.84.0 off, 8.4.0 on, 8.5.0 on, 8.6.0 off, 8.7.1 off, 8.15.0 off). It resolves in
__init__from the running libcurl rather than from the
CURL_VERSIONconst, because that const isevaluated during precompilation and baked into the cache. The
two normally agree, since libcurl is bundled, but they diverge when a different libcurl is
loaded under an existing cache, which is exactly how this workaround gets tested.
Bidirectional, 200 messages of 1 KB responses, burst then drain, 30 rounds,
-t 4:Also in this PR: a separate end-of-stream race, fixed properly
Independent of the libcurl bug, the request pump un-paused libcurl and only then closed
the request stream. A
read_callbacklanding in between saw a stream that was still open,paused again, and left the transfer paused with nobody to resume it. Reproduced at about
1 in 2000 rounds on Julia 1.12, and the wedged state was unrecoverable even by an un-pause,
because
read_callbackreturnedCURL_READFUNC_PAUSEon a setcurl_done_readingbeforeit ever checked whether the stream had ended.
Fixed by making the handoff a single critical section under the lock
read_callbackalreadyruns under, so no callback can interleave with it. 0 wedges in 44000 rounds after the fix.
This one needs no timer and no version gate.
A second libcurl bug: 8.6.0 clobbers a transfer's pending read interest
Julia 1.11 ships libcurl 8.6.0, which has the fix above but a different bug, gone in
8.7.0. It strands a call at the end of the stream: every response has arrived, but the
call never completes.
lib/http2.c drain_stream()setsdata->state.select_bits = CSELECT_IN|CSELECT_OUTto mean"I still hold buffered input, run me and read it".
multi_socket()then records the eventthe application reports:
By then libcurl has already drained the socket, so the kernel receive queue is empty and the
socket is only ever writable. The watcher reports
CSELECT_OUTalone, and that plainassignment overwrites the
CSELECT_INlibcurl staked for itself.Curl_readwriterunsthe send half only, the buffered input is never surfaced, and every later writable wake-up
overwrites the bit again. A TCP socket with room in its send buffer is always writable, so
the watcher then spins on it at ~155k wake-ups/s.
Fixed upstream in 8.7.0: "multi: fix multi_sock handling of select_bits",
curl.se/bug/?i=12971.
It is the exact mirror image of the first bug. Probe ladder on 8.6.0, Julia 1.11, single-step
ladders, 20 rounds each:
curl_multi_socket_action(sock, CURL_CSELECT_IN)curl_easy_pause(CURLPAUSE_CONT)curl_multi_socket_action(CURL_SOCKET_TIMEOUT, 0)curl_easy_pause(CURLPAUSE_SEND)CURLPAUSE_ALLthenSEND, forcingEXPIRE_RUN_NOWThe fix
The watcher claims readability when the socket is writable and not readable, so the socket
action ORs in the bit libcurl is waiting for rather than erasing it. Costs one extra recv
attempt per writable wake-up on an affected libcurl, which returns
EAGAINwhen there isgenuinely nothing to read.
The affected range is 8.6.0 alone, closed at both ends and verified two ways. By source:
8.4.0 and 8.5.0 keep the drain signal in a separate field (
state.dselect_bits, with theevent going to
conn->cselect_bits), so no assignment can clobber it, and 8.6.0 is therelease that merged the two. By experiment, on the shape that fails 30/30 on 8.6.0: 7.84.0,
8.4.0, 8.5.0 and 8.7.1 are each 0/30. The two workarounds are mutually exclusive by
construction: 8.4.0 and 8.5.0 get the un-pause pump, 8.6.0 gets this one, everything else
gets neither.
N=200 SZ=0 -t 1N=200 SZ=0 -t 4N=200 SZ=1024 -t 4Result
Streaming now works on every supported Julia:
What was ruled out
un-pause the same watcher's socket actions immediately deliver every pending byte with no
re-registration, and no
CURL_POLL_REMOVEoccurs anywhere near the wedgesocket_callback. The wedge holds with exactly one stable watcherand zero
socket_callbackinvocations during it, and it vanishes by swapping onlylibcurl while
FileWatchingis byte-identical. It does explain the CPU burn, sinceuv_pollis level-triggered and libcurl refuses to clearPOLLIN, which is worth fixingseparately as robustness
-t 1, flat rates across 1/2/4/8/16threads, and the version matrix above isolates libcurl as the deciding variable