snapshot: decode chunks in parallel out of a pull-wide buffer pool - #12
Merged
Conversation
Push writes every chunk as its own zstd frame, but the reader concatenated
them and fed one stream decoder, so SNAPSHOT_TRANSFER_CONCURRENCY only
parallelised downloads. Decoding inside the fetch goroutines measured
7.7x on the decode step alone (644 -> 4952 MiB/s over 8 cores), and turned
compression from a net loss into a net win: on internal-cocoon-node-7 a
14 GiB snapshot pull went 59.9s -> 32.5s, and the compressed pull now
beats the uncompressed one despite moving 2.24x fewer bytes.
Parallel decode has to materialise whole decompressed chunks, which the
serial decoder never did, so buffers now come from two bounded pools and
the window is charged for both halves a decoding slot holds at once. Two
sizing bugs were worth fixing on the way:
- fetchChunk read into a bytes.Buffer, whose 512-byte grow step
overshoots a pre-sized capacity near EOF and then reallocates and
copies the whole chunk. That was 8% of pull CPU. It now reads an
exact-size slice.
- The pools were per file, so two large files in one snapshot could hold
two sets at once — 7.2 GiB peak against a 2 GiB budget. They are now
sized once against the whole plan, which is why writeImportTar is
split into planLayers -> newChunkPipeline -> stream: the budget belongs
to the pull, and per-file sizing cannot see the other files.
Sharing the pools also made it faster, not just smaller: a fresh 256 MiB
buffer costs a page fault and a kernel-zeroed page per page on first
touch, and a pull touches 14 GiB of them. Same window, 42.6s -> 36.2s.
Peak RSS for one pull ends up at 3.74 GiB against the pre-change 3.65 GiB,
i.e. the same memory for 38% less time. SNAPSHOT_PULL_BUDGET_MIB (vk side)
trades the rest: 3584 gives window 7 and 32.5s for 4.90 GiB.
Type blocks atomic (chunkPipeline methods rejoin their type); utility funcs sink below the method-bearing types.
CMGS
force-pushed
the
feat/parallel-chunk-decode
branch
from
July 26, 2026 08:43
65aae77 to
b3e1dac
Compare
fetch's size guard and fileWindow's per-chunk cap loop are unreachable: validateSnapshotLayers rejects negative sizes before any fetch, and newChunkPipeline zeroes the window when any chunk exceeds the buffering cap. window >= 2 at every newChunkSource call site, so the max(...) floor was dead.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Push stores every chunk as an independent zstd frame. This PR decodes those frames in the fetch workers, preserves chunk order through futures, and shares one bounded buffer pipeline across the whole pull.
The pull path now:
outputCap + window × (inputCap + outputCap)fits the configured prefetch-buffer budget;DecodeAllto the planned output capacity, so malformed compressed content cannot allocate past the slot;The independent maxima and the extra current output buffer are intentional. Files with different compression ratios can make the largest input and output occur in different entries, while the reader holds one output buffer as the next window is fetched.
Performance evidence
Earlier profiling on the target test node motivated the change: parallel stateless decoding increased the isolated decode step from 644 to 4952 MiB/s, and the shared-pool prototype reduced a 14 GiB pull from 58.2s to 36.2s. Review hardening changed the final budget formula and can reduce the allowed window, so those measurements are historical prototype evidence, not current-HEAD runtime acceptance. Current-head performance should be refreshed on the testbed before making a release claim.
Verified
make fmt-checkmake test(go veton linux and darwin;go test -race ./...)make lint(linux and darwin, zero issues)GOOS=darwin asl ./...GOOS=linux asl ./...Regression coverage includes byte-identical v1/v2 reconstruction, pull-wide budget boundaries, cross-file input/output maxima, per-file window narrowing, decoded-size rejection, exact blob-length enforcement, transport-error propagation, and bounded prefetch starts.
All explanatory comments introduced by this PR were removed during review; the code relies on names and regression tests for the contract.