Stream anonymizedBallots from a DB cursor in random order#1424
Stream anonymizedBallots from a DB cursor in random order#1424jacksonloper wants to merge 1 commit into
Conversation
Previously the endpoint loaded every ballot into memory, filtered, shuffled, and serialized the whole array — holding the dataset ~3x at peak, which OOMs the pod on large elections. Now the ballots are streamed row-by-row from a server-side Postgres cursor (kysely .stream(), enabled via pg-cursor on the dialect) and serialized incrementally to the response through stream.pipeline, so peak memory is O(batch) instead of O(ballots) and backpressure from a slow client throttles the cursor. The in-memory secureShuffle is replaced by ORDER BY gen_random_uuid() in the query, so the randomization that hides ballot submission order now happens in Postgres (gen_random_uuid() is backed by a strong RNG) before rows ever reach Node. Also filters status='submitted' in SQL, makes the mock ballot store mirror the real store's head=true insert behavior, and types AnonymizedBallot.precinct as string|null to match what the API has always returned. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
✅ Deploy Preview for bettervoting ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
Warning Review limit reached
Next review available in: 47 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (8)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
1 participant
Problem
GET /API/Election/:id/anonymizedBallotsloads every ballot into memory (selectAll()), then.filter().map(), thensecureShuffle, thenres.json— the full dataset is held roughly 3x at peak. For large elections (e.g. ~31k ballots ≈ 14 MB of JSON, ~30–50 MB of heap to build) this is one of the endpoints that OOMKills the pod and 502s users. Worse, V8 never returns the heap high-water mark to the OS, so one big request permanently inflates the pod's baseline.Change
Stream the ballots row-by-row from a server-side Postgres cursor straight to the response, so peak memory is O(batch) instead of O(ballots):
ServiceLocator: passpg-cursorto the KyselyPostgresDialect(this is what enables kysely's.stream(); no behavior change for existing queries).BallotsDB.streamSubmittedBallotsByElectionID: new model method — cursor-backed (.stream(500)), filtershead = true AND status = 'submitted'in SQL.gen_random_uuid(), which is backed by Postgres's strong RNG (pg_strong_random), so this keeps a CSPRNG-grade shuffle — it just happens in Postgres memory instead of Node.secureShuffleto the database RNG.{"ballots":[...]}incrementally via an async generator (batched into ~64 KB chunks) piped to the response withstream.pipeline, which handles backpressure (a slow client throttles the cursor rather than ballooning buffers) and destroys the cursor if the client disconnects mid-stream.AnonymizedBallot.precincttypedstring | nullto match what the API has always returned ("precinct": null).head = trueon insert like the real store, and implements the new stream method.Response shape
Unchanged:
{"ballots": [{ballot_id, election_id, precinct, votes}, ...]}. The response is chunked (noContent-Length), which clients don't observe throughfetch/axios.Error semantics tradeoff
Since streaming sends the
200and body incrementally, a mid-stream DB failure can no longer produce a 500 — the connection is aborted instead (client sees a network error rather than truncated-but-parseable JSON). Auth/permission errors are unaffected (checked before any bytes are written).Testing
anonymizedBallots.test.ts: returns all submitted ballots anonymized (exact key set per ballot), empty election, and 401 for non-public open elections — 139 backend tests pass.npm run buildpasses for shared, backend, and frontend.Relationship to other work
Independent of #1423 (response compression) — they compose (compression middleware compresses streamed chunks as they flow) and any merge conflict is trivial. The new cursor plumbing (
pg-cursordialect option + the model stream method) is also the foundation the planned compact tabulation loading (docs/ram-reduction.md §2.7) can reuse.🤖 Generated with Claude Code