Add streaming-loadtest ledger backend for apply-load replay - #679
Open
aditya1702 wants to merge 102 commits into
Open
Add streaming-loadtest ledger backend for apply-load replay#679aditya1702 wants to merge 102 commits into
aditya1702 wants to merge 102 commits into
Conversation
A concurrent protocol-migrate engine snapshots membership from committed protocol_contracts rows only, so when it wins a ledger's cursor swap it has folded that ledger without contracts whose classification commits with live's in-flight transaction — their deploy-ledger state (constructor ContractData writes, first events) was extracted by nobody, and cursor-passed ledgers are never replayed. Never-rewritten keys stayed missing forever. On a lost swap where this ledger classified new contracts for the protocol and the lost cursor's committed value is at or past this ledger (a lost CAS blocked on the winner's row lock, so that value is reliably visible in-transaction via the new IngestStore.GetInTx), live now re-stages scoped to exactly the gap contracts and persists the lost halves in the same transaction, without moving any cursor. The CAS-winning path is extracted into stageAndPersistProtocolLedger alongside the new repairClassificationGap. Covered by CAS-gating cases M1-M4: frontier repair, behind-tip non-repair, partial-loss scoped repair, and already-committed exclusion.
…r is fetched The engine refreshed a tracker's classified-contract membership right after each window commit — milliseconds before a concurrent live transaction for that same contested ledger finishes committing a new contract's classification, since live's lost CAS was blocked behind the engine's row lock and still has work to do after unblocking. The engine then staged the next window with a snapshot missing that contract, skipping its events and entries for one more ledger; additive fold columns (cost basis, lifetime claimed totals) never heal from a missed ledger, so those deltas were permanently lost whenever the engine also won that next ledger. The refresh now runs at window start, after GetLedger returns for the window's first ledger: the fetch blocks until that ledger has closed, which is a full ledger interval after any concurrent transaction for the previous one committed. Same cadence — once per window per requiring tracker — with the read taken at the latest useful moment. The refresh test now pins the ordering: the first folded ledger must already carry membership committed after the run-start snapshot (mutation-verified — removing the window-start refresh fails it).
…sons Blend state changes follow the core convention: category names the on-chain object (BLEND_SUPPLY, BLEND_COLLATERAL, BLEND_DEBT, BLEND_AUCTION, BLEND_EMISSIONS, BLEND_BACKSTOP_EMISSIONS, BLEND_BACKSTOP, BLEND_BACKSTOP_QUEUE) and reason names the action. Amount-bearing categories reuse the generic CREDIT/DEBIT/ADD/REMOVE/ BURN verbs; only BORROW, REPAY, FLASH_LOAN, BAD_DEBT, FILL, and CLAIM are added.
…keys Postgres UPDATE ... FROM applies only one matching source row per target row, so duplicate (pool, user, asset) keys in a batch silently dropped deltas. Net deltas reject duplicates (ZeroBorrowed makes merging order-dependent; the processor pre-aggregates); auction adjustments are purely additive and are summed server-side before applying.
…odels Add blend_pool_claimed (pool, user) and blend_backstop_claimed (user) tables plus PoolClaimedModel/BackstopClaimedModel with additive BatchApplyDeltas. These hold lifetime claimed BLND / Comet LP totals, folded from claim events during current-state indexing — the only pass that sees every claim since Blend's first ledger. Mirrors the net_supplied/net_borrowed cost-basis accumulator.
…account_id Both tables are read by GetByAccount (the per-user positions path) filtering on user_account_id, which is the second PK column and so cannot use the primary key. Add single-column B-tree indexes mirroring idx_blend_positions_user, keeping the index defined alongside its table in the same migration.
ApplyAuctionAdjustments converted protocol tokens to underlying with exact numeric division, leaving a fractional tail (e.g. "1100.0000000000000000") in net_supplied/net_borrowed while every other write to those columns stores floored integer text. The contract uses fixed_mul_floor (floor of the positive magnitude). Wrap the conversion in trunc(): for the signed lot/bid deltas, trunc toward zero reproduces floor-of-magnitude-with-sign (trunc(-366.3) = -366, not floor's -367). Adds a subtest covering a fractional product on both the positive (filler) and negative (liquidated user) sides.
… writers BatchApplyNetDeltas and ApplyAuctionAdjustments mutate existing rows only; a delta for a not-yet-inserted position row silently no-ops. Document that callers must upsert the Positions snapshot (and reserves) first, as PersistCurrentState does.
The full-snapshot upsert/zero writers overwrote last_modified_ledger outright while the additive writers (net deltas, claimed, reserve data, reward zone) already use GREATEST. Switch the snapshot writers to GREATEST(<table>.last_modified_ledger, EXCLUDED/u.ledger) too, so the column never moves backward and every writer treats it uniformly. Behavior is unchanged under the strictly ledger-ordered persist path.
StartBlock and LastModifiedLedger were int32 while every other blend row struct uses uint32 for ledger-valued fields, casting to int32 only at the write boundary. Align Auction with that convention.
…ments Flooring doesn't distribute over addition, so n fills folded into one (pool, user, asset) row overstate magnitude by at most n-1 stroops versus per-fill flooring. The existing duplicate-key test uses amounts exact at the fixture rates, so it cannot tell the two orders apart; this one can (two fills of 5 at b_rate 1.1: 11 aggregated vs 10 per-fill). The doc on applyAuctionAdjustmentsSQL states why the single trunc on the summed delta is kept: the skew is strictly below the end-of-window rate approximation already accepted on the same display-only fields.
The Down re-adds the narrowed CHECKs as NOT VALID, which is precisely the modifier that skips checking existing rows — so the stated precondition (no BLEND_* rows) was a comment, not a guarantee, and a rollback over Blend data left rows the restored constraints forbid: unwritable, and fatal to whole stateChanges pages in the pre-Blend resolver. A DO block now probes both the category and reason lists (bloom sparse indexes serve both) and raises before touching any constraint. All four DROP CONSTRAINTs gain IF EXISTS so both directions are re-runnable. The Up keeps NOT VALID: validating would scan every columnstore chunk under an AccessExclusiveLock.
…claim token Verified against blend-contracts-v2 @ ba22b487: fill_auction (pool/src/auctions/*.rs): fill_bad_debt_auction moves the bid dTokens from the backstop's Positions to the FILLER's — the filler assumes the debt — while the lot (backstop LP tokens) is drawn straight to the filler's wallet and never touches pool Positions. fill_interest_auction settles entirely outside pool Positions (bid donated to the backstop, lot paid from the reserves' backstop_credit, captured by the ResData entry snapshot). The decoder previously folded the user side of every asset for all auction types and mirrored to the filler only for type 0: a type-1 filler's net_borrowed missed the assumed debt, and type-2 fills fabricated lot adjustments against the backstop-address row (in the wrong units — underlying, valued as bTokens). Folds now mirror the on-chain Positions moves exactly: type 0 both sides, type 1 bid-only both sides, type 2 none. backstop claim (backstop/src/contract.rs -> emissions/claim.rs): the event's amount is execute_claim's return — the Comet LP tokens minted and auto-deposited (per-pool deposit events are emitted alongside) — never raw BLND. The row's token_id is now NULL with units backstop_lp in key_value, matching every other backstop-LP-denominated row, instead of mislabeling LP amounts as BLND.
Decode a ClaimFold from each pool/backstop claim event and accumulate it into the staged pool/backstop claimed-total maps during current-state indexing, persisting via PoolClaimed/BackstopClaimed.BatchApplyDeltas. History mode is unchanged — it still records the CLAIM feed rows but folds no totals.
Blend pool/positions/earn-options query builders and client methods, plus client-side DTOs and inline fragments for the eight concrete Blend state-change types, registered through the generic unmarshal dispatch and the schema-validation test.
…or tables Read blend_pool_claimed/blend_backstop_claimed via the new GetByAccount readers after phase-2 live ingestion and assert the supplier's pool claim and whale's backstop claim folded positive totals. Guard that phase-1 (no claims) leaves both accumulator tables empty.
…s survive The canonical-backstop pin resolves to empty on the standalone network, which made the processor drop every backstop-shaped entry and event as a non-canonical impostor — BlendMigrationTestSuite failed on the whale's missing blend_backstop_positions row. The suite deploys the backstop from the master account (keypair.Root of the passphrase) with a fixed salt, so its address is a deterministic function of the passphrase alone. Pin it in canonicalBackstopAddress the same way blndTokenAddress pins the standalone BLND SAC: unit test + a deploy-time assertion in SetupBlendStack.
Absorbs the review-round API changes on blend/pr5-graphql: drops the GetBlendEarnOptions client method, query, and types (the query was removed - earn discovery composes from blendPools); BlendPool.status becomes the BlendPoolStatus enum string; BlendReservePosition's single emissionsApr splits into emissionsSupplyApr/emissionsBorrowApr. Claimable-emission assertions were already sign-based, so the projected-to-now claimable math needs no test changes.
executeSorobanOperationAs was a near-verbatim copy of executeSorobanOperation that fixed two real defects only in the copy: it preserved simulation-assigned auth nonces (nonce 0 is one-shot per address) and re-simulated after signing so MinResourceFee reflects the signed entries' size. One executeSorobanOperation(op, source, extraSigners, retries) now carries both behaviors for every caller. Sequence resolution branches on the source: the master account drives SharedContainers' locally tracked counter, any other actor's sequence is fetched from RPC. Since every master submission now advances the local counter, the As-path drift SyncMasterSequence existed to repair can no longer occur, so it and its re-sync call sites are removed. Auth entries an available keypair cannot sign now fail loudly naming the required address (the old shared path silently left them unsigned); source-account-credentialed entries pass through unchanged as before.
…mit for them The wbclient full-detail account-history query measures 10,101 at first:100 — over the previous 10,000 default — because the shared stateChangeFragments const gained 8 Blend fragments (+27 fields per state-change node, ×100 edges). The regression test could not catch this: it asserted a hand-copied mirror of the SDK query that had drifted (no Blend fragments, aliases the builder never emits), and the integration container overrode the limit to 30,000. pkg/wbclient now exports Queries(), the exact documents the client sends, and both server-side guards consume it: schema validation (which previously missed the three Blend queries) and the complexity regression test, which prices every SDK query at the largest accepted page size against the flag's own FlagDefault. Measured: BlendPools=26,150, full-detail history=10,101, state-change queries=8,401, blendPositions=7,583. The default limit rises to 30,000, documented as sized for the SDK's heaviest shipped queries with the DoS tradeoff stated (deployments not serving Blend should lower it). The integration container no longer overrides GRAPHQL_COMPLEXITY_LIMIT, so the suite proves the shipped default serves the SDK's full-selection queries end-to-end.
A dev-only LEDGER_BACKEND_TYPE=streaming-loadtest that reads stream-framed LedgerCloseMeta from named pipes written by stellar-core apply-load (one FIFO per transaction profile), renumbers each stream onto the consumer's requested sequence with per-pipe diffs, merges the per-sequence frames into one mixed-traffic ledger via the SDK's loadtest.MergeLedgers, and stamps monotone wall-clock close times (apply-load emits closeTime 0). Renumbering makes both sides restartable without a database reset: a restarted apply-load resets to raw sequence 1 and is mapped onto the next requested ledger; a restarted consumer resumes from its cursor. Pacing via --loadtest-ledger-close-duration bounds the ledger rate, and FIFO backpressure throttles the generators to match. Because apply-load's benchmark mode publishes no history archive, this backend type skips the archive connection and the cursor-0 checkpoint bootstrap: ingestion starts from ledger 1 on an empty database and balance state accumulates from the ledger stream. Everything downstream of the backend (live ingest loop, processors, persistence) is unchanged, so a load test exercises the same code path as production ingestion. Includes an opt-in corpus test (STREAMING_LOADTEST_CORPUS) that replays real apply-load output through the backend and the production transaction reader; verified against v27 sac/custom_token/soroswap corpora. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
aditya1702
force-pushed
the
replay-loadtest-backend-pr6
branch
from
August 7, 2026 18:25
8bf66b4 to
d5ec0ae
Compare
Renumber only the ledger header. The per-entry rewrite of lastModifiedLedgerSeq/TTL/seqLedger was a full XDR marshal/parse/walk/re-marshal round trip over every frame, and nothing in wallet-backend reads those fields — measured at ~5x the entire ledger-processing cost at full per-ledger volume (the stream's cadence was bound by it), and a 37x speedup on the corpus replay test once removed. Merging now appends transaction-set phases and results directly instead of going through the SDK's renumbering merge, which also drops the goxdr dependency. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…data Two fixes for protocol-framework behavior surfaced by the loadtest rig: Protocol state production (SEP-41 balances/state changes) is gated by per-protocol cursor rows that only the protocol-migrate CLI creates, and that CLI needs a replayable ledger source. A streaming-loadtest deployment has none, so a fresh database silently skipped every protocol processor. The archive-less bootstrap now seeds each registered protocol's cursors at startLedger-1 — exactly the value the first ledger's compare-and-swap expects — and refreshes the cursor snapshot so production is live from ledger 1. The SEP-41 metadata fetcher re-fetched failing contracts on every ledger they were active in: claimed contracts re-enter Prefetch each classification pass, and a persistently failing name() simulation costs its full retry-with-backoff (600ms of sleep) every time. A negative cache now skips recently failed contracts for 5 minutes, keeping eventual enrichment while removing the per-ledger tax. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Prefetch runs for every claimed contract observed in a ledger's changes and has no database access by design, so it re-simulated name/symbol/decimals for tokens whose metadata was already persisted — on every ledger that touched their instance. An in-process success cache skips them; a restart refetches each contract once, which doubles as the refresh path for tokens whose on-chain metadata changed. Persist retries are unaffected: the fetched values live in the classification plan, which is reused across retry attempts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Each pipe's reader now buffers two decoded frames beyond the one in flight, so a writer streams its next frames through the FIFO while the consumer merges and processes earlier ledgers. With lockstep delivery every GetLedger waited on the slowest writer's in-flight frame — a constant ~0.6s/ledger at full volume that the one-frame handoff could not hide. Backpressure still bounds the writers, now with three frames of slack; an epoch's terminating error stays ordered behind its frames, so restart handling is unchanged.
…allel apply-load's meta write is synchronous: core does not start generating its next ledger until the consumer drains the current frame, and the reader drained only as fast as it decoded — putting XDR decode inside every producer's ledger cycle (measured: cycle = generation + decode, which capped the merged stream well under target cadence). Each pipe's reader now slurps a record's raw bytes at transfer speed and hands them to a per-pipe decode goroutine, so the producer starts its next ledger while the previous frame decodes. Frame and error ordering are preserved: a drain or decode error is always the last element delivered and ends the epoch exactly as before.
The streaming-loadtest backend's meta sources generalize beyond FIFOs: a tcp-listen://HOST:PORT entry binds a listener eagerly at construction and serves each producer connection as one stream epoch, so apply-load producers can run in their own pods and dial in. Connection close is the epoch boundary (as FIFO EOF is), the listener outlives epochs to serve producer restarts, and keepalive surfaces a vanished peer as a read error. Frame draining, decoding, renumbering, and merging are shared between both source kinds; entry order still defines merge order. The flag is now --loadtest-meta-sources / LOADTEST_META_SOURCES.
Each buffered lookahead frame is a fully decoded LedgerCloseMeta — tens of MB of pointer-dense XDR per source, ~600MB resident across 12 sources per lookahead unit — and that mass is scanned by every GC cycle. One frame of lookahead still overlaps the writer's streaming with the consumer's processing; the second bought no cadence and cost scan time.
The metadata fetcher's fetched/failure caches are in-memory only, so a token whose contract_tokens row already carries metadata was still re-fetched over RPC — once per process lifetime after every restart on a real network, and forever on a 5-minute backoff against a deployment whose RPC can never resolve metadata (the loadtest rig's dead endpoint with externally seeded rows), where each retry burned the ~850ms simulate backoff ladder inside prepare_classification. After Apply persists a batch's rows, contracts whose row has metadata are marked fetched, making the database the durable cache. Tokens whose metadata is genuinely missing stay unmarked and keep the existing backoff retry.
This was referenced Aug 13, 2026
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Start with
streaming_loadtest_ledger_backend.go(the backend commit), then the TCP-producers commit. Review time: ~40 min.Dev-only ledger backend that replays
stellar-core apply-loadmeta straight into live ingestion — the real ingest loop at rates no test network can produce. Only the ledger backend is swapped; processors and persistence are untouched, so what it measures predicts production.Based on
blend/pr6-integration-tests(what dev runs today); re-targets tomainwhen pr6 lands.What's in it
LEDGER_BACKEND_TYPE=streaming-loadtest: merges framed meta from N producer sources into one ledger sequence, renumbers so restarts never need a DB reset, stamps advancing close times, paces via--loadtest-ledger-close-duration. No history archive — starts from ledger 1 on an empty DB.--loadtest-meta-sourcestakes FIFO paths and/ortcp-listen://HOST:PORT, so producers run in their own pods and dial in. Connection close = epoch boundary (same as FIFO EOF); the listener outlives epochs so producers can restart.Review path
streaming_loadtest_ledger_backend.go— merge, renumber, pacing (the bulk)Testing
Unit tests: renumbering, multi-source merge, generator restart, truncated frames, pacing, TCP epoch lifetime —
-raceclean. Opt-in corpus test (STREAMING_LOADTEST_CORPUS) replays real v27 apply-load output: 200 merged ledgers, ~57k txs.Deploy note
The rig env must rename
LOADTEST_META_PIPE_PATHS→LOADTEST_META_SOURCESwith this image. The producer side (dialer shim, StatefulSets, jitter) lives in the kube repo.First of four PRs from the live-ingest loadtest campaign (replaces #682). Review order: this → #684 → #685; #686 is independent. Combined rig result: 18,458 tx/s, process p99 0.992 s (window p90 0.768 / p99 0.995); PRs not separately re-benchmarked.