feat(salvage): total recovery for SSTs and blob files with in-place ECC autoheal

[polaz]

Hardens recovery, salvage, and manifest repair into a total, deterministic
pipeline that always yields a valid, openable tree, for both SST tables and
vlog blob files.

SST salvage

• Block-level SST salvage recovers readable data blocks and drops corrupt
  ones, re-emitting the survivors into a fresh, verifiable SST under the
  tree's comparator, encryption, and dictionary context.
• A fast surface copies clean blocks through verbatim (no decode/re-encode)
  and heals single-block ECC in place; the faithful surface decodes and
  rewrites when a block cannot be copied through.
• Salvage fails closed on a table it cannot re-emit faithfully (range
  tombstones), preserving correctness over a lossy rewrite.
• The re-emit's ordering guard accepts the one legitimate seqno TIE: a write
  batch may add several merge operands for a key, and they share the batch's
  seqno, so a flush stores them all at it. Rejecting that would drop an
  otherwise clean block and change the value the repaired tree merges to. The
  tie is allowed in-block and across a block edge, and only between operands —
  for other kinds the internal key ties with no tie-breaker, so their order
  would not be reproducible.
• A key's versions are contiguous and can straddle a block boundary, so the
  walk suppresses the boundary key from the block it emits after a drop: a key
  whose newest version died with the dropped block is dropped outright rather
  than rolled back to the older version that opened the next block (which
  would resurrect a deleted key or shadow the write that replaced it). The
  suppression holds until a surviving entry with a DIFFERENT key proves the
  version run ended, so a chain spanning several blocks cannot re-surface in
  the second one; it applies on every emit path (row, columnar, and the
  delete-masked columnar one); a filtered block is re-emitted row by row
  rather than byte-copied, since its bytes still carry the suppressed key; and
  losing a blob record takes the rest of that key's chain with it, because the
  record it lost was the chain's head. A lost region recorded before the walk
  (a resynced-past gap) arms the suppression by OFFSET, against the block that
  actually follows it. What the suppression is armed WITH is the lost block's
  own separator, and its absence — a gap-probed region, or a block whose
  separator did not survive — arms the unknown boundary that suppresses the
  following block's first key instead. Key identity here is the table's
  comparator, not the bytes: under a comparator that folds spellings together,
  an older version spelled differently from the lost newest one is still the
  same key and is suppressed with it.

Blob-file salvage (KV separation)

• salvage_blob_file gives vlog blob files the same block-granular treatment
  as SSTs: it walks the file record by record, keeps every intact record, and
  drops only the damaged ones instead of condemning the whole file.
• BlobSalvageReport returns the dropped records (Vec<DroppedBlob>) and an
  offset_remap mapping every salvaged record's source offset to its
  relocation in the rewritten file — new offset AND new on-disk size (the
  re-emit re-compresses, and a live read cross-checks the handle's size
  against the frame header) — so the SST value handles that point into the
  blob file can be rewritten rather than invalidated.
• A truncated tail is never partially reconstructed: a record whose header or
  payload does not fully land is dropped, because keeping part of it would
  fabricate a value and a bogus remap entry.
• Manifest repair is blob-aware end to end: a table whose referenced blob file
  is unrecoverable is excluded (and its file removed once the manifest is
  durable) rather than wired into a
  rebuilt manifest as a dangling reference, and blob frontiers survive across
  recovery so a reclaimed blob file verifies against its live suffix.
• When the manifest itself is lost, a punched blob file's frontier is
  re-derived from the punch geometry: the zeroed data-section prefix is
  anchored at a validly decoding frame (a zero-filled value payload can never
  move the frontier; a partially completed punch is walked hole by hole), the
  digest covers the live suffix from the derived frontier, and the rebuilt
  snapshot re-persists the restriction — so a later relocation resumes exactly
  where the punch stopped instead of erroring inside the zeroed prefix. An
  unpunched file short-circuits on its first non-zero data byte at zero extra
  read cost. Zeros through the whole data section — with nothing live anchored
  below them, since reclaim punches top-down — mean the relocation completed
  and only the file's removal lagged the crash: repair completes that drop
  rather than publishing an empty-suffix handle whose whole-file
  metadata blob GC could never retire; if the removal fails, the repair
  fails with that error instead of committing a manifest (left in blobs/,
  the next open's orphan sweep would hit the same failure — success would
  describe a tree that cannot open). The same rule covers leftover salvage
  temps. A partially punched file recovers
  with its garbage accounting SEEDED from the validation scan (whole-file
  metadata minus live-suffix totals), so the consumed prefix — which no
  future compaction can observe — still counts toward is_dead and the
  file remains retirable once its suffix handles are gone. See
  docs/manifest-recovery.md § Blob frontier resolution.
• Every in-place reclaim now fails closed on a shared inode AND stands down
  while a checkpoint's deletion pause is active: both the SST and the blob
  tight-space prefix punches probe the link count and the pause before
  punching, so a checkpoint's hard-linked copy can never be zeroed (MemFs
  answers the probe instead of inheriting the Unsupported default, which had
  silently disabled the blob punch too). Standing down does not DISCARD the
  reclaim: the intent lives in the view that is dropping, so it is handed to
  the pause, which re-probes the link count and punches once the checkpoint's
  window closes — otherwise the space would be stranded permanently, in
  exactly the tight-space situation that needs it. A reclaim that still cannot
  be proven safe THEN (the checkpoint's link is still there, or the probe
  cannot answer) is RETAINED rather than dropped: unlinking the checkpoint only
  decrements the link count while the live restricted table keeps holding the
  inode, so nothing else would ever free the consumed prefix. A punch that FAILS
  mid-pass retains just as much: the pass still stops there (punching below an
  unreclaimed extent would break the top-down hole pattern a sidecar-less repair
  reads), but the failed extent and the untried remainder are kept for the same
  retry rather than discarded. Tight-space
  compaction drains that backlog before it plans, since the space is what it is
  short of. The residual probe-to-punch window is
  closed by lifetimes: the checkpoint captures its version under the held link
  window and that version pins every handle it links.
• Block verification runs on EVERY manifest repair, not only under salvage:
  plain repair() no longer blesses an SST whose corruption sits in a lazily
  read data block (the freshly computed digest would launder it past
  verify_integrity while reads fail). The salvage flag only decides what
  happens to a damaged table — rewritten (on) or dropped with a reason
  pointing at the salvage-enabled repair (off); rotted-parity-but-readable
  tables stay admitted, entering the normal attributable-heal path.
• Marker-based heal reconciliation syncs the SST's data before refreshing the
  manifest digest, and refuses the refresh when that sync fails: a heal whose
  write landed but whose sync did not must never have its post-heal digest
  recorded over bytes a power loss can still discard.
• A repair run has exactly two outcomes: a tree that opens, or an error.
  There is no third state, no directory of files for someone to deal with
  later, and no step that ends in "fix this by hand". Every file the rebuilt
  manifest does not name is removed before the run reports success — a foreign
  name, a duplicate id, a table no bound can make safe, a source its
  replacement supersedes — because a file left behind is an orphan the next
  open must sweep, and an open that cannot sweep it does not open. A removal
  the filesystem refuses is therefore an error, not a warning. Recovering the
  CONTENT of a damaged file is replication, a checkpoint plus journal replay,
  or a backup; it is never a copy the engine hides beside the tree.
• Repair carries NO state between runs, so a crashed attempt leaves nothing to
  reconcile, and the scan mutates nothing. The manifest commit is the only act
  that publishes anything, and nothing before it displaces a source: a table's
  replacement is built at {id}.repair-tmp — a name no scan adopts — and
  swapped onto {id} afterwards, while a salvaged blob takes a fresh id beside
  its untouched original. Publishing a table under a fresh id instead would
  leave a crash with the source AND its half-published copy both readable, and
  the retry, unable to tell them apart, would rebuild one history into L0
  twice, applying its merge operands twice on read. A leftover temp is
  garbage: the run whose committed manifest names its id finishes the swap
  (from an open() as well as a repair), any other run drops it. A copy that
  cannot be completed (its restriction could not be re-imposed) is removed on
  the spot, and a removal that fails propagates — an undiscardable
  half-finished replacement is the one thing that could corrupt the retry.
• The resurrection flag is an input to the run that reads the damaged bytes,
  not a state machine spread across runs. A table dropped only because the flag
  was off goes with that decision; nothing is stashed for a later run to
  reconsider, which is what keeps every later report describing the world the
  operator actually handed the engine.
• Only blob files a surviving table references enter the rebuilt manifest, and
  the rest are removed: an unreferenced one holds nothing reachable, and
  admitting it would strand it forever, since repair cannot rebuild the
  fragmentation stats blob GC needs to retire a file.
• Excluding a table loses what it said about its keys, and older versions of
  them survive elsewhere — a value it had overwritten, or a key its tombstone
  had deleted, becomes visible again. Nothing on disk distinguishes that from
  "the key was simply never rewritten", and covering the range with a synthetic
  deletion instead would destroy intact data (a flushed table spans most of the
  keyspace with seqnos above every older level, so one corrupt block would
  erase most of the tree, irreversibly). RepairReport::lost_coverage
  therefore NAMES each excluded table's key range and highest seqno, so a
  caller knows exactly where a superseded value may now be served. The seqno
  bound is None when the table's own sequence base lived in the lost manifest
  (a bulk-ingested SST stores every entry at local seqno 0), since the on-disk
  value there is a fabricated zero that would scope the affected history far
  too low. The allow_resurrection flag governs ambiguous VISIBILITY (a lost
  restriction bound, a lost or forged delete mask), not lost bytes; its docs
  now say so.
• scan_since_seqno mirrors the read path, which is what a consumer replaying
  it has to reproduce. Merge operands are all delivered: the read path collects
  every physically stored operand for a key and never deduplicates them by
  seqno, and apply_batch takes a caller-chosen seqno without requiring it to
  be unique, so identical operands can legitimately sit in one source or in
  two. Idempotent events — a write, a deletion, a range deletion — collapse
  across sources instead, since replaying them twice reaches the same state and
  the read path shadows the copies rather than compounding them; that is the
  manifest-loss-repair shape, where both the inputs and the outputs of an
  unfinished compaction are published. Events sharing a seqno are emitted
  oldest SOURCE first, so the value the tree serves is applied last: two
  sources can hold different values for one key at one seqno, and deciding
  that by payload bytes would hand precedence to byte order. Within ONE source
  a tied run keeps the order that source applies it — a batch may add several
  operands for a key under its shared seqno, and an order-sensitive merge
  operator would otherwise converge somewhere the tree never was. Within one
  seqno a
  range deletion is emitted FIRST — ahead of source recency, so a deletion in
  the newer source still leads — because suppression is strictly
  entry.seqno < tombstone.seqno: the tree keeps a write made at the
  tombstone's own seqno, so replaying the deletion last would drop it. The
  snapshot watermark is an INCLUSIVE bound, so an entry written at the maximum sequence
  number — which defines that watermark — is delivered from an SST exactly as
  from a memtable. The snapshot is taken under the version-history write guard:
  the watermark, the active memtable's entries and its range tombstones are
  captured together with writers excluded, because apply_batch takes the
  seqno from the caller and could otherwise commit at or below the watermark
  mid-walk, splitting one batch across the boundary and losing the remainder
  for a consumer that had advanced past it. The guard is released before
  mapping, since resolving blob indirections is I/O.
• A repair's own working files in blobs/ never make the tree unopenable: the
  scan sweeps the staging file its atomic write renames from instead of
  aborting on a name that does not parse as a blob id.
• Compaction installs the tree-wide sinks on every output it publishes,
  through a single binding point shared with flush, bulk ingest, and recovery.
  A compaction output previously carried no heal-hint sink, so a
  confirmed-persistent ECC correction while reading it was applied in memory
  but could never queue that SST for a durable healing rewrite. Bulk
  ingestion binds the BLOB files it publishes as well as its tables: without
  the deletion pause, one file's Drop can unlink it while a checkpoint is
  capturing, and a prefix punch can zero bytes the checkpoint already linked.
• A blob file's live frames are validated before any digest is recorded:
  hashing damaged content would launder the corruption past every later
  integrity check while reads of the affected values still fail. The gate
  verifies four independent properties — clean framing/checksums, payload
  decompression (a re-stamped checksum over an undecodable compressed payload
  frames cleanly), key order under the tree comparator (reordered frames
  break the relocation scanner's sorted-input contract), and metadata
  counters against the scanned frames (blob GC's dead-file arithmetic trusts
  them): exact equality for unpunched files, and lower bounds for punched
  ones, whose whole-file metadata must count at least the live suffix's
  items and bytes and whose key range must contain the scanned suffix. A file that fails validation is salvaged instead — its
  intact records are re-emitted into a compacted replacement under a fresh id
  (the damaged original is removed once the manifest names it), and the SSTs
  referencing it are rewritten onto the new offsets, dropping only entries
  whose record was lost (those keys read as absent, never as an error). The
  same rewrite drops a stale handle below a punched file's frontier. Blob
  salvage now also recovers COMPRESSED sources (each record is decompressed,
  proving it round-trips, then re-emitted under the same descriptor),
  including dictionary-compressed blobs on the repair path — repair passes
  the tree's configured dictionary; only the true standalone case without a
  dictionary fails closed.
• The blob directory scan mirrors the SST scan's discipline: persistently
  unreadable blob files and distinct duplicate spellings of one id are
  recorded (deterministically, canonical name kept) and removed after the
  commit, never left in blobs/ for the next open's orphan sweep to trip
  over. The SST punch guard is
  structure-anchored (a zero run counts only when it ends at a decodable
  block header, the extent end, or the data end) and evidence-backed (the run
  must cover a proven hole — zeros alone are the shape of a reclaim, not the
  proof, since destroyed bytes read the same; a backend that cannot answer
  leaves the run unproven, so the zeros stay damage and the table is
  salvaged), the report's salvaged
  count is derived after blob-dependency filtering so it stays a subset of
  recovered, and edit replay replaces the restriction sets wholesale (each
  edit carries its version's full set), so a removed blob file's frontier can
  never attach to a later file reusing its id.
• A restricted-blob reopen failure mid-slice routes through the same
  pre-install rollback as the SST reopen, retracting the slice's finalized
  outputs instead of leaking them under the exact low-space condition that
  engaged tight-space.
• A tight-space slice applies NO removal semantics: bottommost GC was already
  deferred, and the user compaction filter now defers the same way (the next
  normal compaction applies it). The slice output is therefore a strict
  superset shadowing any surviving input prefix, so an unpunched, sidecarless
  input republished whole by a manifest-loss repair can never resurrect a
  removed record. Shadowing is not enough on its own, though: merge operands
  are deliberately never deduplicated across sources, so publishing both
  histories would apply the consumed prefix's operands twice. The post-commit
  sidecar-failure window is therefore CLOSED rather than tolerated — the
  manifest is the authority for the bound, so an open that finds a restricted
  table whose .restrict-bound file is absent, unreadable, or DISAGREEING
  (a stale bound from a later slice whose own write failed, or another table's
  id) republishes it from the manifest — presence alone proves nothing, and a
  stale bound restricts less than reality. An input stays unpunched until its
  sidecar exists.
• A tight-space RESTRICTED view is excluded from merge-on-read verbatim block
  reuse: its scan() starts at the restriction bound and numbers rows from
  zero, while the relocation copies the whole physical section (punched prefix
  included) and publishes it unrestricted. Restricted inputs take the
  copy-on-write path instead.

Columnar scans under restriction and MVCC

• Table::columnar_scan clamps to a tight-space-restricted view's live
  suffix: data blocks wholly below the bound (hole-punched, reading as zeros)
  are stepped over with their zone-map row counts preserving positional
  delete mapping, and the straddling block's sub-bound rows are masked.
• The tree-level scan's singleton path dedups MVCC versions inside a single
  segment: a flush / compaction product physically stores every version of an
  overwritten key, so singletons that can carry duplicates (the writer's
  persisted key_count differing from item_count) get per-key
  newest-visible selection, with the predicate running after dedup exactly
  like the overlap-merge path. Provably-unique segments keep the zero-copy
  verbatim fast path.
• A projected seqno column is emitted in EFFECTIVE (tree-global) coordinates on
  every path: a bulk-ingested segment stores every row at local seqno 0 and
  takes its ordering from a per-segment offset, so the stored value would name
  a commit the tree never had. The overlap path writes each row's effective
  seqno (already computed for its dedup), since that union spans segments with
  different offsets. Masking still compares in local space — one subtraction
  per segment rather than one addition per row — so only the emitted column is
  translated.
• A key whose newest row DELETES it yields no row at all, on every emit path
  (dedup, verbatim, overlap): the scan used to pick that row without reading
  the value type and emit it, returning a key the point read reports absent —
  indistinguishable from a live row with an empty value unless the caller
  projected the type column. Only a segment whose metadata counts deletions
  decodes it, so one without them keeps its columns and its fast path.
• That dedup is newest-version-wins, which a MERGE chain is not: its older
  rows are the merge's inputs, so the scan would return the raw operand where
  a read returns the merged value, having already dropped the base row. A tree
  carrying a merge operator is therefore refused rather than served
  unresolved chains — the guard is on the operator, since without one the read
  path returns the newest entry unchanged too and nothing diverges.

Recovery progress

• RecoveryProgress (wired via Config::with_recovery_progress or
  SalvageOptions::progress) exposes live counters a UI thread can poll
  while a repair runs: tables and blob files as they are discovered and
  recovered, blocks as the salvage walk inspects / re-emits / drops them,
  ECC-healed blocks, and recovered KV entries and columnar columns. The
  recovered counts are published once the surviving set is final — after
  duplicate displacement and blob-dependency / reference filtering — so a
  snapshot can never claim more than the rebuilt manifest holds.

In-place ECC autoheal

• Corrected reads emit heal hints; a background pass rewrites healed blocks
  under fresh parity, guarded by a tri-state attestation marker so an
  inconclusive check preserves rather than deletes it.

Manifest recovery

• Rebuilding a lost manifest by scanning tables/ is total: every branch
  yields a valid tree, and re-running is deterministic.
• A tight-space-punched SST is recovered restricted to its live suffix.
  The .restrict-bound sidecar is written strictly after the slice's
  install commits, so a valid sidecar proves a committed restriction and
  its exact bound is honored directly, with no dead-prefix probing; absent
  a trustworthy sidecar, the bound derives from the punch geometry — but only
  once the ZEROED BLOCK'S OWN EXTENT proves it was punched (a new extent-local
  Fs hole probe, lseek(SEEK_DATA) on Linux), since corruption that destroys
  a data block leaves the same read-as-zeros shape a completed punch does, and
  a file-wide allocation total attributes nothing to a particular block — on a
  filesystem with transparent compression it reads low for an ordinary file.
  The same evidence gates the leading-bytes probe, so ordinary corruption there
  is salvaged rather than mistaken for a punch with a lost bound and dropped. The readable suffix is never
  discarded, even when it is itself corrupt (salvaged, then re-restricted).
  Ids named by a surviving sidecar are never handed to a rewritten copy, so no
  replacement can inherit a restriction belonging to another file.
• The single recovery policy knob, allow_resurrection (default drop),
  governs the ambiguous cases: a lost restriction bound and an
  unauthenticated or concealed delete mask. Off, ambiguous data is dropped
  to avoid resurrecting superseded or deleted rows; on, it is kept. Neither
  setting requires a manual step.
• Transient I/O propagates for retry; persistent failures classify
  deterministically. Genuinely unrecoverable files (foreign name, redundant
  duplicate, undecodable) are dropped, still leaving a valid tree.
• Every file the repair removes or swaps is resolved through the backend it
  was found under, so a per-level route's tables are handled on their own
  tier rather than looked for in the primary namespace.
• A repair whose post-commit cleanup failed leaves a durable, correct
  manifest beside the files it could not remove or swap. The next repair (or
  the next open(), for a pending swap) finishes
  that cleanup before it scans — the committed manifest is the tree's own
  authority on which files count, exactly as an open treats everything it does
  not name as an orphan — so a retry cannot rebuild one history twice. A
  manifest that does not load cleanly is the case repair exists for and is not
  consulted at all.
• L0 order is derived from the files, never from the directory scan or from
  sequence numbers: callers may assign seqnos explicitly, so a table's highest
  seqno says nothing about which table is newer. Ids are allocated in
  increasing order, so descending id is the recency order — and being total,
  it also makes a repeated repair over the same files reproduce the same tree.
  The scan itself is
  ordered for the same reason, canonical {id} spelling first, so an alternate
  spelling of one id can never displace the writer's own file.
• The out-of-band verifier crosses a reclaimed prefix in bulk reads, scanning
  for the block magic instead of decoding a header at every byte — a
  multi-gigabyte punched prefix would otherwise take the diagnostic out of
  reach on exactly the files it exists to inspect. It also matches the
  restriction sidecar to the file it sits beside (from the caller's id, else
  from the SST's numeric name): an unmatchable sidecar never skips blocks, so a
  copied or stale one cannot make a zeroed leading block read as reclaim. What
  a matching sidecar permits is bounded by the table's own index — where its
  bound actually falls — not by the first zeros: the punch runs
  highest-block-first and stops at its first failure, so a table with a
  committed restriction and no hole at all is a supported state, and there the
  first zeros are destroyed data.

Checkpoint interaction

• A checkpoint that reflinks a file syncs the clone at the requested
  durability before treating it as complete.
• Tight-space prefix reclamation refuses to punch a file whose inode is
  shared with a checkpoint, so reclaiming space in the live tree can never
  hole-punch a checkpoint's data.

Tooling and CI

• A heal fuzzer drives single-bit corruption across a fixed corpus of SSTs
  (varied block size, per-KV checksum, columnar layout, compression,
  encryption, Page-ECC), asserting recover-and-scan never panics and never
  returns a wrong value, and dumping the exact failing SST for repro.

• The fuzzer needs a CI home, so this PR adds the fuzz-heal job that runs
  it on a fixed budget with retries disabled (a retry would overwrite the
  reproducer dump or mask a non-replaying failure) and surfaces the failing
  SST inline in the log.

• Pinning policy: first-party GitHub actions (actions/*, dependabot/*)
  are pinned by major version tag (actions/checkout@v7) — SHA-pinning a
  GitHub-owned org adds no supply-chain protection while churning a
  Dependabot bump PR per patch release. Third-party actions (dtolnay,
  Swatinem, taiki-e, codecov, release-plz, benchmark-action) stay pinned to
  full commit SHAs; release-plz/action is aligned to v0.5.131. No workflow
  behaviour changes beyond the new job.

• Coverage policy: the patch status uses a fixed 80% target instead of the
  auto compare-to-project bar (~92%), which recovery/verification diffs cannot
  meaningfully meet — they are dominated by defensive error-classification
  arms (Display impls, double-fault plumbing, exotic-corruption classifiers)
  whose fail-closed contract the representative refuse/abort tests already
  pin. The heal fuzzer harness is excluded from coverage: it runs under the
  dedicated uninstrumented fuzz-heal job, so its lines can only ever read as
  unexecuted.

Testing

• Full suite green with --all-features (nextest), no_std (alloc) clean,
  cargo doc --all-features -D warnings and doctests clean, clippy
  all-features/all-targets clean. docs/manifest-recovery.md diagrams the
  recovery algorithm and its invariants.

Related

• feat(tools): lossless V3/V4 to V5 format converter (CLI + API) #580 — offline pre-V5 → V5 conversion. FormatVersion reserves the retired
  discriminants as numbers, not as public variants: legacy decoding lives only
  in that tool, while recovery, salvage, patrol scrub and verify stay V5-only.

Closes #568
Closes #570

[coderabbitai]

Important

Review skipped

Too many files!

This PR contains 130 files, which is 30 over the limit of 100.

To get a review, reduce the PR to 100 files or fewer by splitting it into smaller PRs or changing its base branch.

Upgrade to a paid plan to raise the limit.

This review couldn't start because sufficient usage credits or metered capacity aren't available. Add credits or update usage-based reviews in the billing tab, then retry.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 4463d01d-b553-411b-8eb3-6e272d9be5c1

📥 Commits

Reviewing files that changed from the base of the PR and between db61920 and 2f4d602.

📒 Files selected for processing (130)

• .codecov.yml
• .config/nextest.toml
• .github/workflows/benchmark.yml
• .github/workflows/cleanup-branches.yml
• .github/workflows/coordinode-ci.yml
• .github/workflows/coordinode-release.yml
• .github/workflows/dependabot-auto-merge.yml
• .github/workflows/issue-labeler.yml
• .github/workflows/release.yml
• .gitignore
• Cargo.toml
• README.md
• benches/index_block.rs
• docs/data-integrity.md
• docs/manifest-recovery.md
• docs/tight-space-compaction.md
• src/abstract_tree.rs
• src/blob_tree/ingest.rs
• src/blob_tree/ingest/tests.rs
• src/blob_tree/mod.rs
• src/cache.rs
• src/checkpoint.rs
• src/checkpoint/tests.rs
• src/compaction/flavour.rs
• src/compaction/flavour/tests.rs
• src/compaction/stream.rs
• src/compaction/worker.rs
• src/compaction/worker/tests.rs
• src/comparator.rs
• src/config/mod.rs
• src/deletion_pause.rs
• src/deletion_pause/tests.rs
• src/encryption/mod.rs
• src/error.rs
• src/file.rs
• src/file_accessor.rs
• src/format_version.rs
• src/fs/crash_fs.rs
• src/fs/fault_fs.rs
• src/fs/fault_fs/tests.rs
• src/fs/io_uring_fs.rs
• src/fs/io_uring_raw.rs
• src/fs/io_uring_raw/tests.rs
• src/fs/mem_fs.rs
• src/fs/mem_fs/tests.rs
• src/fs/mod.rs
• src/fs/std_fs.rs
• src/fs/std_fs/tests.rs
• src/fuzz_heal.rs
• src/io/mod.rs
• src/key_range.rs
• src/lib.rs
• src/mvcc_stream.rs
• src/range_tombstone.rs
• src/recovery_progress.rs
• src/repair.rs
• src/repair/tests.rs
• src/restrict_bound.rs
• src/restrict_bound/tests.rs
• src/salvage.rs
• src/salvage/tests.rs
• src/scan_since.rs
• src/scrub.rs
• src/scrub/ecc_tests.rs
• src/scrub/heal_attest.rs
• src/scrub/heal_attest/tests.rs
• src/scrub/tests.rs
• src/sfa/reader.rs
• src/table/block/decoder.rs
• src/table/block/mod.rs
• src/table/block/tests.rs
• src/table/block_layout.rs
• src/table/columnar.rs
• src/table/columnar/tests.rs
• src/table/columnar_predicate.rs
• src/table/columnar_predicate/tests.rs
• src/table/data_block/iter_test.rs
• src/table/data_block/mod.rs
• src/table/filter/block.rs
• src/table/index_block/mod.rs
• src/table/inner.rs
• src/table/meta.rs
• src/table/meta/tests.rs
• src/table/mod.rs
• src/table/multi_writer.rs
• src/table/multi_writer/tests.rs
• src/table/relocate.rs
• src/table/relocate/tests.rs
• src/table/scanner.rs
• src/table/seqno_bounds.rs
• src/table/tests.rs
• src/table/util.rs
• src/table/writer/mod.rs
• src/table/writer/tests.rs
• src/table/zone_map.rs
• src/test_forge.rs
• src/tree/columnar_scan.rs
• src/tree/ingest.rs
• src/tree/mod.rs
• src/tree/scan_since_freeze_tests.rs
• src/verify.rs
• src/verify/block_verify_tests.rs
• src/version/diff.rs
• src/version/edit.rs
• src/version/edit/tests.rs
• src/version/mod.rs
• src/version/recovery.rs
• src/version/recovery/tests.rs
• src/version/run.rs
• src/version/run/tests.rs
• src/vlog/blob_file/meta.rs
• src/vlog/blob_file/meta/tests.rs
• src/vlog/blob_file/mod.rs
• src/vlog/blob_file/multi_writer.rs
• src/vlog/blob_file/reader.rs
• src/vlog/blob_file/reader/tests.rs
• src/vlog/blob_file/scanner.rs
• src/vlog/blob_file/scanner/tests.rs
• src/vlog/blob_file/writer.rs
• src/vlog/mod.rs
• src/vlog/tests.rs
• tests/columnar_scan.rs
• tests/recovery_healtmp_sweep.rs
• tests/repair.rs
• tests/scan_since_seqno.rs
• tests/tree_bulk_ingest.rs
• tools/db_bench/src/main.rs
• tools/sst-dump/Cargo.toml
• tools/sst-dump/src/main.rs
• tools/sst-dump/tests/verify_smoke.rs

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 30c1ccabef

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[polaz]

@codex review

[codecov]

Codecov Report

❌ Patch coverage is 85.95016% with 451 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/scrub.rs	71.50%	100 Missing ⚠️
src/verify.rs	80.12%	94 Missing ⚠️
src/tree/mod.rs	91.47%	29 Missing ⚠️
src/fs/std_fs.rs	27.02%	27 Missing ⚠️
src/fs/io_uring_fs.rs	0.00%	19 Missing ⚠️
src/table/writer/mod.rs	93.77%	18 Missing ⚠️
src/fs/mod.rs	30.00%	14 Missing ⚠️
src/table/relocate.rs	68.88%	14 Missing ⚠️
src/table/block/mod.rs	89.60%	13 Missing ⚠️
src/blob_tree/mod.rs	63.63%	12 Missing ⚠️
... and 21 more

📢 Thoughts on this report? Let us know!

[chatgpt-codex-connector]

💡 Codex Review

coordinode-lsm-tree/src/vlog/mod.rs

Lines 236 to 237 in 1ddd0e8

	let metadata_slice = crate::file::read_exact(&*file, metadata_section.pos(), metadata_len)?;
	Metadata::from_slice(&metadata_slice)?

Reject blob metadata IDs that disagree with filenames

Manifest repair takes the numeric filename as id, but this recovery helper parses the authenticated metadata without checking meta.id == id and then constructs the BlobFile under the filename's ID. If blob files were renamed or swapped while the manifest was unavailable, repair therefore publishes a valid-looking manifest that routes SST handles to the wrong physical file; when the misplaced file happens to contain the same key and compatible offset/length, the reader can return its stale value because it validates the key and frame checksum but not the frame seqno against the SST entry. Treat an ID mismatch as an unreadable file and quarantine it rather than rebinding its contents.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 86487633e4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[polaz]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a7c8252f5b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[polaz]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

coordinode-lsm-tree/src/tree/columnar_scan.rs

Line 163 in 8b5fce0

for table in super_version.version.iter_tables() {

Apply persisted range tombstones during columnar scans

When columnar mode flushes rows together with a remove_range, the resulting SST retains that range tombstone until a later compaction converts it into a delete bitmap. This selection loop includes the SST, but no subsequent path reads its range tombstones, so columnar_scan returns covered rows that normal reads suppress. Apply snapshot-visible range tombstones while merging the selected segments, or reject RT-bearing segments instead of returning deleted data.

---

coordinode-lsm-tree/src/checkpoint.rs

Lines 715 to 717 in 8b5fce0

	let target_root = normalized_target.as_path();
	prepare_target(target_root, include_blobs, &**target_fs)?;

Reject checkpoint targets inside live data directories

When a caller chooses a fresh target beneath the source tree's tables/ or blobs/ directory, this proceeds to create the checkpoint there without checking its relationship to src_root. For example, $DB/tables/checkpoint succeeds but leaves a nonnumeric directory among the SST files; the next source-tree open scans that entry as a table ID and fails recovery. On a shared filesystem namespace, reject targets that resolve inside the source's live data directories before creating them.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d78d47910c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

coordinode-lsm-tree/src/tree/columnar_scan.rs

Lines 712 to 715 in 218e68d

	order.sort_by(|&a, &b| {
	cmp.compare(key_at(a), key_at(b))
	.then_with(|| eff_at(b).cmp(&eff_at(a)))
	});

Break tied columnar rows by source recency

When overlapping columnar SSTs contain the same key at the same effective seqno, this comparator declares the rows equal and the later dedup keeps whichever segment happened to be combined first. group_by_overlap orders segments by minimum key rather than table recency, so an older SST containing a,k@10 can precede a newer SST containing b,k@10; columnar_scan then returns the older value for k even though the normal read path serves the newer L0 source. Carry source precedence into the row metadata and use it as the tie-break after effective seqno.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

coordinode-lsm-tree/src/tree/columnar_scan.rs

Lines 712 to 715 in 287f624

	order.sort_by(|&a, &b| {
	cmp.compare(key_at(a), key_at(b))
	.then_with(|| eff_at(b).cmp(&eff_at(a)))
	});

Break equal-seqno columnar ties by source recency

When overlapping columnar SSTs contain the same key at the same effective seqno but with different values or a newer tombstone—reachable because callers assign seqnos—this comparison returns Equal, so the stable sort keeps their concatenation order. group_by_overlap orders segments by minimum key rather than source precedence, meaning an older table with a smaller minimum key can win and columnar_scan can return a value that the normal point-read path shadows with the newer L0 source. Carry source recency into the row metadata and use it as the tie-break after effective seqno.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7e62b9b48e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

coordinode-lsm-tree/src/tree/mod.rs

Line 2123 in 82a8154

self.scan_since_seqno_with(target_seqno, true, |_version, _entry| {

Reject scans older than retained MVCC history

When compaction has run with an MVCC GC watermark above target_seqno, older versions and tombstones in the requested interval may already have been physically dropped or had their sequence numbers zeroed, but this still returns a successful iterator over the remaining records. A CDC consumer therefore silently receives an incomplete or mis-sequenced history despite the method's promise to return every committed change at or after the target. Persist and check a retained-history watermark, or otherwise reject targets whose history may have been reclaimed instead of presenting the result as a complete CDC stream.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

coordinode-lsm-tree/src/tree/columnar_scan.rs

Lines 720 to 721 in 828d740

	cmp.compare(key_at(a), key_at(b))
	.then_with(|| eff_at(b).cmp(&eff_at(a)))

Break tied columnar versions by source recency

When overlapping SSTs contain different values for the same key at the same effective seqno, this comparison treats the versions as equal and the stable sort preserves segment order. That order is not recency order: group_by_overlap first sorts segments by minimum key, so an older segment with a smaller minimum can precede and win over a newer segment. Explicit tied seqnos are supported, and point reads give the newer L0 source precedence, so columnar_scan can return a different value; carry source recency into this tie-break.

---

coordinode-lsm-tree/src/tree/columnar_scan.rs

Lines 165 to 168 in 828d740

	let super_version = self.version_history.read().get_version_for_snapshot(seqno);
	let mut segments: Vec<Segment> = Vec::new();
	for table in super_version.version.iter_tables() {

Apply range tombstones during columnar scans

When a visible range tombstone covers rows in an older columnar SST, this scan collects only table row segments and never gathers or applies range tombstones from the captured version. This remains wrong after the deletion itself is flushed into an SST, so it is not merely the documented omission of active memtable rows: a normal point/range read reports the covered keys absent while columnar_scan emits them. Collect the visible table range tombstones for the snapshot and suppress covered rows using the same seqno rules as the regular read path.

---

coordinode-lsm-tree/src/abstract_tree.rs

Lines 1347 to 1348 in 828d740

	self.write_admission()?;
	Ok(self.insert(key, value, seqno))

Include the pending write in admission accounting

When the remaining quota or physical free space is larger than the fixed reserved band but smaller than the value being inserted, this pre-check succeeds because it accounts only for the existing on-disk footprint and already-pending memtables; it never includes the incoming key/value size. For example, with 1.5 MiB remaining and an empty memtable, the 1 MiB reserve admits a 2 MiB value, after which its flush can exceed the configured budget or hit ENOSPC despite use of the admission-gated API. Pass the proposed write size into the check or reserve it atomically before inserting.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[polaz]

Superseded by #588: same tree, squashed to one commit — this PR's review history (117 threads, all resolved) had grown past the point of usability.

[chatgpt-codex-connector]

💡 Codex Review

coordinode-lsm-tree/src/tree/columnar_scan.rs

Lines 167 to 168 in 2f4d602

	let mut segments: Vec<Segment> = Vec::new();
	for table in super_version.version.iter_tables() {

Apply range tombstones during columnar scans

When a columnar SST contains a range tombstone that has not yet been materialized into its positional delete bitmap—for example, immediately after inserting rows, calling remove_range, and flushing but before compaction—this scan collects only the tables' columnar rows and never evaluates visible_range_tombstones(). It therefore returns keys covered by the newer tombstone even though point and ordinary range reads report them deleted; gather visible range tombstones from the snapshot and suppress each affected row by its effective seqno.

---

coordinode-lsm-tree/src/tree/columnar_scan.rs

Lines 719 to 722 in 2f4d602

	order.sort_by(|&a, &b| {
	cmp.compare(key_at(a), key_at(b))
	.then_with(|| eff_at(b).cmp(&eff_at(a)))
	});

Retain source recency when columnar seqnos tie

When two overlapping columnar ingestions have the same effective seqno—which is possible after the public sequence counter is reset with set—this ordering has no source-recency tie-break. Because group_by_overlap previously sorted the sources by minimum key, the first row retained for a tied key can come from the older table solely because that table's range starts earlier, while the normal L0 point-read path keeps the newer run on equal seqnos. Carry the table/run recency into the merged rows and use it after effective seqno so the projected scan returns the same value as the tree.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".