feat(salvage): total recovery for SSTs and blob files with in-place ECC autoheal

.config/nextest.toml

@@ -49,6 +49,27 @@ slow-timeout = { period = "30s", terminate-after = 4 }
 filter = "test(partial_decode_) | test(lazy_block) | test(ecc_heal_scheduled) | test(block_layout_section_roundtrips_for_large_zstd) | test(compatibility_matrix_round_trips)"
 slow-timeout = { period = "30s", terminate-after = 4 }
 
+# Wall-clock concurrency stress: 4 writers + 2 readers contend on one
+# encrypted tree for a fixed 2-second window, then exact-equality
+# assertions run over the committed tally. The test's correctness envelope
+# assumes its threads actually get CPU / disk / fds during that window;
+# co-scheduled with the rest of the suite it can starve and flake on
+# resource pressure that has nothing to do with encryption. Claim every
+# scheduler slot so it runs ALONE — the assertions stay at full strength
+# (a real engine race still fires solo), only cross-test contention is
+# removed. Mirrored in the ci profile below.
+[[profile.default.overrides]]
+filter = "test(concurrent_encrypted_no_corruption)"
+threads-required = "num-cpus"
+
+# The single-byte-bitrot heal fuzzer runs for a fixed ~45s wall-clock budget
+# (`#[ignore]`d, so it only runs under `--run-ignored`), well past the 10s
+# ceiling by design. Give it a generous window so a slower runner's corpus
+# build does not push it into a spurious termination.
+[[profile.default.overrides]]
+filter = "test(fuzz_heal_bitrot)"
+slow-timeout = { period = "90s", terminate-after = 2 }
+
 # CI profile: retries for flaky tests, JUnit XML output, longer timeouts.
 [profile.ci]
 retries = 2
@@ -65,3 +86,16 @@ store-failure-output = true
 [[profile.ci.overrides]]
 filter = "test(prop_)"
 slow-timeout = { period = "120s", terminate-after = 4 }
+
+# See the default-profile override of the same filter: the wall-clock
+# stress test claims the whole scheduler so cross-test contention cannot
+# starve its 2-second window.
+[[profile.ci.overrides]]
+filter = "test(concurrent_encrypted_no_corruption)"
+threads-required = "num-cpus"
+
+# The bitrot heal fuzzer's fixed ~45s budget (see the default-profile note); a
+# generous timeout keeps it from a spurious termination on a slow CI runner.
+[[profile.ci.overrides]]
+filter = "test(fuzz_heal_bitrot)"
+slow-timeout = { period = "90s", terminate-after = 2 }

.github/workflows/benchmark.yml

@@ -83,7 +83,7 @@ jobs:
       # token leakage where actions/upload-artifact captures the
       # whole working tree, including the .git/config that the
       # default checkout populated with GITHUB_TOKEN).
-      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0  # v7.0.0
+      - uses: actions/checkout@v7
         with:
           persist-credentials: false
 
@@ -104,7 +104,7 @@ jobs:
           github.event_name == 'workflow_dispatch' ||
           (github.event_name == 'pull_request' &&
            github.event.pull_request.head.repo.full_name == github.repository)
-        uses: actions/create-github-app-token@bcd2ba49218906704ab6c1aa796996da409d3eb1  # v3.2.0
+        uses: actions/create-github-app-token@v3
         with:
           app-id: ${{ secrets.RELEASER_APP_ID }}
           private-key: ${{ secrets.RELEASER_APP_PRIVATE_KEY }}

.github/workflows/cleanup-branches.yml

@@ -13,7 +13,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Delete stale release-plz branches
-        uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3  # v9.0.0
+        uses: actions/github-script@v9
         with:
           script: |
             const branches = await github.paginate(

.github/workflows/coordinode-ci.yml

@@ -210,7 +210,7 @@
         if: needs.changes.outputs.code == 'false'
         run: |
           echo "::notice::No code paths changed — skipping format + clippy. Required-check name still emitted as success."
-      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0  # v7.0.0
+      - uses: actions/checkout@v7
         if: needs.changes.outputs.code != 'false'
         with:
           persist-credentials: false
@@ -283,7 +283,7 @@
         if: needs.changes.outputs.code != 'false' && needs.lint.result != 'success'
         run: |
           echo "::notice::Lint failed (result=${{ needs.lint.result }}) — skipping test work to avoid duplicate noise on an already-failed PR. Required-check name still emitted; fix the lint failure to unblock real test runs."
-      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0  # v7.0.0
+      - uses: actions/checkout@v7
         if: needs.changes.outputs.code != 'false' && needs.lint.result == 'success'
         with:
           persist-credentials: false
@@ -327,7 +327,55 @@
         working-directory: tools/sst-dump
         run: cargo nextest run
 
+  fuzz-heal:
+    # Reproducible single-byte-bitrot fuzzer over the SST read / heal path:
+    # flips one bit in a corpus of SSTs (varied block size, per-KV checksum,
+    # columnar, compression, encryption, Page-ECC) for a fixed ~45s budget and
+    # asserts the read path never panics and never returns a wrong value (a
+    # flipped block heals via ECC or fails its checksum, never silent corruption).
+    # `#[ignore]`d, so it is excluded from the normal `test` job and run here with
+    # `--run-ignored=only`. Ubuntu-only (the invariant is platform-independent);
+    # not a required check.
+    needs: [changes, lint]
+    if: ${{ needs.changes.outputs.code != 'false' && needs.lint.result == 'success' }}
+    timeout-minutes: 15
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v7
+        with:
+          persist-credentials: false
+      - uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8  # stable
+        with:
+          toolchain: stable
+      - uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4  # v2.9.1
+        with:
+          prefix-key: ubuntu-cargo
+      - uses: taiki-e/install-action@1ef5c5f58e85d25baaaa1704478fdd6c2921f2b5  # nextest
+      - name: Run bitrot heal fuzzer
+        # Pin the reproducer directory explicitly so the test's dump location does
+        # not depend on the process working directory (robust if the crate ever
+        # moves into a workspace subdirectory). The dump step reads the same path.
+        env:
+          FUZZ_HEAL_REPRO_DIR: ${{ github.workspace }}
+        # `--retries 0` overrides the ci profile's retries: the fuzzer dumps the
+        # EXACT failing SST, and a retry (on a non-deterministic corpus) would
+        # either overwrite that dump or mask a real failure that does not replay.
+        run: cargo nextest run --profile ci --all-features --run-ignored=only --retries 0 -E 'test(fuzz_heal_bitrot)'
+      - name: Dump reproducer on failure
+        if: failure()
+        # The corpus is not byte-deterministic (encrypted / timestamped SSTs), so
+        # the seed alone cannot replay a failure in those. The test writes the
+        # EXACT failing SST to `fuzz_heal_repro.sst`; surface it (base64) in the
+        # log so the case reproduces directly, no artifact upload needed.
+        env:
+          FUZZ_HEAL_REPRO_DIR: ${{ github.workspace }}
+        run: |
+          echo "=== fuzz_heal_repro.txt ==="
+          cat "$FUZZ_HEAL_REPRO_DIR/fuzz_heal_repro.txt" 2>/dev/null || echo "(no repro txt)"
+          echo "=== fuzz_heal_repro.sst (base64) ==="
+          base64 "$FUZZ_HEAL_REPRO_DIR/fuzz_heal_repro.sst" 2>/dev/null || echo "(no repro sst)"
+
   no-std-check:
     # Gates the `#![no_std]` + alloc engine path. The engine modules compile
     # unconditionally; only the std default trait implementations (the system
     # filesystem, the io_uring backend, the system clock) stay behind
@@ -346,7 +394,7 @@
     timeout-minutes: 10
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0  # v7.0.0
+      - uses: actions/checkout@v7
         with:
           persist-credentials: false
       - uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8  # stable
@@ -429,7 +477,7 @@
         target: ${{ fromJSON(needs['cross-matrix'].outputs.targets) }}
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0  # v7.0.0
+      - uses: actions/checkout@v7
         with:
           persist-credentials: false
       - uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8  # stable
@@ -447,7 +495,7 @@
     timeout-minutes: 20
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0  # v7.0.0
+      - uses: actions/checkout@v7
         with:
           persist-credentials: false
       - uses: dtolnay/rust-toolchain@5b842231ba77f5c045dba54ac5560fed2db780e2  # nightly

.github/workflows/coordinode-release.yml

@@ -11,7 +11,7 @@ jobs:
     steps:
       - name: Generate token (SW Release Bot)
         id: app-token
-        uses: actions/create-github-app-token@bcd2ba49218906704ab6c1aa796996da409d3eb1  # v3.2.0
+        uses: actions/create-github-app-token@v3
         with:
           app-id: ${{ secrets.RELEASER_APP_ID }}
           private-key: ${{ secrets.RELEASER_APP_PRIVATE_KEY }}
@@ -23,7 +23,7 @@ jobs:
           permission-pull-requests: write
 
       - name: Checkout
-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0  # v7.0.0
+        uses: actions/checkout@v7
         with:
           fetch-depth: 0
           # persist-credentials: false — release-plz reads the token via
@@ -44,7 +44,7 @@ jobs:
 
       # Step 1: Create or update release PR (version bump + changelog)
       - name: Create release PR
-        uses: release-plz/action@e8792575c7f2366cf6ff3ccc33ead9ace5b691c7  # v0.5.130
+        uses: release-plz/action@2eb1d8bcb770b4c48ccfaad919734b38b51958c9  # v0.5.131
         with:
           command: release-pr
           config: .release-plz.toml
@@ -54,7 +54,7 @@ jobs:
       # Step 2: If version in Cargo.toml > latest tag, create GitHub Release + tag
       # This triggers release.yml → cargo publish via OIDC
       - name: Create GitHub Release
-        uses: release-plz/action@e8792575c7f2366cf6ff3ccc33ead9ace5b691c7  # v0.5.130
+        uses: release-plz/action@2eb1d8bcb770b4c48ccfaad919734b38b51958c9  # v0.5.131
         with:
           command: release
           config: .release-plz.toml

.github/workflows/dependabot-auto-merge.yml

@@ -15,22 +15,22 @@ jobs:
 
     steps:
       - name: Checkout
-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0  # v7.0.0
+        uses: actions/checkout@v7
         with:
           # gh CLI authenticates via the GH_TOKEN env var on the merge
           # step; no git config credentials are needed here.
           persist-credentials: false
 
       - name: Dependabot metadata
         id: metadata
-        uses: dependabot/fetch-metadata@25dd0e34f4fe68f24cc83900b1fe3fe149efef98  # v3.1.0
+        uses: dependabot/fetch-metadata@v3
         with:
           github-token: "${{ secrets.GITHUB_TOKEN }}"
 
       - name: Generate App Token
         id: app-token
         if: steps.metadata.outputs.update-type == 'version-update:semver-minor' || steps.metadata.outputs.update-type == 'version-update:semver-patch'
-        uses: actions/create-github-app-token@bcd2ba49218906704ab6c1aa796996da409d3eb1  # v3.2.0
+        uses: actions/create-github-app-token@v3
         with:
           app-id: ${{ secrets.RELEASER_APP_ID }}
           private-key: ${{ secrets.RELEASER_APP_PRIVATE_KEY }}

.github/workflows/issue-labeler.yml

@@ -12,7 +12,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Label issue by title prefix
-        uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3  # v9.0.0
+        uses: actions/github-script@v9
         with:
           script: |
             const title = context.payload.issue.title.toLowerCase();

.github/workflows/release.yml

@@ -19,7 +19,7 @@ jobs:
       contents: read
       id-token: write
     steps:
-      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0  # v7.0.0
+      - uses: actions/checkout@v7
         with:
           # cargo publish authenticates via OIDC against crates.io
           # (rust-lang/crates-io-auth-action below); no git config

.gitignore

@@ -23,6 +23,10 @@ mutants*
 profile.json
 fuzz*/**/out*
 
+# The bitrot heal fuzzer dumps the exact failing SST here on a failure so the
+# case reproduces; a runtime artifact, never committed.
+fuzz_heal_repro.*
+
 old
 .claude/
 .forge/

Cargo.toml

@@ -161,13 +161,25 @@ spin = { version = "0.12", default-features = false, features = ["mutex", "spin_
 # userspace fast-path, concurrent readers); `spin::RwLock` is the no_std lock.
 hashbrown = { version = "0.17", default-features = false, features = ["default-hasher"] }
 parking_lot = { version = "0.12", optional = true }
-lz4_flex = { version = "0.13.0", optional = true, default-features = false }
-# Default features carry the runtime-dispatched SIMD decode kernels (avx2/bmi2/
-# neon/sve/...); `default-features = false` would silently drop them and leave
-# the sequence decoder on the scalar path, measurably slower on cold point reads.
-# 0.0.48 is the published crates.io release; it fixes a dictionary match
-# binary-tree encoder panic present in 0.0.46/0.0.47 on high (BT-strategy) levels.
-structured-zstd = { version = "0.0.48", optional = true, features = ["lsm"] }
+# LZ4 block codec for hot-path blocks. `default-features = false` drops the
+# `std` and `frame` defaults (we use only the block API and stay no_std-capable
+# via `alloc`), but the malformed-input safety flags are NOT dropped: verify and
+# salvage decompress SST bytes that may be DAMAGED, so keep `checked-decode`
+# (decompression bounds checks) plus the `safe-decode` / `safe-encode` pure-safe
+# paths — disabling them would risk UB on adversarial/corrupt input.
+lz4_flex = { version = "0.14.0", optional = true, default-features = false, features = [
+    "alloc",
+    "checked-decode",
+    "safe-decode",
+    "safe-encode",
+] }
+# structured-zstd default features carry the runtime-dispatched SIMD decode
+# kernels (avx2/bmi2/neon/sve/...); `default-features = false` would silently
+# drop them and leave the sequence decoder on the scalar path, measurably slower
+# on cold point reads. 0.0.49 is the published crates.io release; it carries the
+# dictionary match binary-tree encoder panic fix (introduced in 0.0.48, absent
+# in 0.0.46/0.0.47 on high BT-strategy levels) plus encoder decode/parse perf.
+structured-zstd = { version = "0.0.49", optional = true, features = ["lsm"] }
 # `once_cell::race::OnceBox` — the no-std + alloc one-shot primitive.
 # We pick it over `std::sync::OnceLock` (which has both `set` and the
 # stabilised `get_or_try_init` on our 1.92 MSRV) because OnceLock is

README.md

@@ -13,7 +13,7 @@ LSM-tree storage engine in Rust. Embedded library; provides keyed point reads, p
 
 ## Status
 
-On-disk format version **V5**. V5 introduces a wire-format break for filter blocks (BuRR replaces Bloom); V3 and V4 databases are not readable by this version and vice versa. Versioning is single-monotonic: every breaking format change bumps to the next version with explicit migration notes.
+On-disk format version **V5** — the ONLY supported format. The engine carries no legacy decode paths, no backward-compat variations, and no in-place upgrade: a pre-V5 database fails to open with `InvalidVersion`. Versioning is single-monotonic: every breaking format change bumps to the next version with explicit migration notes. Conversion of pre-V5 databases is planned as standalone migration tooling, kept strictly outside the live engine.
 
 ## Quick start
 

docs/data-integrity.md

@@ -255,8 +255,31 @@ inspection, or rollback to a known-good point.
   fully-valid file, quarantine the corrupt ones, and report the key range each
   dropped, so one bad block costs only its own keys instead of the whole file. A
   columnar segment with a damaged sidecar degrades conservatively: a torn
-  sub-column drops just its block, and a corrupt delete-bitmap reads as "all rows
-  live, pending recompaction" rather than failing the open.
+  sub-column drops just its block. A delete-bearing segment whose positional
+  delete bitmap cannot be applied (unreadable bitmap, or a bitmap whose
+  positioning zone map is unreadable) fails the salvage closed by default —
+  recovering "all rows live" would resurrect deleted rows — unless explicitly
+  opted in (`SalvageOptions::allow_delete_resurrection`, `sst-dump salvage
+  --allow-delete-resurrection`).
+- **`salvage::salvage_blob_file(src, dest, &fs, id, &comparator) -> crate::Result<BlobSalvageReport>`**:
+  record-granular salvage of one blob (vlog) file. The `comparator` must be the
+  SAME `SharedComparator` the source tree was written with (pass the tree's
+  configured comparator, or `comparator::default_comparator()` for the default
+  lexicographic ordering): the salvage walk orders and validates recovered
+  records under it, so a mismatched comparator would mis-order the output. When a frame fails checksum,
+  header-CRC, or structural validation, the record stream re-syncs to the next
+  frame magic WHEN one is found in-bounds; if none is (for example a CRC-vouched
+  frame end overruns the data section), the scan terminates. Either way the
+  resync magic (and every frame chained after it) has an unproven boundary (it
+  may be nested in the damaged frame's user bytes), so the walk **drops the
+  entire tail past the first resync / termination** (fail closed): the
+  conservative loss is as much as everything after the first damaged record, not
+  just that one record, because a fabricated chain of checksum-valid frames is
+  indistinguishable from genuine ones and re-emitting it would forge records.
+  Only records BEFORE the first resync are recovered. The salvaged file is written COMPACTED, so it is **not a drop-in
+  replacement** while SST entries hold `ValueHandle::offset` values into the
+  source: re-target them through `BlobSalvageReport::offset_remap` first (a
+  source offset absent from the map is a lost record).
 - **`Config::repair_with_salvage(true)`** (also `tools/sst-dump repair
   --salvage`): the manifest rebuild above, but an SST that fails verification is
   block-salvaged in place instead of being left out, and