Implement exact EnsureIndices v8 recovery

Author: aaltshuler

AGENTS.md

@@ -253,11 +253,11 @@ omnigraph policy explain --cluster ./company-brain --graph knowledge --actor act
 | Columnar storage on object store | ✅ Arrow/Lance | URI normalization, S3 env-var plumbing |
 | Per-dataset versioning + time travel | ✅ | `snapshot_at_version`, `entity_at`, snapshot-pinned reads across many tables |
 | Per-dataset branches | ✅ | **Graph-level** refs are logically atomic through authoritative `__manifest` `BranchContents`; native create/delete crash gaps are classified and reclaimed under a single-writer-process boundary; live names are path-prefix-disjoint; data-table forks are lazy; system branches are filtered |
-| Atomic single-dataset commits | ✅ | **Multi-table publish via three layers**, NOT a single Lance primitive: (1) per-table Lance `commit_staged` for the data write, (2) `__manifest` row-level CAS via `ManifestBatchPublisher` for cross-table ordering, (3) the open-time recovery sweep for the residual gap between (1) and (2). All three layers ship; the five migrated writers (`MutationStaging::commit_all`, `schema_apply`, `branch_merge`, `ensure_indices`, `optimize_all_tables`) write a `__recovery/{ulid}.json` sidecar before Phase B and delete it after Phase C. Under their final schema → branch → table gates, mutation/load, SchemaApply, BranchMerge, and EnsureIndices prove every existing effect target still equals its manifest pin before arming; first-touch refs or datasets are created only after the sidecar is durable. The next `Omnigraph::open` (gated on `OpenMode::ReadWrite`) runs the sweep in `db/manifest/recovery.rs`: classify, decide all-or-nothing per sidecar, roll forward via single `ManifestBatchPublisher::publish` or roll back via `Dataset::restore` followed by a manifest publish of the restored version (so both directions converge to `manifest == HEAD` — no residual drift), and record an internal audit row in `_graph_commit_recoveries.lance`. Schema-v3 Mutation/Load, schema-v4 BranchMerge, and schema-v7 SchemaApply roll-forward preserve the interrupted writer's fixed commit lineage and actor. SchemaApply v7 captures native main authority + accepted schema identity, pre-mints exact existing-table `Overwrite` and strict read-version-zero AddType/RenameType create identities, and confirms the complete registration/update/tombstone delta only after every table effect and schema staging file is durable. `Armed` rolls back; `EffectsConfirmed` rolls forward only under the captured token. The fixed manifest outcome lands before schema staging promotion. Rollback reclaims an exact owned first-touch path; an unregistered foreign first-touch winner is preserved but never adopted, while foreign movement on a manifest-owned table or an owned effect buried by a same-table winner fails closed. Metadata-only applies use the same protocol with an empty effect set. Query, export, graph-index, and blob reads capture their snapshot + catalog under the same process-local schema gate, so they cannot observe the manifest-before-catalog window on the applying handle. Read-only open performs no recovery writes but refuses a fixed SchemaApply manifest outcome until the matching schema identity is live. The reader remains backward-compatible with schema-v5 bridge files under their original target-hash/Phase-C-confirmation and loose-classification semantics. EnsureIndices v6 retains loose effect classification but pre-mints rollback lineage and persists the rollback audit plan before restore, making compensation retry-exact. Other rollback and legacy recovery commits use `omnigraph:recovery`. BranchMerge v4 also owns first-touch refs, pre-mints each table's exact ordered data-transaction chain with zero transparent conflict retries, carries pointer-only updates in its complete confirmed manifest delta, and recognizes an interrupted compensation restore on restart. There is currently no public CLI query for the recovery-audit table, and ordinary commit history is not a complete recovery enumeration. The write entry points (`load_as`, `mutate_as`, `apply_schema_as`, `branch_merge_as`) and `refresh` additionally run an in-process roll-forward-only heal, serialized against same-process live writers through the shared root-scoped gate manager, so a long-lived server converges on its next write without restart; only rollback-eligible sidecars still defer to the next read-write open (a future background reconciler's goal). These gates are process-local: exact v7 ownership prevents false adoption but does not make destructive recovery safe against a still-running writer in another process; multi-process recovery and live schema capture still need a distributed fence. Engine writes route through a sealed `TableStorage` trait (`db.storage()`) exposing only `stage_*` + `commit_staged` + reads; the sole inline-commit residual (`create_vector_index`) is split onto a separate sealed `InlineCommitResidual` trait reached via `db.storage_inline_residual()` (MR-854), so the default surface cannot couple a write with a HEAD advance — §1 holds by construction. `delete` migrated to the staged path in MR-A (`stage_delete` via Lance 7.0 `DeleteBuilder::execute_uncommitted`, [#6658](https://github.com/lance-format/lance/issues/6658)); `create_vector_index` remains inline until the exact EnsureIndices adapter migrates the usable beta.21 one-segment full-table staged shape, while [#6666](https://github.com/lance-format/lance/issues/6666) remains relevant to generic multi-segment exact publication; `LoadMode::Overwrite` uses Lance `Overwrite` staged transactions. |
+| Atomic single-dataset commits | ✅ | **Multi-table publish via three layers**, NOT a single Lance primitive: (1) per-table Lance `commit_staged` for the data write, (2) `__manifest` row-level CAS via `ManifestBatchPublisher` for cross-table ordering, (3) the open-time recovery sweep for the residual gap between (1) and (2). All three layers ship; the five migrated writers (`MutationStaging::commit_all`, `schema_apply`, `branch_merge`, `ensure_indices`, `optimize_all_tables`) write a `__recovery/{ulid}.json` sidecar before Phase B and delete it after Phase C. Under their final schema → branch → table gates, mutation/load, SchemaApply, BranchMerge, and EnsureIndices prove every existing effect target still equals its manifest pin before arming; first-touch refs or datasets are created only after the sidecar is durable. The next `Omnigraph::open` (gated on `OpenMode::ReadWrite`) runs the sweep in `db/manifest/recovery.rs`: classify, decide all-or-nothing per sidecar, roll forward via single `ManifestBatchPublisher::publish` or roll back via `Dataset::restore` followed by a manifest publish of the restored version (so both directions converge to `manifest == HEAD` — no residual drift), and record an internal audit row in `_graph_commit_recoveries.lance`. Schema-v3 Mutation/Load, schema-v4 BranchMerge, schema-v7 SchemaApply, and schema-v8 EnsureIndices roll-forward preserve the interrupted writer's fixed commit lineage and actor. SchemaApply v7 captures native main authority + accepted schema identity, pre-mints exact existing-table `Overwrite` and strict read-version-zero AddType/RenameType create identities, and confirms the complete registration/update/tombstone delta only after every table effect and schema staging file is durable. `Armed` rolls back; `EffectsConfirmed` rolls forward only under the captured token. The fixed manifest outcome lands before schema staging promotion. Rollback reclaims an exact owned first-touch path; an unregistered foreign first-touch winner is preserved but never adopted, while foreign movement on a manifest-owned table or an owned effect buried by a same-table winner fails closed. Metadata-only applies use the same protocol with an empty effect set. Query, export, graph-index, and blob reads capture their snapshot + catalog under the same process-local schema gate, so they cannot observe the manifest-before-catalog window on the applying handle. Read-only open performs no recovery writes but refuses a fixed SchemaApply manifest outcome until the matching schema identity is live. The reader remains backward-compatible with schema-v5 bridge files under their original target-hash/Phase-C-confirmation and loose-classification semantics. EnsureIndices v8 captures exact branch/schema authority, fixed original/rollback lineage, one pre-minted mixed BTREE/FTS/vector CreateIndex transaction per table, the complete confirmed pointer delta, and exact first-touch ref identity; `Armed` rolls back and `EffectsConfirmed` rolls forward only under the captured token. Schema-v6 EnsureIndices files remain readable under their original loose classification and fixed-rollback semantics, never reinterpreted as v8 ownership. Other rollback and legacy recovery commits use `omnigraph:recovery`. BranchMerge v4 also owns first-touch refs, pre-mints each table's exact ordered data-transaction chain with zero transparent conflict retries, carries pointer-only updates in its complete confirmed manifest delta, and recognizes an interrupted compensation restore on restart. There is currently no public CLI query for the recovery-audit table, and ordinary commit history is not a complete recovery enumeration. The write entry points (`load_as`, `mutate_as`, `apply_schema_as`, `branch_merge_as`) and `refresh` additionally run an in-process roll-forward-only heal, serialized against same-process live writers through the shared root-scoped gate manager, so a long-lived server converges on its next write without restart; only rollback-eligible sidecars still defer to the next read-write open (a future background reconciler's goal). These gates are process-local: exact v7/v8 ownership prevents false adoption but does not make destructive recovery safe against a still-running writer in another process; multi-process recovery and live schema capture still need a distributed fence. Engine writes route through a sealed `TableStorage` trait (`db.storage()`) exposing only `stage_*` + `commit_staged` + reads. `stage_create_indices` batches every missing BTREE, FTS, and full-table vector artifact for one table into one staged CreateIndex transaction; `InlineCommitResidual` and `storage_inline_residual()` are removed, so the surface cannot couple writing artifacts with a HEAD advance — §1 holds by construction. `delete` migrated to the staged path in MR-A (`stage_delete` via Lance 7.0 `DeleteBuilder::execute_uncommitted`, [#6658](https://github.com/lance-format/lance/issues/6658)); the beta.21 one-segment full-table vector shape now stages through the exact EnsureIndices v8 adapter, while [#6666](https://github.com/lance-format/lance/issues/6666) remains relevant only to generic multi-segment exact publication; `LoadMode::Overwrite` uses Lance `Overwrite` staged transactions. |
 | Compaction (`compact_files`) + reindex (`optimize_indices`) | ✅ | `omnigraph optimize` orchestrates over all node/edge tables, bounded concurrency; per table runs `compact_files` **then Lance `optimize_indices`** (folds appended/rewritten fragments back into existing indexes — incremental merge, not retrain) and **publishes the resulting version to `__manifest`** (so the manifest tracks the Lance HEAD — required for reads to observe the work and for schema apply / strict writes to pass their HEAD-vs-manifest precondition), under the per-`(table, main)` write queue with `SidecarKind::Optimize` recovery coverage spanning both ops. Because every published table pointer also advances shared `graph_head:main`, Optimize's final recovery barrier is branch-wide: after the graph-wide fast probe it acquires the process-local main gate, relists, and rejects every main-target sidecar plus graph-global SchemaApply before reading table HEADs or arming its own sidecars. It retains that gate across its internally parallel table phase, and through final physical `__manifest` compaction so a new main recovery intent cannot arm before raw manifest movement finishes. The coarse legacy-adapter cost is that same-process sidecar-enrolled main writers wait for the entire Optimize run, although Optimize's own table tasks remain parallel. It **commits even with no compaction work if index coverage is stale**; **refuses on an unrecovered graph**; **skips uncovered HEAD > manifest drift** with `DriftNeedsRepair`; and **compacts blob-bearing tables** (the pre-9 `LANCE_SUPPORTS_BLOB_COMPACTION` skip was removed once Lance 8.0.0+ shipped blob-v2 compaction — see [docs/dev/invariants.md](docs/dev/invariants.md) Known Gaps) |
 | Repair uncovered drift | — | `omnigraph repair` explicitly classifies uncovered table `HEAD > manifest` drift: verified maintenance drift (`ReserveFragments`/`Rewrite`) can be published with `--confirm`; suspicious or unverifiable drift requires `--force --confirm`. Sidecar-covered crash residuals still recover automatically on open. |
 | Cleanup (`cleanup_old_versions`) | ✅ | `omnigraph cleanup` derives requested `--keep` / `--older-than` cutoffs from each table's available versions; Lance refs plus OmniGraph's live-lazy-branch and recovery floors may retain additional versions. It fails closed on unopenable pins, recovery intent, or uncovered main-table HEAD drift |
-| BTREE / inverted (FTS) / vector indexes | ✅ | `@index`/`@key` declares intent; the physical index is derived state that never fails a logical op. Built per column through one chokepoint (`build_indices_on_dataset_for_catalog`, type-dispatched by `node_prop_index_kind`: enum + orderable scalar → BTREE, free-text String → FTS, Vector → vector); idempotent; lazy across branches. **Schema apply and mutation/load build no indexes inline**: the latter publish only their exact data effects, leaving physical intent pending. `ensure_indices`/`optimize` materializes declared-but-missing indexes, restores fragment coverage, and continues to report untrainable Vector columns as pending. |
+| BTREE / inverted (FTS) / vector indexes | ✅ | `@index`/`@key` declares intent; the physical index is derived state that never fails a logical op. Built per column through one chokepoint (`build_indices_on_dataset_for_catalog`, type-dispatched by `node_prop_index_kind`: enum + orderable scalar → BTREE, free-text String → FTS, Vector → vector); idempotent; lazy across branches. **Schema apply and mutation/load build no indexes inline**: the latter publish only their exact data effects, leaving physical intent pending. `ensure_indices` materializes declared-but-missing indexes through one staged mixed CreateIndex transaction per table under its exact schema-v8 authority/lineage/delta protocol, while continuing to report untrainable Vector columns as pending. `optimize` separately restores fragment coverage and remains on its legacy recovery adapter. |
 | `merge_insert` upsert | ✅ | `LoadMode::Merge`, mutation `update`/`insert`/`delete` lowering |
 | Vector search | ✅ | `nearest()` query op; embedding pipeline (Gemini / OpenAI clients); `@embed` in schema |
 | Full-text search | ✅ | `search/fuzzy/match_text/bm25` query ops |

crates/omnigraph/src/db/graph_coordinator.rs

@@ -361,6 +361,7 @@ impl GraphCoordinator {
             .map(|id| id.map(SnapshotId::new))
     }
 
+    #[cfg(test)]
     pub(crate) async fn commit_updates_with_actor(
         &mut self,
         updates: &[SubTableUpdate],

crates/omnigraph/src/db/manifest.rs

@@ -43,11 +43,11 @@ pub(crate) use recovery::{
     RecoverySchemaApplyEffect, RecoverySchemaApplyEffectKind, RecoverySidecar,
     RecoverySidecarHandle, RecoveryTableUpdateSlot, SidecarKind, SidecarTablePin,
     SidecarTableRegistration, SidecarTombstone, confirm_branch_merge_sidecar_phase_b,
-    confirm_occ_sidecar_phase_b, confirm_schema_apply_sidecar_v7, delete_sidecar,
-    ensure_read_only_schema_coherent, heal_pending_sidecars_roll_forward, list_sidecars,
-    new_branch_merge_sidecar, new_ensure_indices_sidecar, new_occ_sidecar,
-    new_schema_apply_sidecar_v7, new_sidecar, recover_manifest_drift,
-    schema_apply_serial_queue_key, write_sidecar,
+    confirm_ensure_indices_sidecar_v8, confirm_occ_sidecar_phase_b,
+    confirm_schema_apply_sidecar_v7, delete_sidecar, ensure_read_only_schema_coherent,
+    heal_pending_sidecars_roll_forward, list_sidecars, new_branch_merge_sidecar,
+    new_ensure_indices_sidecar_v8, new_occ_sidecar, new_schema_apply_sidecar_v7, new_sidecar,
+    recover_manifest_drift, schema_apply_serial_queue_key, write_sidecar,
 };
 pub use state::SubTableEntry;
 #[cfg(test)]