fix(sync): damp metadata-pending VM reconcile scans

[branarakic]

What

• apply the existing generation/topology-gated negative cache to batched VM reconciliation misses
• avoid DHT-wide connection priming while scanning batched exact-recovery targets
• revalidate once immediately after an exact fetch, then clear or refresh the cache from the authoritative result
• distinguish no-SWM and metadata-pending retry metrics

Why

A node with historical VM content but missing transaction provenance repeatedly re-ran the same expensive metadata verification for every sweep slice. On the local testnet edge this produced hundreds of identical cannot-repair-without-provenance attempts even though neither local evidence nor connected recovery peers had changed.

The cache remains fail-open: a local graph generation change, recovery-peer topology change, or bounded expiry causes an authoritative retry. It never synthesizes provenance and never promotes unverifiable VM state.

Validation

• pnpm --filter @origintrail-official/dkg-agent build
• agent unit lane: 144 files passed; 1,888 tests passed; 5 skipped; 0 failed
• focused VM reconcile lane: 69 tests passed

Live local-node validation will be added before this draft is marked ready. No core rollout is authorized by this PR.

[branarakic]

Live validation refinement (2026-07-24)

The first local soak correctly exposed that the original one-minute first backoff was too short on a loaded node: a traversal across the subscribed CG set outlived the timer, and the cursor retried the first historical metadata-pending assets before any negative-cache deferral could occur. I did not count that run as passing evidence.

Follow-up commit cc4b1280b keeps metadata-provenance misses at the existing bounded ten-minute ceiling (deterministic jitter gives an 8–10 minute timer), while the existing correctness gates still fail open immediately when:

• local SWM generation/content changes;
• candidate namespace resolution changes;
• the connected recovery-peer topology changes; or
• authoritative revalidation succeeds.

Validation on the isolated PR branch:

• focused core-fills-gap.test.ts: 69/69 passed;
• agent build (tsc, type tests, package-root): passed;
• full agent unit lane: 144 files, 1,888 passed, 5 skipped, 0 failed;
• git diff --check: passed.

The new assertion pins a first retry horizon that survives a multi-minute loaded traversal. Live node validation of the updated commit is next; the PR remains draft until the natural cursor revisit produces cache-hit evidence without hiding new graph or topology evidence.

[branarakic]

Follow-up: loaded-node retry history

Live validation of the previous 10-minute cap showed it did not survive a full subscribed-CG traversal: the local edge reached 389 metadata-pending rows across 9 CGs, 326 already timer-eligible again, with retry histories up to 5. Logs continued to perform exact fetches and re-defer unchanged assets, with no negative-cache hits.

Commit 027199fb4 keeps ordinary no_swm behavior unchanged, but gives metadata_pending a retained, bounded 10 -> 20 -> 40 -> 60 minute retry curve. New local evidence, chain roots, and peer-topology changes still invalidate entries immediately. The one-hour cap is environment-overridable.

Validation:

• focused VM reconcile lane: 69/69 passed
• agent build (TypeScript, type tests, package-root): passed
• isolated memory-layers aggregate failure: passed when run alone, confirming suite-order contamination
• aggregate agent lane: 278 files passed, 2 skipped, 4 failed; 3232 tests passed, 10 skipped, 9 failed
• isolated rerun reproduced five pre-existing failures in untouched code: one legacy two-argument mock expectation now receives trailing undefined arguments, plus four shutdown fixtures lacking finalizationRuntime
• of the three publish tests that timed out in the long aggregate run, two passed in isolation; the off-node pre-signed test still times out during its two-node setup and is outside this three-file diff

Live gate remains open: this PR stays draft until the new commit produces real detail="negative-cache" hits and materially lowers redundant provenance/exact-fetch churn on the preserved local edge store. No core rollout and no branch merge.

[branarakic]

Live testnet-edge validation update (local validation layer only; no canary/core deployment):

• running exact combined validation commit: f99132880c85c606b41039aa0cc2989d5fc963e0
• focused reconcile suite: 69/69 passed
• agent build, type tests, and package-root tests: passed
• after the initial conversion pass, the latest observed window produced 73 VM negative-cache reuses, 86 exact-fetch log entries, 120 queued backpressure events, and 0 backpressure rejections
• the CG-scoped generation correctly stopped unrelated context-graph writes from invalidating all metadata-pending entries
• restart remains fail-open: process-local generation entries are not treated as durable negative evidence

The branch is now updated to aa30c25. This PR remains draft and must not be merged until refreshed CI/review and the offline receiver completeness checkpoint can be resumed.

[branarakic]

One-hour local testnet-edge soak completed successfully on the combined local validation artifact f99132880c85c606b41039aa0cc2989d5fc963e0 (this is not a canary/core deployment).\n\nFinal gate evidence:\n- uptime at gate: 3,625,975 ms\n- exact running commit: f9913288\n- 10 connected peers; relay connected\n- admission rejectedTotal: 0\n- all 4 configured RPCs healthy; no RPC exhaustion\n- final 11-minute window: 43 VM negative-cache reuses, 80 exact-fetch log entries, 46 queued backpressure events, 0 backpressure rejections, 0 runtime errors\n- Windows and SPARQL CI checks are green\n\nDistributed completeness remains explicitly unresolved because the second edge receiver is intentionally offline at its preserved SWM 391/400 and VM 390/400 checkpoint. No merge or core rollout is authorized.