What I see on 0.9.28
indexPersistence.scheduleSave() has exactly two call sites, both in the bootstrap:
- after
rebuildIndex() when the loaded index was empty
- after the legacy memory backfill into BM25
Nothing schedules a save when an observation is compressed and added to the index at runtime, and there is no periodic flush. The only other save is the one in the shutdown handler.
So between boot and a clean shutdown, the persisted snapshot never advances. On this install, 30 minutes and several hundred observations after start, no shard under mem:index:bm25:* had been rewritten, while mem:obs:* and mem:memories were being written continuously. Search was fine — the in-memory index has everything — which is exactly why this is easy to miss.
Why it matters
Any exit that does not run the shutdown handler loses the whole runtime delta:
- SIGKILL (systemd
TimeoutStopSec expiring is a realistic one — viewerServer.close() can wait on keep-alive connections)
- OOM kill
- host reset / power loss
- container stop that outruns the grace period
After that, boot restores the last snapshot, which is not empty, so rebuildIndex() does not run — the gap silently survives every subsequent restart. The user-visible symptom is "search stopped finding recent work", with a healthy-looking server.
Repro
- Start with a non-empty persisted index.
- Generate observations for a few minutes.
ls -lt the mem:index:bm25:* shards — mtimes are still from boot.kill -9 the server, restart: the runtime observations are missing from BM25/vector search but present in KV.
Suggested fix
Whichever of these fits your design:
- call
scheduleSave() where the index is mutated at runtime (the debounce already coalesces bursts), and/or
- a periodic flush, e.g.
setInterval(() => indexPersistence.scheduleSave(), 300_000).unref() next to the existing bootstrap wiring
I am running the periodic-flush version locally (5 min, .unref()ed) and it is one line. Happy to send it as a PR — say which shape you prefer, or if both.
Related: #1179 (no way to rebuild the index without a restart) — the two together are why a stale index can persist indefinitely once it happens.
What I see on 0.9.28
indexPersistence.scheduleSave()has exactly two call sites, both in the bootstrap:rebuildIndex()when the loaded index was emptyNothing schedules a save when an observation is compressed and added to the index at runtime, and there is no periodic flush. The only other save is the one in the shutdown handler.
So between boot and a clean shutdown, the persisted snapshot never advances. On this install, 30 minutes and several hundred observations after start, no shard under
mem:index:bm25:*had been rewritten, whilemem:obs:*andmem:memorieswere being written continuously. Search was fine — the in-memory index has everything — which is exactly why this is easy to miss.Why it matters
Any exit that does not run the shutdown handler loses the whole runtime delta:
TimeoutStopSecexpiring is a realistic one —viewerServer.close()can wait on keep-alive connections)After that, boot restores the last snapshot, which is not empty, so
rebuildIndex()does not run — the gap silently survives every subsequent restart. The user-visible symptom is "search stopped finding recent work", with a healthy-looking server.Repro
ls -ltthemem:index:bm25:*shards — mtimes are still from boot.kill -9the server, restart: the runtime observations are missing from BM25/vector search but present in KV.Suggested fix
Whichever of these fits your design:
scheduleSave()where the index is mutated at runtime (the debounce already coalesces bursts), and/orsetInterval(() => indexPersistence.scheduleSave(), 300_000).unref()next to the existing bootstrap wiringI am running the periodic-flush version locally (5 min,
.unref()ed) and it is one line. Happy to send it as a PR — say which shape you prefer, or if both.Related: #1179 (no way to rebuild the index without a restart) — the two together are why a stale index can persist indefinitely once it happens.