Unbounded sourceObservationIds / sourceMemoryIds are what push collections past the worker limit in #1142 / #1124

[krickwix]

Summary

sourceObservationIds on graph nodes and sourceMemoryIds on insights grow without any cap. In both collections the provenance array ends up as ~97% of all bytes, which is what pushes a collection past the worker limit described in #1142 and triggers the blast radius described in #1124.

Those two issues cover the symptom well. This one is about why collections get big enough to hit it in the first place — the size is not a function of how much you actually stored.

I hit this on two separate collections on one install and reset both, so there are before/after numbers below. The reflect measurement is probably the most useful part for #655.

The two fields

Collection	Field	Records	Field share of bytes	Worst single record
mem:insights	sourceMemoryIds	8,969	97.0%	1,545 ids
mem:graph:nodes	sourceObservationIds	8,894	97.3%	8,750 ids — 256 KB in one node

4,562,366 memory ids and 2,623,559 observation ids respectively. Strip the provenance and the same data is a few MB.

The worst node is a single file node for a MEMORY.md path. Nothing caps how many times a hot path can be re-observed.

Two different mechanisms

1. mergeNode / mergeEdge — monotonic union, never pruned:

sourceObservationIds: [...new Set([
  ...existing.sourceObservationIds,
  ...incoming.sourceObservationIds,
  ...obsIds
])],

Re-observing a node only ever adds. A frequently-touched file node grows unboundedly.

2. mem::reflect — cluster-proportional at creation:

sourceMemoryIds: cluster.factIds,

Not accumulation (reinforcements was 0 on my largest record) but it scales with corpus size. At 25,408 semantic memories a single cluster contributed 1,545 ids.

Measured impact, and a data point for #655

#655 attributes memory_reflect timeouts to sequential N+1 KV operations. That's plausible, but on my install collection size dominated. mem::reflect enumerates six collections (graphNodes, graphEdges, semantic, lessons, crystals, insights).

I reset the two bloated collections without changing anything else — no code change, no config change, same cluster count:

	before	after
mem::reflect	Invocation timeout after 180000ms, failing 5 of 6 runs	success in 15.2 s
clusters processed	1, with 6 skipped	4, with 0 skipped
memory_diagnose (default)	HTTP 500 Invocation stopped	HTTP 200 in 0.43 s
largest value in store	132.2 MiB	31.5 MB

The 6 skipped clusters were being dropped under time pressure, so the bloat was silently degrading reflect's output quality as well as its reliability — not just making it slow.

That's a >10x improvement from removing bytes that were pure provenance. Whatever is done about sequential KV ops in #655, capping these arrays looks like the larger lever.

Why a record-count ceiling won't catch it

graph-snapshot-rebuild already guards this failure mode:

REBUILD_SAFE_NODE_CEILING = 25e3
// "…heartbeat-crashes the worker on corpora past the iii state response budget (~25K nodes)."

But the ceiling counts records while the failure is driven by bytes, and unbounded provenance decouples the two. My mem:insights broke at 8,969 records — about a third of the ceiling — because provenance pushed each record from ~700 bytes to ~15 KB average.

Any count-based bound will keep missing this. The guard needs to be on serialized size.

A note on the 16 MiB threshold from #1142

My numbers don't line up cleanly with a flat 16 MiB limit, so this may be a second data point worth having. Reads that succeeded while the oversized collection failed:

Collection	On-disk size	kv.list
mem:semantic	31.5 MB	OK (0.53 s)
mem:graph:nodes	76 MB	OK — but see below
mem:insights	132.2 MiB	fails, ~2.0 s

mem:semantic at 31.5 MB is well past 16 MiB and enumerated fine, so either the internal KV response is serialized more compactly than the on-disk JSON, or the internal path has a different budget than the REST response path #1142 bisected. mem:graph:nodes may not be a real counterexample — graph-query appears to serve from paginateFromSnapshot when no nodeId is given, so it likely never enumerated.

I could not pin the exact internal limit. The max_frame strings in the iii binary belong to its AMQP and h2 dependencies, not the worker transport.

Suggested fixes

1. Cap the arrays — a ring buffer of the most recent N ids (N≈50–100). Modelled against my data, capping sourceMemoryIds at 100 takes mem:insights from 132.2 MiB to 26.4 MiB — 20% of current — with every insight's content, confidence and tags intact.
2. Store provenance out-of-line, so a hot node's history doesn't inflate the collection that every unrelated read has to load.
3. Guard on serialized bytes, not record count, wherever REBUILD_SAFE_NODE_CEILING is applied.

Operator-facing gaps found along the way

These made the problem much harder to diagnose than it needed to be, and each is small to fix.

There is no way to disable the reflect tier. AGENTMEMORY_REFLECT looks like the switch and is not — it gates mem::slot-reflect via isReflectEnabled() and additionally requires AGENTMEMORY_SLOTS=true. The .env.example comment reads "Periodically auto-synthesize lessons from memories", which describes the consolidation tier and sends operators to the wrong flag. mem::consolidate-pipeline triggers the tier unconditionally:

if (tier === "all" || tier === "reflect") try {
  results.reflect = await sdk.trigger({ function_id: "mem::reflect", … });

No getEnvVar call anywhere in that function. Verified: with AGENTMEMORY_REFLECT=false live in /proc/<pid>/environ, the next cycle still logged "reflect":{…,"success":true}. The only lever is CONSOLIDATION_ENABLED, which also disables semantic, procedural and decay. Request: a per-tier opt-out such as CONSOLIDATION_TIERS=semantic,procedural,decay, and either rename the slots flag or fix that comment.

memory_diagnose's description lists 8 categories; ALL_CATEGORIES has 14. lessons, summaries, semantic, procedural, crystals and insights are undocumented. Passing categories explicitly is the only way to work around a single poisoned category, so the description being wrong hides the workaround. Bisecting that list is how I found insights was the culprit.

MCP tool 500s emit no log line. Nothing about any of this reached the journal. /agentmemory/livez stayed at ~4 ms throughout — the daemon was healthy, one collection was not. Over 15 hours the only visible symptom was two Reflect tier failed lines. Journal-based monitoring is completely blind to this class of failure.

Workaround, for anyone who lands here

No supported prune exists. mem::graph-reset only resets the snapshot; mem::export/mem::import round-trips the whole store through the same bottleneck (and see #1142). Hand-editing the .bin is not viable — the tail is not a stable JSON+trailer layout: length varies between files, and the back-pointer that holds for graph:nodes and crystals does not hold for insights.

What worked is a file-level reset, since the store treats a missing collection file as healthy-and-empty:

systemctl stop agentmemory
mv ~/data/state_store.db/'mem%3Ainsights.bin' ~/backup/
systemctl start agentmemory

For the graph, all six blobs must move together — nodes, edges, snapshot, name-index, node-degree, edge-key. Leaving name-index resolves names to nonexistent nodes; leaving edge-key suppresses re-creation of edges.

Both collections rebuild incrementally from new extracts. Insights regrow at roughly 1.7 MB/day on my install, so this is a recurring chore rather than a fix — hence the cap request above.

Environment

• @agentmemory/agentmemory 0.9.28 (daemon); @agentmemory/mcp shim resolving 0.9.21 from the npx cache
• iii engine 0.11.2 (pinned), store_method: file_based
• Node 24.15.0, WSL2 (Linux 6.6.87.2-microsoft-standard-WSL2)
• CONSOLIDATION_ENABLED=true, GRAPH_EXTRACTION_ENABLED=true, AGENTMEMORY_AUTO_COMPRESS=true
• Scale: 8,969 insights, graph 8,894 nodes, 25,408 semantic memories, 1,128 procedural, 186 summaries

Related: #1142 (16 MiB response limit), #1124 (worker unregistration blast radius), #655 (reflect timeouts), #890 (export pagination), #1157 (graph provenance staleness).

[Tomauskasz]

The store sizes in this report also expose a separate lifecycle gap. Current consolidation merges Semantic facts only by exact case-insensitive text and Procedures only by exact case-insensitive name. Its decay tier floors strength at 0.1 and deletes neither store. Procedural Memory is not covered by the retention scorer.

I opened #1191 for a bounded Semantic and Procedural Memory retention path. That does not replace the provenance caps requested here: provenance arrays can break a store even when record counts are reasonable, while retention addresses total corpus growth and recall noise.