Mental model refresh OOM-kills the container at scale: a SINGLE refresh = 17.7GB RSS on a 40K-node / 815K-link bank

[paulsto]

Mental model refresh OOM-kills the container at scale: a SINGLE refresh = 17.7GB RSS on a 40K-node / 815K-link bank

Summary

On a single large bank (~40K memory nodes, ~815K links, 15.6K observations), one refresh_mental_model operation deterministically OOM-kills the hindsight-api process. The kernel reports Out of memory: Killed process (hindsight-api) anon-rss: ~17.5GB within ~60-90s of the refresh starting. This happens with tick-triggered refreshes and with a single manually-triggered refresh in isolation. Container memory is unbounded (no --memory cap), swap fills (3.8/3.8Gi), and the container crash-loops because the worker re-claims the pending refresh op on every restart.

This is the memory-side of the problem in #3290 and the mechanism described in #3122, but I could not find the OOM-at-scale report filed anywhere, so here is the reproduction data.

Environment

• Hindsight v0.9.0 (image ghcr.io/vectorize-io/hindsight:latest, built 2026-08-07)
• Docker, no --memory limit, --shm-size=2g
• LLM: DeepSeek v4-flash via API (HINDSIGHT_API_LLM_PROVIDER=deepseek), HINDSIGHT_API_LLM_MAX_CONCURRENT=64
• Host: 8-core / 19GiB Linux VM
• Bank: hermes-main, 40,619 nodes (23,471 world / 1,500 experience / 15,648 observation), 815,676 links, 1,009 documents
• 13 mental models / knowledge pages (12 knowledge pages + 1 mental model), all with refresh_cron: "0 6 * * *", delta mode, refresh_after_consolidation: false

Controlled test matrix (all single-refresh, nothing else queued)

Each test: container recreated clean (tick disabled, HINDSIGHT_API_MENTAL_MODEL_REFRESH_TICK_SECONDS=0), all prior refresh ops marked failed in pg0, exactly ONE refresh submitted via POST /v1/default/banks/{bank}/mental-models/{id}/refresh, memory sampled every 30s.

Test	Config	t+30s RSS	Peak RSS	Result
1	baseline (chunks on, default recall)	12.2 GiB	17.7 GiB anon	kernel OOM kill
2	HINDSIGHT_API_RECALL_INCLUDE_CHUNKS=false	12.3 GiB	17.5 GiB anon	kernel OOM kill
3	chunks off + HINDSIGHT_API_RECALL_MAX_TOKENS=512 (default 2048)	1.3 GiB	17.4 GiB anon	kernel OOM kill

Kernel log (same signature all three runs):

Out of memory: Killed process (hindsight-api) total-vm:33.7GB, anon-rss:17.5GB

Test 3 is the informative one: capping the recall text budget delayed the initial climb (1.3GiB vs 12GiB at t+30s) but the process still blew past 17GiB before the refresh could complete. That matches #3122's mechanism — the serialized tool-result envelope (id, chunk_id, document_id, context, entities, metadata, scores, tags, type per observation) is not bounded by the recall token budget, so on a bank with 815K links the graph-expanded recall + envelope serialization in the reflect agent loop accumulates until the kernel kills the process.

Also found: the refresh concurrency knob appears to be dead code in v0.9.0

HINDSIGHT_API_MENTAL_MODEL_REFRESH_CONCURRENCY (default 8, documented as "Max concurrent mental model refreshes") is read into config (config.py L533/L1126/L3480) but never referenced by any engine file — worker.py, maintenance.py, memory_engine.py don't use it. Setting it to 1 changed nothing: all 13 due refreshes still ran concurrently. If the intent is a working cap, it needs wiring; if it's meant to be removed, the docs should say so.

Impact

• Any bank that grows past the point where a single refresh's serialized recall envelope fits in container memory will crash-loop whenever refreshes are due (all models sharing refresh_cron become due simultaneously, so the whole sweep fires at once).
• This is not just an agent-memory edge case: the same operation that powers knowledge pages (same engine, same refresh behaviour per the FAQ) makes the daily refresh cycle fatal at scale.
• Community reports predate this (e.g. hindsight-api Python process memory grows unbounded in container (~1GB in <1 hour) #996, April 2026: unbounded hindsight-api growth; the accepted workaround there was a hard --memory cap so the OOM killer restarts the container — which loses the refresh and crash-loops if the op re-claims).

Suggested fixes (in priority order)

1. Ship the feat(engine): two-tier mental-model refresh - a deterministic fast path that serves most refreshes in one LLM call or none, with the agentic loop as escape hatch #3290 two-tier refresh (deterministic fast path, agentic loop as escape hatch) — the fast path should be bounded-memory by construction.
2. Budget the tool-result envelope against max_tokens (per reflect: tool-result envelope isn't counted against max_tokens (~6x inflation) and forced synthesis drops oversized blocks whole — confident 'no information' refusal with 503 citations (post-#668 residual) #3122), not just the observation text — so a single refresh cannot accumulate gigabytes.
3. If feat(engine): two-tier mental-model refresh - a deterministic fast path that serves most refreshes in one LLM call or none, with the agentic loop as escape hatch #3290 is weeks out: a working refresh-concurrency cap plus an in-flight guard so a stale sweep can't re-queue a full wave after a crash.
4. If neither lands soon: at minimum, document the failure mode + --memory cap band-aid on the perf page.

Happy to provide more data — bank dumps, pg0 queries, full docker logs, or run a targeted repro against a build that includes a fix.

[nicoloboschi]

Thanks for the incredibly detailed writeup — the controlled matrix, the kernel signatures, and the pointer to the REFRESH_CONCURRENCY dead code are all really useful, and you're right that it's dead: mental_model_refresh_concurrency is read into config but never referenced by any engine file. We'll fix that separately.

Before we chase the memory number down, we'd like to pin the attribution a bit tighter, because a couple of details in the setup mean the reported figure might not be "one refresh in isolation":

1. No pre-refresh baseline. The table starts at t+30s — is there an idle RSS reading right after startup, before the refresh is submitted? That baseline isn't zero: the defaults are HINDSIGHT_API_EMBEDDINGS_PROVIDER=local and HINDSIGHT_API_RERANKER_PROVIDER=local, so a freshly-started container already has the embedder and the cross-encoder reranker resident in-process. We'd like to subtract that.

2. Other background loops were still running. Setting HINDSIGHT_API_MENTAL_MODEL_REFRESH_TICK_SECONDS=0 only silences the refresh tick. The consolidation reconcile loop (HINDSIGHT_API_CONSOLIDATION_RECONCILE_INTERVAL_SECONDS) and the retain batch poller (HINDSIGHT_API_RETAIN_BATCH_POLL_INTERVAL_SECONDS), plus graph maintenance, are still ticking. On a bank this size (40K nodes / 815K links / 15.6K observations), consolidation firing concurrently could account for a meaningful share of that RSS, and 30s sampling can't separate it from the refresh. Could you re-run with those intervals also set to 0 (or attach the docker logs for the window, so we can see whether consolidation/GM ran alongside)?

3. Embeddings/reranker provider — local vs TEI? Local adds in-process model RSS that a TEI deployment wouldn't have; knowing which you're on helps us reason about the baseline.

One more data point that's already interesting: you mention the reported synthesis was ~730 input tokens with 503 citations. If the 17GB were the serialized recall envelope reaching the LLM (the #3122 mechanism), we'd expect token usage to be enormous, not 730 — so the memory is very likely in Python-side intermediate objects during retrieval/fusion, upstream of the token budget. That's also consistent with your Test 3: capping RECALL_MAX_TOKENS=512 knocked the t+30s spike down 12GiB→1.3GiB but the run still climbed to 17GiB, which the token budget shouldn't have let happen if the envelope were the whole story.

If, after disabling the other loops and recording an idle baseline, a genuinely isolated single refresh still climbs from ~1.5GB to ~17GB, that would be a clean repro and we'll add streaming, OOM-survivable instrumentation (per-arm row counts + RSS logged to stdout at each stage boundary) so we can localize exactly which retrieval arm/iteration crosses the GB thresholds — the current traces are persisted only at the end of a refresh, so an OOM-killed run writes nothing, which is why there's no visibility today.

Thanks again — happy to keep digging once we have the isolated numbers.