Consolidate long-session heap OOM and autocompact recovery failures

[jatmn]

Summary

Several reports describe the same broad failure family: long-running OpenClaude sessions can either exhaust the Node/V8 heap or stop after context-window failures even though auto-compact is expected to keep the workflow alive.

This issue is the canonical tracker for consolidating the related reports:

• Memory error and no autocompact #1046
• Auto-recover from context-window errors after large tool results #1105
• windows oom #1148
• bug: Node/V8 process crashes with JavaScript heap out of memory during long-running OpenClaude tasks #1258
• [critical]: JavaScript heap OOM after prolonged use — auto-compact circuit breaker permanently disables compaction #1373
• Track remaining long-session OOM guards after auto-compact cooldown #1379
• Memory Leak: RSS grows ~115 MB/hour while idle (PerformanceMeasure accumulation via Ink reconciler) #1839

The symptoms are not all identical, but they overlap enough that tracking them separately made it hard to tell what is fixed, what is mitigated, and what still needs validation.

Consolidated Problem

OpenClaude had at least three related failure modes:

1. Long sessions could keep growing in process memory until Node/V8 aborted with FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory.
2. Auto-compact could reduce model context while retained in-memory transcript objects, native toolUseResult payloads, tool outputs, session state, or teammate-agent histories still grew in the process heap.
3. When a provider returned a context-window error after a large tool result, the current task could stop instead of automatically compacting/reducing context and retrying once.

These are provider-independent in the reports: affected configurations include Anthropic/litellm proxy, Codex/GPT-5.5, MiMo via OpenGateway, MiniMax, and first-party Anthropic. Reports also span macOS, Windows, and Linux.

Implementation Status

The main implementation work is now covered by #1856 and #1858:

• fix(memory): prevent reported idle retention paths #1856 fixed Memory Leak: RSS grows ~115 MB/hour while idle (PerformanceMeasure accumulation via Ink reconciler) #1839's idle-retention path by bundling production React/reconciler/scheduler for the CLI, bounding FPS samples, capping MCP startup stderr retention, and making repeated manual heap dumps unique within a session.
• fix(memory): recover from autocompact overflow failures #1858 covers the runtime/autocompact recovery slice: context-overflow tagging and one-shot compact-and-retry, active-message hard-cap compaction and safety blocking, in-process teammate allMessages bounding, compact-boundary/tool-state cleanup, OpenAI-compatible conversion pruning, memory guard diagnostics, and focused long-session guard tests.

Earlier mitigations that remain relevant:

• FIX: Reduce stable stringify heap usage #1104 reduced peak heap usage during stable request serialization.
• chore(main): release 0.12.0 #1173 released the FIX: Reduce stable stringify heap usage #1104 mitigation in v0.12.0.
• fix(entrypoint): apply --max-old-space-size=8192 universally, not jus… #1191 attempted to raise the heap cap, but did so from inside the running process and therefore did not fully fix parent-process heap sizing.
• Fix launcher heap setup for long sessions #1242 added the launcher relaunch guard so the parent Node process starts with the intended heap cap and --expose-gc.
• fix(compact): clear native tool results after time compaction #1278 clears retained native tool-result payloads after time-based microcompact replaces old tool-result content.
• fix(launcher): route direct Node launch paths through launcher #1363 routed direct Node launch paths through the launcher so dev/start/container paths use the heap-aware wrapper.
• fix(autocompact): retry circuit breaker after cooldown #1375 added auto-compact circuit-breaker retry after cooldown.

Report Coverage

#1046 - Memory error and no autocompact

This overlaps both the heap/OOM reports and context/autocompact recovery. #1858 addresses the no-autocompact/context-overflow and retained transcript growth paths that apply here. The remaining proof point is a representative long-session heap soak after #1858 lands.

#1105 - Auto-recover from context-window errors after large tool results

#1858 implements the retry-after-failure path preferred in the issue discussion: provider context-overflow errors are tagged, withheld from the stream while recoverable, followed by forced compaction, and retried once with explicit recovery context. It does not add pre-flight token estimation before provider send.

#1148 - Windows OOM

Launcher heap fixes already landed before this tracker, and #1856/#1858 address the concrete retained-memory and autocompact recovery paths identified since then. The broad Windows long-session claim still needs soak evidence after #1858 lands.

#1258 - Node/V8 process crashes during long-running tasks

#1858 targets the retained-history/autocompact/provider-conversion mechanisms called out by this report, including active-message hard caps, teammate history bounding, and OpenAI-compatible message pruning. The remaining proof point is a real long-session heap soak across representative providers/OSes.

#1373 - Auto-compact circuit breaker disables compaction

#1375 already made the breaker retry after cooldown. #1858 adds the active-message safety cap, forced compaction, failure diagnostics, and pre-provider-call safety blocking so repeated failed compactions do not silently allow unbounded state.messages growth.

#1379 - Remaining long-session OOM guards after auto-compact cooldown

#1858 covers the listed remaining work: bounded main active-message behavior, in-process teammate allMessages bounding, provider conversion pressure reduction, compact-boundary/tool-state cleanup, and tests for pruned tool-call/tool-result histories.

#1839 - Idle PerformanceMeasure/FPS/MCP/heapdump retention

#1856 fixed the concrete #1839 root cause and secondary retention paths. This path is no longer an open implementation blocker for this tracker.

Still Open / Remaining Work

This tracker should stay open until the implementation PR has landed and a representative long-session heap validation has been completed or explicitly split into a follow-up issue.

Remaining checklist:

• Merge fix(memory): recover from autocompact overflow failures #1858.
• Run or document a real long-session heap soak across representative providers/OSes after fix(memory): recover from autocompact overflow failures #1858 lands.
• Confirm from the soak that long sessions do not show unbounded process heap growth after repeated compactions, context-overflow recovery, teammate activity, and OpenAI-compatible provider request conversion.
• If soak validation finds a remaining retained-transient path, split that concrete path into a focused follow-up issue.

Acceptance Criteria

• Long-running sessions do not grow heap without bound after repeated compactions.
• Retained native tool results and old transcript/tool payloads are released or bounded after compaction.
• Auto-compact failure/cooldown states are visible and recoverable.
• Provider context-window overflow from a large tool result either pre-compacts before send or retries once after compacting/reducing context.
• The behavior is covered by focused tests plus a documented long-session validation procedure or follow-up issue for any remaining soak-only risk.

Supersedes

This issue supersedes #1046, #1105, #1148, #1258, #1373, #1379, and #1839 for tracking the consolidated long-session OOM/autocompact recovery failure family.

[jatmn]

Follow-up scan of current open issues found two additional related reports:

• Track remaining long-session OOM guards after auto-compact cooldown #1379 - Direct follow-up to [critical]: JavaScript heap OOM after prolonged use — auto-compact circuit breaker permanently disables compaction #1373 / fix(autocompact): retry circuit breaker after cooldown #1375 for remaining long-session OOM guards after auto-compact cooldown. This is duplicate tracking for the same remaining work already listed here, especially bounded state.messages retention, teammate allMessages retention, and convertMessages transient heap pressure.
• Memory Leak: RSS grows ~115 MB/hour while idle (PerformanceMeasure accumulation via Ink reconciler) #1839 - Newer, more specific idle-memory leak report: RSS grows ~115 MB/hour while idle due to PerformanceMeasure accumulation from the Ink/react-reconciler render loop. This is related to the broader "long-running sessions should not grow heap without bound" acceptance criteria, but it has a distinct root-cause investigation and should remain open as a focused implementation/root-cause issue linked to this canonical tracker.

[jatmn]

Linking current implementation PR: #1858 fixes #1105, addresses the remaining hard-cap/safety-stop portion of #1373 after merged #1375, and partially addresses #1379.

It deliberately does not close this canonical tracker because #1839 is covered by #1856 and the long-session heap validation / remaining retained-transient paths still need to be resolved or explicitly split out.