Agent update (automatic and manual modes) can permanently trap long-lived agents with no recovery path

[almoehi]

Summary

Updating a long-lived, high-oplog-entry-count agent to a new component revision — via either update-agents --update-mode automatic or --update-mode manual — can permanently trap the agent with no recovery path back to a working, on-the-new-binary state. Both modes report success (or fail with a revertable error) yet neither delivers a genuinely safe path to actually run the new code on an already-durable, long-lived agent. This was hit in production-adjacent testing on a self-hosted Golem app with ~37 long-running agents (one long-lived "coordinator" agent with an ~18,400-entry, 11-hour oplog, plus ~14 shorter-lived, several-hundred-to-few-thousand-entry task-worker agents, plus assorted short-lived agents).

This report covers every path we tried, in the order we tried them, what happened, and what (if anything) worked. None of the paths below actually get you "same agent instance, new binary, full history preserved" — that's the gap this issue is about.

Environment

• Self-hosted Golem (local dev server + fat-image Docker test container), golem CLI + golem-worker-executor from a stock checkout of golemcloud/golem (no engine changes made).
• App: TypeScript agents via @golemcloud/golem-ts-sdk, single WASM component, ~37 concurrently-running agent instances of varying types and history lengths, several with @agent({ snapshotting: { periodic: "2m" } }) configured.
• Trigger: redeployed the component (new WASM revision reflecting an app-level code change — one new method added to an agent class, one method signature change on another) and attempted to bring already-running agents onto it.

Attempt 1 — update-agents --update-mode automatic (default)

golem-worker-executor/src/worker/mod.rs:2894-2915 (worker load path used during an update):

let mut last_snapshot_index = worker_metadata.last_known_status.last_manual_update_snapshot_index;

// automatic snapshots are only considered until the first failure.
// additionally, if there are updates, the automatic snapshot is temporarily ignored to catch issues earlier
if let Some(snapshot_idx) = worker_metadata.last_known_status.last_automatic_snapshot_index
    && pending_update.is_none()
    && !parent.snapshot_recovery_disabled.load(Ordering::Acquire)
{
    ...
    last_snapshot_index = Some(snapshot_idx);
}

Periodic snapshots are gated behind pending_update.is_none() — i.e. categorically unusable during an update, regardless of how recent or plentiful they are. Our long-lived coordinator agent had ~130 periodic snapshots taken reliably every ~2 minutes throughout its active history; automatic update still replayed from OplogIndex::INITIAL (entry 1) and diverged at entry #690 out of ~18,400 — inside the agent's very first invocation, 38 seconds after creation.

Result: 15 of 37 agents trapped into Status: Failed (Component trapped: Unexpected oplog entry during replay: expected X, got Y); 22 succeeded (short-history agents where full replay happened to be cheap/compatible). The 15 failures included the coordinator agent itself.

We understand the design intent (automatic mode wants the stronger "full history since creation still replays under the new binary" guarantee), but there's no way to opt a specific update into "best effort, anchor on the most recent periodic snapshot" without accepting that much stronger, and for long-lived agents often infeasible, guarantee.

Recovering from Attempt 1's failures: because these are failed automatic updates (never reached SUCCESSFUL UPDATE), agent revert --last-oplog-index <N> --yes is available — Golem hasn't marked anything as superseded yet. For the coordinator, the only oplog index the CLI's own reported divergence point (retry from: N) allows reverting to is #690 — meaning "recovery" here means discarding virtually the entire operational history, not really a recovery of anything usable. We did not do this for the coordinator (see Attempt 4). For 14 of the 15 shorter-history worker agents, we didn't need revert at all — see Attempt 4, which addressed them via a different mechanism.

Attempt 2 — update-agents --update-mode manual (snapshot-based)

Manual mode is documented/intended to sidestep replay entirely via saveSnapshot()/loadSnapshot() state transfer (durable_host/mod.rs:2153, UpdateDescription::SnapshotBased, finalize_pending_snapshot_update). We used this specifically hoping it would avoid Attempt 1's full-history-replay requirement.

The update itself reports success cleanly — oplog shows a normal entry, e.g.:

#00479: SUCCESSFUL UPDATE  target revision: 3
#00480: SNAPSHOT
#00481: SNAPSHOT

But invoking the agent immediately afterward fails deterministically:

error: Agent Service - Error: 500 Internal Server Error, Previous Invocation Failed

with the real cause (found in the executor's own log, not surfaced to the CLI) being the same class of replay-order mismatch as Attempt 1:

Previous invocation failed: Unexpected oplog entry during replay: expected io::poll::poll, got http::types::outgoing_body_stream::check_write

Root cause, traced through source:

• worker/status.rs:730-740 — a successful SnapshotBased (manual) update correctly clears the poisoning field on the fold: last_automatic_snapshot_index = None; last_automatic_snapshot_timestamp = None.
• worker/status.rs:743-746, same fold, immediately after — any subsequent OplogEntry::Snapshot (ordinary periodic snapshots aren't distinguished from anything else at the type level) re-sets it: last_automatic_snapshot_index = Some(*oplog_idx).
• A periodic snapshot landed seconds after our SUCCESSFUL UPDATE (entries 378 benchmark matrix #480/Reuse cluster across benchmark iterations #481 above — the agent's 2-minute periodic tick fired right in that window), so last_automatic_snapshot_index was live again by the time any ordinary (non-update) load happened.
• worker/mod.rs:2894-2911 (same override block as Attempt 1) then applies unconditionally on every ordinary load, not just updates — pending_update.is_none() is true for a plain invoke, so the override fires, discarding the manual-update-derived skip boundary (last_manual_update_snapshot_index) and forcing replay against INITIAL..=last_automatic_snapshot_index again, which still spans the pre-update, now-incompatible history.
• There is a designed self-healing path for one flavor of this failure: durable_host/mod.rs:3285-3300 — if try_load_snapshot itself reports SnapshotRecoveryResult::Failed, snapshot_recovery_disabled (an in-process AtomicBool, never persisted) is set and the load retries, correctly bypassing the override on retry. But this only fires when the snapshot-load step itself fails; here the failure surfaces during the subsequent resume_replay instead, which has no equivalent recovery — the error propagates to WorkerExecutorError::failed_to_resume_worker directly, and every fresh load attempt repeats the identical doomed sequence (deterministic, not transient — confirmed by retrying the same invoke multiple times with identical results).

Result: applied to all 37 agents in one update-agents --update-mode manual --await pass. 35 succeeded and remained invokable (mostly short-lived/low-entry agents plus, notably, the coordinator itself — its most recent periodic snapshot must have landed outside the poisoning window). 14 of the ~15 worker agents landed in this permanently-poisoned state — reporting SUCCESSFUL UPDATE in their oplog and even Status: Running/Idle in agent list, while every actual invoke 500s with Previous Invocation Failed. This is worth flagging on its own: agent list/agent get status does not reflect this failure mode at all — the only way we found to detect it was to actually invoke each agent.

Attempt 3 — trying to force a second manual-update cycle to win the race

Reasoning: if a periodic snapshot re-poisoning the load path within the same few-second window is what breaks Attempt 2, maybe re-triggering a fresh SuccessfulUpdate fold (which clears the poisoning field again) and invoking immediately, before the next 2-minute tick, would work.

1. Re-run manual update to the same target revision — rejected outright, no update even attempted: Invalid request: Worker is already at the target version.
2. Deploy is content-addressed, so re-deploying byte-identical WASM doesn't produce a new revision to target (Deployment: no changes required [UP-TO-DATE]). We instead updated the agent down to a prior (older) revision, then immediately back up to the target revision, hoping to force two fresh SuccessfulUpdate events. Both calls returned: Unexpected agent state: update is not pending anymore, but no outcome has been found. A subsequent invoke still failed identically.

Result: once an agent is in the Attempt-2 poisoned state, it rejects all further operations against that instance — not just plain invokes, but new update requests too. There is no retry/race-window workaround available at the CLI level.

Attempt 4 — agent revert on the poisoned agents

agent revert --last-oplog-index <N> --yes for any N inside the pre-update history is hard-rejected once a manual update has committed:

error: Worker Service - Error: 500 Internal Server Error, Invalid request: Attempted to revert to a deleted region in oplog to index 474

This traces to the DeletedRegionsBuilder marking (durable_host/mod.rs:~4162) applied on successful manual update — by design, revert refuses to target an index inside an already-"superseded" region. Reasonable as a guard against un-committing a completed migration, but combined with Attempt 2 and Attempt 3, it means: once an agent hits the manual-update poisoning bug, there is no CLI-exposed way back to a working state for that agent instance at all — not revert, not retry-update, not plain invoke.

Attempt 5 — delete + let the app recreate a fresh instance (worked, but is a workaround, not a fix)

For the 14 poisoned worker agents specifically — whose application-level role is "do one piece of work, then report the result back to the coordinator agent, which is the actual source of truth for whether that work succeeded" — we deleted each poisoned instance outright (agent delete <AgentId>, permanently removes the instance and its oplog) and let the app's own logic construct a brand-new instance under the same deterministic agent id on next reference. Verified live: the fresh instances construct cleanly on the new binary and respond correctly to invocations; the coordinator agent (which tracks task/artifact status independently, not by querying the workers) still reports every task's completion status correctly afterward, since that was never stored only in the now-deleted worker instances.

This is a real, working recovery for this specific case — but it is not a general answer. It only works because:

• the worker agents' durable state has an external source of truth (the coordinator) that survives their deletion, and
• losing a deleted worker's own internal history (e.g. its past LLM conversation/tool-call trace) is an acceptable, disclosed cost for us.

It is not available for an agent whose own instance-local state is the source of truth — e.g. the coordinator agent itself in our case, or any standalone/decoupled agent generally. For those, none of Attempts 1-4 provide a working path to "same instance, new binary, state preserved" either.

What we deliberately did not do: roll the whole component back to the old binary

At one point mid-investigation we did fully roll the deployed component back to the previous (already-compatible) WASM and re-ran update-agents to bring every agent back onto it, purely to restore a healthy baseline before continuing to investigate. We are not counting this as a solution and don't want it read as one — it doesn't upgrade anything; it just undoes the attempted upgrade entirely, which is the opposite of the problem we're trying to solve (getting a real code change safely onto already-running, long-lived agents). We mention it only for completeness: it is the one thing we found that reliably restores full health with zero data loss, precisely because it changes nothing that the agents' existing oplogs weren't already compatible with. It says nothing about how to actually ship an update.

Questions for maintainers / possible fixes

1. Should periodic snapshotting really be entirely disabled during automatic updates (Attempt 1), or could there be a supported "best-effort" mode that anchors on the most recent periodic snapshot instead of requiring full-history-since-creation compatibility? This seems like it would be the single biggest risk reducer for long-lived agents, and the ~130 unused snapshots we had sitting right there make the current behavior feel like an available fast path is simply not being taken.
2. Attempt 2 looks like a straightforward bug: a periodic snapshot landing after a successful manual update re-poisons the ordinary load path by re-setting last_automatic_snapshot_index with no awareness that a manual-update boundary now supersedes it. Should last_manual_update_snapshot_index take precedence over a later automatic snapshot rather than being unconditionally overridden? Or should a manual update disable/coalesce periodic snapshotting up to the update boundary so a stray tick in the following seconds can't recreate the poisoned condition?
3. The retry/self-heal loop in durable_host/mod.rs:3285-3300 only covers snapshot-load failures, not failures surfacing later during resume_replay (which is what we hit). Should it be extended to cover this case too?
4. Given Attempts 3 and 4 both dead-end, is there any supported recovery path we're missing for an agent already in the Attempt-2 poisoned state, short of deleting the instance? Right now "delete and let the app rebuild from an external source of truth" (Attempt 5) is the only thing that worked, and it's fundamentally not available to every agent shape.
5. Is agent list/agent get reporting Status: Idle/Running for agents that are actually permanently un-invokable (Attempt 2's poisoned state) something worth surfacing? We only discovered the true scope of the problem by invoking every agent individually — the status command actively suggested everything was fine.

Happy to provide the full oplogs/logs from our repro (with app-specific content redacted) if useful — every attempt above was reproduced live end-to-end against a real running deployment, not inferred from source reading alone.

[vigoo]

Thanks for the detailed report!

[almoehi]

Oh - this was actually meant to be created in my own repo lol 😂

But yeah - I'm trying to figure out the operational side of my golem project. For dev I simply worked around by resetting the golem server before each new wasm release. Clearly not gonna work in the live deployment 😬