Skip to content

Commit bca6dc7

Browse files
authored
[Agents] Document unconditional chat recovery (#32608)
* docs: document unconditional chat recovery * docs: address chat recovery review feedback
1 parent 726cc88 commit bca6dc7

9 files changed

Lines changed: 43 additions & 45 deletions

File tree

src/content/docs/agents/communication-channels/chat/autonomous-responses.mdx

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -504,7 +504,9 @@ if (result.status === "aborted") {
504504

505505
</TypeScriptExample>
506506

507-
`continueLastTurn()` accepts the same `options.signal` argument. `AbortSignal` objects cannot cross Durable Object RPC boundaries, and the signal is in memory only. If the Durable Object hibernates mid-turn and chat recovery is enabled, the recovered turn usually continues without the original signal; for pre-stream interruptions, recovery can instead retry the latest unanswered user message automatically. An abort fired after restart has no effect on the recovered turn.
507+
`continueLastTurn()` accepts the same `options.signal` argument. `AbortSignal` objects cannot cross Durable Object RPC boundaries, and the signal is in memory only. If the Durable Object hibernates mid-turn, durable recovery usually continues without the original signal. For pre-stream interruptions, recovery can retry the latest unanswered user message. An abort fired after restart has no effect on the recovered turn.
508+
509+
Persist cancellation intent when cancellation must survive a restart. Read that state in `onChatRecovery()` and return `{ continue: false }` to prevent another model call.
508510

509511
Use `cancelSubmission(submissionId)` for durable cancellation when work was accepted with `submitMessages()` or when cancellation must cross Worker and Durable Object RPC boundaries.
510512

src/content/docs/agents/communication-channels/chat/chat-agents.mdx

Lines changed: 21 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -399,7 +399,7 @@ if (result.status === "aborted") {
399399

400400
</TypeScriptExample>
401401

402-
`continueLastTurn()` accepts the same `options.signal` argument. `AbortSignal` objects cannot cross Durable Object RPC boundaries, so construct the controller inside the Durable Object that calls `saveMessages()` or `continueLastTurn()`. The signal is in memory only; if the Durable Object hibernates mid-turn and `chatRecovery` is enabled, the recovered turn runs without the original signal.
402+
`continueLastTurn()` accepts the same `options.signal` argument. `AbortSignal` objects cannot cross Durable Object RPC boundaries, so construct the controller inside the Durable Object that calls `saveMessages()` or `continueLastTurn()`. The signal is in memory only. If the Durable Object hibernates mid-turn, the recovered turn runs without the original signal. Persist cancellation intent when cancellation must survive a restart.
403403

404404
### `onChatResponse`
405405

@@ -577,25 +577,13 @@ Use `abortRequest()` when you know the request ID. Use `abortAllRequests()` for
577577

578578
### Stream recovery
579579

580-
Automatic stream resumption (the `resume` option on `useAgentChat`) is **client reconnect recovery** — it resumes an active stream when a client disconnects and reconnects. It does not cover Durable Object eviction: if the Worker process or Durable Object is evicted while the model call is in flight, the stream itself is gone. `chatRecovery` handles that case.
580+
Automatic stream resumption (the `resume` option on `useAgentChat`) is **client reconnect recovery** — it resumes an active stream when a client disconnects and reconnects. It does not cover Durable Object eviction. If the Worker process or Durable Object is evicted while the model call is in flight, the stream itself is gone. Durable chat recovery handles that case.
581581

582-
When a Durable Object is evicted mid-stream (code update, inactivity timeout, resource limit), the LLM connection is severed permanently and the in-memory streaming state is lost. `chatRecovery` wraps each chat turn in a [`runFiber()`](/agents/runtime/execution/durable-execution/), providing automatic `keepAlive` during streaming and a recovery hook on restart.
582+
A mid-stream Durable Object eviction permanently severs the LLM connection. Durable recovery wraps every `AIChatAgent` and [`Think`](/agents/harnesses/think/) chat turn in a [`runFiber()`](/agents/runtime/execution/durable-execution/). The fiber provides automatic `keepAlive` during streaming and a recovery hook on restart.
583583

584-
<TypeScriptExample>
585-
586-
```ts
587-
export class ChatAgent extends AIChatAgent {
588-
override chatRecovery = true;
589-
}
590-
```
591-
592-
</TypeScriptExample>
584+
The fiber row survives in SQLite after an eviction. On the next activation, the framework detects the interrupted fiber. It reconstructs the partial response from buffered stream chunks and calls `onChatRecovery`.
593585

594-
`AIChatAgent` defaults `chatRecovery` to `false`, so existing chat agents only get client reconnect and resumable-stream behavior unless they opt in. [`Think`](/agents/harnesses/think/) defaults it to `true`.
595-
596-
When enabled, every `onChatMessage` call runs inside a fiber. If the agent is evicted mid-stream, the fiber row survives in SQLite. On the next activation, the framework detects the interrupted fiber, reconstructs the partial response from buffered stream chunks, and calls `onChatRecovery`.
597-
598-
`chatRecovery` can also be set to a configuration object to bound recovery and customize the terminal experience when recovery cannot succeed:
586+
Durable recovery is always on. Use `chatRecovery` only to tune recovery budgets and terminal behavior:
599587

600588
<TypeScriptExample>
601589

@@ -608,8 +596,8 @@ export class ChatAgent extends AIChatAgent {
608596
// Primary stuck-turn bound. Resets on every progress-bearing attempt, so a
609597
// turn that keeps producing content survives unbounded interruption.
610598
noProgressTimeoutMs: 5 * 60 * 1000,
611-
// Runaway-loop guard. Defaults to Infinity (no cap). Set a finite value to
612-
// seal a turn that keeps emitting content but never converges.
599+
// Runaway-loop guard. Defaults to 1,000. Set a higher value for a long
600+
// agentic turn, or Infinity to remove the cap.
613601
maxRecoveryWork: 200,
614602
// Caller policy consulted from the second recovery attempt onward. Return
615603
// false to stop recovery. This is where you enforce a token/cost budget.
@@ -636,7 +624,8 @@ The `chatRecovery` object accepts the following configuration options:
636624
| `stableTimeoutMs` | `10_000` | How long a recovery attempt waits for the isolate to reach stable state before rescheduling. |
637625
| `terminalMessage` | generic message | The message shown to the user when recovery is given up on. |
638626
| `noProgressTimeoutMs` | `300_000` (5 min) | Primary stuck-turn bound: how long an incident may go without forward progress before it is sealed (`no_progress_timeout`). **Resets on every progress-bearing attempt**, so a turn that keeps producing content survives unbounded interruption. |
639-
| `maxRecoveryWork` | `Infinity` | Runaway-loop guard. Maximum produced content/tool units since the incident began before a still-progressing turn is sealed. Defaults to no cap. |
627+
| `maxRecoveryWork` | `1,000` | Runaway-loop guard. Maximum produced content/tool units since the incident began before a still-progressing turn is sealed. Set a higher value or `Infinity` for a long agentic turn. |
628+
| `maxOomRetries` | `3` | Retry budget for Durable Object memory-limit resets. Set `0` to stop after the first memory-limit reset. |
640629
| `shouldKeepRecovering` || Caller policy consulted from the second recovery attempt onward. Return `false` to stop recovery. Use it to enforce a token or cost budget. `ctx.work` is a coarse segment count, not tokens, so track real spend yourself. |
641630
| `onExhausted` || Called once when recovery is given up on, before the terminal message is delivered. Inspect `ctx.reason` for why. |
642631

@@ -653,16 +642,17 @@ The `chatRecovery` object accepts the following configuration options:
653642
| `work` | `number` | Coarse, monotonic count of content/tool segments produced since the incident opened (not tokens). |
654643
| `ageMs` | `number` | Wall-clock ms since the incident's first interruption. |
655644

656-
A progressing turn is never terminated by the framework on its own — it survives unbounded interruption (for example a dense deploy window) as long as it keeps making forward progress. Recovery is sealed only by one of these `ctx.reason` values:
645+
A progressing turn survives repeated interruptions as long as it stays within the `maxRecoveryWork` limit. Recovery is sealed by one of these `ctx.reason` values:
657646

658647
- `no_progress_timeout` — no forward progress within the no-progress window (a stuck turn).
659648
- `max_attempts_exceeded` — the attempt cap was spent on a tight no-progress alarm loop.
660649
- `work_budget_exceeded` — the turn kept producing content but exceeded `maxRecoveryWork` (a runaway loop).
661650
- `recovery_aborted` — your `shouldKeepRecovering` hook returned `false`.
651+
- `out_of_memory` — recovery exceeded the memory-limit retry budget.
662652
- `stable_timeout` — recovery attempts kept timing out waiting for stable state until the budget drained (extreme churn).
663653

664654
:::tip
665-
A finite `maxRecoveryWork` can seal a legitimately long turn. Set a cap well above what a healthy turn produces, or use `shouldKeepRecovering` with real token or cost accounting for a precise budget.
655+
The `maxRecoveryWork` default prevents a progressing turn from running forever. Increase it for long agentic turns. Use `shouldKeepRecovering` with durable token or cost data for a precise budget.
666656
:::
667657

668658
#### Turns waiting on a human are not sealed
@@ -702,8 +692,6 @@ import type {
702692
} from "@cloudflare/ai-chat";
703693

704694
export class ChatAgent extends AIChatAgent {
705-
override chatRecovery = true;
706-
707695
override async onChatRecovery(
708696
ctx: ChatRecoveryContext,
709697
): Promise<ChatRecoveryOptions> {
@@ -765,6 +753,15 @@ override async onChatRecovery(
765753
}
766754
```
767755

756+
#### Control automatic continuation
757+
758+
Durable bookkeeping remains active when automatic continuation is not appropriate.
759+
760+
- Return `{ continue: false }` when another model call is unsafe.
761+
- Persist cancellation intent and read it in `onChatRecovery()`.
762+
- Record idempotency keys before external side effects.
763+
- Use recovery budgets with durable spend data to limit cost.
764+
768765
#### `continueLastTurn`
769766

770767
Appends to the last assistant message by re-calling `onChatMessage` with the saved request body. The response is streamed as a continuation — appended to the existing assistant message, not a new one. No synthetic user message is created.
@@ -786,8 +783,6 @@ Use `this.stash()` inside `onChatMessage` to persist provider-specific data for
786783

787784
```ts
788785
export class ChatAgent extends AIChatAgent {
789-
override chatRecovery = true;
790-
791786
async onChatMessage(_onFinish, options) {
792787
const result = streamText({
793788
model: openai("gpt-5.4"),

src/content/docs/agents/concepts/agentic-patterns/long-running-agents.mdx

Lines changed: 2 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -540,7 +540,7 @@ For the full `subAgent()` API — typed RPC stubs, client routing, access contro
540540

541541
The patterns above handle the project manager's coordination work — scheduling, delegating, polling. But the project manager also uses an LLM directly: generating plans, summarizing progress, drafting status emails. Those LLM calls stream tokens over a connection that cannot be resumed if the agent is evicted mid-response.
542542

543-
For chat-oriented agents built on `AIChatAgent`, this is an even sharper problem — the user is watching the response stream in real time and sees it stop mid-sentence. `chatRecovery` wraps each chat turn in a `runFiber`, providing automatic `keepAlive` during streaming and a recovery hook when the agent restarts:
543+
For chat-oriented agents built on `AIChatAgent` or `Think`, this is an even sharper problem — the user watches the response stream in real time and sees it stop mid-sentence. Durable recovery wraps every chat turn in a `runFiber`. This provides automatic `keepAlive` during streaming and a recovery hook when the agent restarts:
544544

545545
```ts
546546
import { AIChatAgent } from "@cloudflare/ai-chat";
@@ -550,8 +550,6 @@ import type {
550550
} from "@cloudflare/ai-chat";
551551

552552
class ProjectChat extends AIChatAgent<Env> {
553-
override chatRecovery = true;
554-
555553
override async onChatRecovery(
556554
ctx: ChatRecoveryContext,
557555
): Promise<ChatRecoveryOptions> {
@@ -575,7 +573,7 @@ The right recovery strategy depends on the LLM provider:
575573

576574
Use `ctx.createdAt` to suppress stale recoveries. For example, if a recovered chat turn is older than a few minutes, you may persist the partial answer but skip automatic continuation to avoid surprising the user with an old response.
577575

578-
[`Think`](/agents/harnesses/think/) enables `chatRecovery` by default. The default path persists partial output and auto-continues or retries the turn when safe, so many apps do not need a custom hook. Override `onChatRecovery` when a provider has a better recovery strategy, or configure `chatRecovery = { maxAttempts, terminalMessage, onExhausted }` to tune the terminal user experience.
576+
`AIChatAgent` and [`Think`](/agents/harnesses/think/) always use durable recovery. The default path persists partial output and continues or retries the turn when safe. Override `onChatRecovery` when a provider has a better recovery strategy. Configure `chatRecovery = { maxAttempts, terminalMessage, onExhausted }` to tune the terminal experience.
579577

580578
If the agent is interrupted before any assistant stream chunks are written, there is no partial assistant message to continue. When the latest persisted message is still the unanswered user message from that turn, chat recovery retries the turn automatically unless `onChatRecovery` returns `{ continue: false }`.
581579

src/content/docs/agents/harnesses/think/client-tools.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -116,7 +116,7 @@ When a turn produces several client tool calls at once, Think waits for **all**
116116

117117
## Survive restarts while waiting for a human
118118

119-
A Durable Object can be evicted at any time, including while a turn is paused on an approval prompt or a client-side tool call. Because `Think` enables [`chatRecovery`](/agents/harnesses/think/recovery/) by default, the SDK treats such a turn as waiting on the human, not stuck. It parks the turn instead of failing it, and the user's eventual approval or tool result resumes the conversation.
119+
A Durable Object can be evicted at any time, including while a turn is paused on an approval prompt or a client-side tool call. [`Think` durable recovery](/agents/harnesses/think/recovery/) is always on. The SDK treats such a turn as waiting on the human, not stuck. It parks the turn instead of failing it. The user's eventual approval or tool result resumes the conversation.
120120

121121
For which interactions are exempt from recovery budgets, refer to [Turns waiting on a human are not sealed](/agents/communication-channels/chat/chat-agents/#turns-waiting-on-a-human-are-not-sealed).
122122

src/content/docs/agents/harnesses/think/configuration.mdx

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -31,8 +31,8 @@ Think is configured by overriding methods and properties on your `Think` subclas
3131
| `messageConcurrency` | `"queue"` | How overlapping submits behave — refer to [Client tools](/agents/harnesses/think/client-tools/#message-concurrency) |
3232
| `includeMcpTools` | `true` | Convert connected MCP tools to AI SDK tools and add them to model turns. Refer to [MCP tools](/agents/harnesses/think/tools/#mcp-tools) |
3333
| `waitForMcpConnections` | `false` | Wait for MCP servers before inference |
34-
| `chatRecovery` | `true` | Wrap WebSocket, sub-agent, programmatic, and continuation turns in `runFiber` for durable execution. Set to a configuration object with `maxAttempts`, `stableTimeoutMs`, `terminalMessage`, and `onExhausted` to tune bounded recovery |
35-
| `chatStreamStallTimeoutMs` | `0` (off) | Opt-in inactivity watchdog: abort a turn whose model stream produces no chunk for this long (measures the gap between chunks, including tool execution). With `chatRecovery` on, a stall routes into bounded recovery |
34+
| `chatRecovery` | Always on | Durable recovery configuration. Refer to [Durable recovery](/agents/harnesses/think/recovery/) for all options and defaults |
35+
| `chatStreamStallTimeoutMs` | `0` (off) | Opt-in inactivity watchdog: abort a turn whose model stream produces no chunk for this long (measures the gap between chunks, including tool execution). A stall routes into bounded recovery |
3636
| `contextOverflow` | `undefined` | Opt-in mid-turn context-overflow handling with `reactive`, `maxRetries`, and `proactive` options. Requires `classifyChatError` plus a session compaction function — refer to [Context-window overflow recovery](/agents/harnesses/think/recovery/#context-window-overflow-recovery) |
3737

3838
For `chatRecovery` and `chatStreamStallTimeoutMs` behavior, refer to [Durable recovery](/agents/harnesses/think/recovery/).

src/content/docs/agents/harnesses/think/index.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -247,7 +247,7 @@ Key behaviors:
247247

248248
- **Blocking modes cannot nest.** Calling `wait`/`stream`/`continuation` (or the equivalent shortcut) from _inside_ an active turn — for example, from a tool's `execute` — throws, because it would deadlock the turn queue. From inside a turn, use `runTurn({ mode: "submit" })` (durable, runs after the current turn frees the queue) or [`addMessages()`](#add-messages-without-a-turn) (transcript only, no inference).
249249
- **`submit` is idempotent.** Pass `submissionId` and/or `idempotencyKey`; re-submitting a known key returns the existing record with `accepted: false` instead of starting a second turn. See [Programmatic submissions](/agents/harnesses/think/programmatic-submissions/).
250-
- **Recovery-safe.** When `chatRecovery` is enabled, the `wait`, `stream`, and drained `submit` paths all run inference inside a recovery fiber, so an interrupted turn resumes after eviction.
250+
- **Recovery-safe.** The `wait`, `stream`, and drained `submit` paths run inference inside a recovery fiber, so an interrupted turn resumes after eviction.
251251

252252
`runTurn` is exported alongside its option and result types: `RunTurnOptions`, `RunTurnWait`, `RunTurnSubmit`, `RunTurnStream`, `TurnInputMessages`, and `TurnResult`.
253253

0 commit comments

Comments
 (0)