docs(design): plan warm and resumable Daytona sessions (F-020)

docs/design/agent-workflows/projects/warm-daytona-sessions/README.md

@@ -0,0 +1,65 @@
+# Warm and resumable Daytona sessions
+
+## What this project is about
+
+When you chat with an agent that runs on Daytona, every message waits about fifteen seconds before
+the agent starts to answer. That wait is the whole per-turn setup being redone from scratch, once
+per message: build a cloud machine, install and start the agent inside it, mount its files, reload
+the conversation, even though the previous turn already did all of that. This workspace plans how
+to reuse the running setup between turns so the second message, and every message after it, starts
+fast. It is a design plan only. No code ships in this pull request.
+
+A few words you will meet throughout, defined once:
+
+- **Sandbox:** the isolated cloud machine an agent runs in. On Daytona it is a billed resource.
+- **Daytona:** the cloud provider that hosts these sandboxes.
+- **Runner:** the Agenta service that drives one agent turn. It builds the sandbox, runs the turn
+  inside it, and tears it down.
+- **Harness:** the agent program that runs inside the sandbox (Claude Code or Pi).
+- **Park a sandbox:** stop it but keep its disk, so the next turn can restart the same one instead
+  of rebuilding it. This is the whole idea behind the project.
+
+## Read the files in this order
+
+1. **context.md.** What a user sees today, what recent work already tried, why it still fails, and
+   the finding that shaped the first draft: the code to keep a sandbox warm was already written,
+   but the piece of it that talks to Daytona is missing two functions it needs.
+2. **research.md.** The current code, function by function, with the exact reasons warm reuse does
+   not work yet, plus the measured Daytona lifecycle numbers and prices (2026-07-11) that reshaped
+   the plan. Read this for the evidence behind context.md.
+3. **plan.md.** The proposal: one progressive sequence of slices from the correctness base through
+   park-to-stopped and a provider-aware pool refactor up to park-to-running. Its section "The
+   three resume paths, compared" prices fresh create, stop-then-restart, and kept-running from
+   the measured stages; start there if you only want the decision.
+4. **open-questions.md.** What still needs a human: three correctness decisions for a reviewer and
+   two proposed defaults to confirm. The former cost questions are answered by the measurement.
+5. **status.md.** Where the project stands, what was decided and why, the measurement summary, and
+   what the two design-review rounds changed.
+
+## The recommendation, in one paragraph
+
+Build the whole ladder in slices, each shippable on its own. First the correctness base (the two
+missing Daytona provider functions plus the leak and race fixes), then park-to-stopped (stop the
+sandbox at turn end instead of deleting it; parked cost is disk storage only, about $0.0009/hour),
+then a refactor that makes the existing local keepalive pool provider-aware, then park-to-running
+(keep the sandbox running for two minutes after each turn, with a hard cap of 20 warm sandboxes,
+about $0.0028 per parked minute), and finally credit-controlled live verification before the
+defaults change. The finished feature has no feature flags; the off switch is configuration
+(setting the live window to zero). The measurements behind this order: creating a Daytona sandbox takes about 1 second of a measured ~15-second turn; the
+rest is our own per-turn setup, stage by stage in research.md, and only a sandbox that stays
+running skips it (a resumed turn is roughly 2 to 3 seconds, estimated). The archive state is
+dropped entirely (restoring from archive is slower than creating fresh). Today's always-correct
+fallback (rebuild and replay the transcript) stays underneath everything.
+
+## Related workspaces
+
+- `docs/design/agent-workflows/projects/session-keepalive/`: the local, in-memory pool that already
+  gives non-Daytona sessions warm reuse. This project refactors that pool to be provider-aware;
+  its once-deferred Daytona slice is this project's park-to-running level.
+- `docs/design/agent-workflows/projects/harness-session-resume/`: how a restarted sandbox reloads the
+  past conversation. This is the fallback both levels rely on.
+- `docs/design/agent-workflows/projects/daytona-gate-delivery/`: the F-018 fix (approval gates on
+  Daytona), in implementation. This plan follows its pending-approval resume model.
+- `docs/design/agent-workflows/projects/qa/findings.md`: the QA findings referenced here. F-020 (this
+  slow-turn problem), F-018 (a separate Daytona bug that hangs tool calls), and F-017 (a mount bug
+  already fixed).

docs/design/agent-workflows/projects/warm-daytona-sessions/context.md

@@ -0,0 +1,127 @@
+# Context: warm and resumable Daytona sessions
+
+## What a user sees today
+
+When you chat with an agent that runs on Daytona, every message waits fifteen to twenty seconds
+before the agent starts to answer. QA measured this on a live run (the E3 scenario, a scripted
+chat used to test the agent runtime). The wait is the same on every turn. Your second message
+in a conversation waits just as long as your first, because the runner builds a brand-new
+sandbox for it, mounts its files, and starts the harness from scratch. Nothing from the
+previous turn is reused.
+
+A few terms, since they recur below:
+
+- **Sandbox**: the isolated cloud machine an agent runs in. On Daytona it is a billed resource.
+- **Runner**: the Agenta service that drives one agent turn. It creates the sandbox, runs the
+  turn inside it, and tears it down.
+- **Harness**: the agent program inside the sandbox (Claude Code or Pi).
+- **Park a sandbox**: stop it but keep its disk, so the next turn can restart the same one
+  instead of rebuilding it.
+
+The slow turn is the whole problem. This is QA finding F-020.
+
+Measurement sharpened where the wait goes (research.md, 2026-07-11; the cold turn measured about
+15 seconds on that day's runs). Creating the Daytona sandbox itself takes about 1 second. The
+rest is our own per-turn setup inside and around it: installing the Pi CLI (redundantly; the
+skip is already live on the dev sidecar), starting the harness, mounting files, uploading
+assets, and reloading the conversation. Research.md has the stage-by-stage table. That setup
+only disappears entirely when the sandbox, with its processes, stays running between turns. This
+is why the plan builds up to keeping sandboxes running for a short window, not just to stopping
+and restarting them.
+
+The answer itself is still correct. A separate feature, durable session continuity (PR #5197),
+saves the conversation to storage and replays it into the fresh sandbox, so the agent remembers
+what was said. The only cost is speed. Correctness is fine; every turn just pays the full build
+time.
+
+## What recent work already tried
+
+The working tree has moved past the state F-020 described. A recent, still-untested commit
+(`60990d396e`, "Resolve hot/warm/cold/dead/new lifecycle") already wired up most of the
+warm-reuse machinery. Any plan has to start from this real current state, not from F-020's
+older premise. Here is what that commit put in place:
+
+- **Keep the sandbox instead of deleting it.** `provider.ts` now creates the sandbox with
+  `ephemeral: false`, which means a stopped sandbox is parked, not auto-deleted. It also sets
+  three idle timers: stop after 5 minutes, move to cheaper cold storage after 15, delete after
+  30. Daytona runs these timers itself.
+- **Park at a clean turn end.** The teardown path (`sandbox_agent.ts`) takes a `keepWarm`
+  option. On a Daytona turn that finished cleanly, it calls `pauseSandbox()` (stop, keep the
+  disk) instead of `destroySandbox()` (delete). Both run paths request `keepWarm` only when the
+  turn succeeded, was not aborted, and the user did not disconnect.
+- **Reconnect on the next turn.** The runner stores the sandbox id, and on the next turn it
+  reads that id back and restarts the same sandbox instead of provisioning a fresh one.
+- **Reload the conversation in place.** A patch on the vendored `sandbox-agent` package adds a
+  native "reload this session" call, so a restarted sandbox resumes the harness where it left
+  off, with transcript replay as the fallback.
+
+So on paper, a follow-up turn should restart the parked sandbox and pick up the conversation.
+In practice it does not, for the reason below.
+
+## Why it still fails
+
+The runner asks for the right behavior, but the piece that carries it out is missing.
+
+The code that talks to Daytona is called the *provider*. The runner's park and reconnect calls
+depend on two provider functions: one to pause (stop) a sandbox, and one to reconnect to a
+stopped one. The Daytona provider implements neither. As a result:
+
+- `pauseSandbox()` finds no pause function and falls through to a plain delete. The sandbox is
+  gone at turn end, `keepWarm` or not.
+- The reconnect call finds a deleted sandbox id and cannot revive it, so the runner builds a
+  fresh sandbox anyway.
+
+The net effect is a full rebuild on every turn, exactly what F-020 reported. `research.md`
+walks the code that proves each step.
+
+## The key finding
+
+The warm-reuse machinery is mostly written already and sitting untested in the working tree.
+What blocks it is small and specific: two missing functions in the Daytona provider, plus a few
+correctness gaps around them that a design review surfaced. This project is a takeover. Verify
+the existing code, add the two functions, close the gaps, decide how the runner cleans up
+abandoned sandboxes, and decide how far to push reuse.
+
+Two older facts are worth carrying forward, because a later decision depends on them:
+
+- The Daytona sandbox used to be created `ephemeral: true`, which auto-deleted it the moment it
+  stopped. That was a safety backstop: a crashed runner could not leave a sandbox billing for
+  long. Switching to `ephemeral: false` removes that reflex, so the plan has to answer what now
+  cleans up an abandoned sandbox.
+- The local (non-Daytona) sessions already get warm reuse from an in-memory pool, but that pool
+  is deliberately kept off Daytona. A leaked local session costs only host memory; a leaked
+  Daytona sandbox costs real money. `research.md` covers the pool in full.
+
+## Goals
+
+- Make a second Daytona turn in the same conversation near-instant while the conversation is
+  active (the sandbox stays running for a short window), and cheap to resume after the window
+  closes (restart the stopped sandbox instead of rebuilding).
+- Keep the parked cost honest and bounded, as configuration with measured, conservative defaults:
+  storage only for a stopped sandbox, a short window and a hard cap on concurrently running
+  parked sandboxes.
+- Never leak a running sandbox. Cleanup of abandoned sandboxes has to be at least as safe as the
+  `ephemeral: true` behavior it replaces.
+- Reuse the existing local keepalive pool logic (`session-pool.ts`), refactored to be
+  provider-aware, instead of building a second pooling mechanism.
+- Keep durable continuity (PR #5197) as the always-correct fallback.
+
+## Non-goals
+
+- Fixing the Daytona tool-call hang (F-018, a separate bug with its own implementation workspace,
+  `daytona-gate-delivery`). Warm reuse helps chat first. Tool turns fail on Daytona until F-018
+  lands, and a failed turn does not park.
+- Routing across multiple runner replicas. The runner is single-replica; a miss just falls back
+  to a cold build.
+- Changing the wire contract, the SDK, or the frontend. This is a runner-only change.
+- Running the full app path against live Daytona during this design pass. One credit-controlled
+  lifecycle measurement was run directly against the Daytona API on 2026-07-11 (two sandboxes,
+  created and deleted, numbers in research.md); end-to-end verification through the app is the
+  plan's final slice, not done here.
+
+## Who is affected
+
+Anyone running a multi-turn chat agent on Daytona through the deployed app. Today they wait
+fifteen to twenty seconds per turn. The build-kit default agent feels it most, because its "read the
+skill first" instruction makes the model open with a tool call. That path is currently blocked
+by F-018 rather than by build latency, so it will not benefit until F-018 lands.

docs/design/agent-workflows/projects/warm-daytona-sessions/open-questions.md

@@ -0,0 +1,68 @@
+# Open questions
+
+The decisions a human still has to make, each with the context and trade-offs needed to make it
+from this file alone. Answered questions move to the bottom with their answers.
+
+## Still open
+
+1. **Which counter guards the sandbox-pointer write** (Slice 1). Background, in brief (plan.md
+   must-fix item 3 has the full story): the runner records which sandbox belongs to a
+   conversation in a database row. Two turns racing on one conversation can leave the row
+   pointing at the wrong sandbox, which orphans one instance (it runs until the idle timers reap
+   it, cents per incident) and sends the next turn through one doomed reconnect. The fix is to
+   wait for the write and make it conditional, so an older turn cannot overwrite a newer one.
+   The open choice is what the condition compares against:
+   - The turn counter already on the `session_states` row. No schema change; the counter is
+     maintained by the continuity code, so this couples the pointer guard to that code's
+     correctness.
+   - A new generation column on `session_states`. Cleanest semantics, needs a schema migration.
+   - The Redis session-owner claim PR #5197 added. No schema change and already per-session,
+     but Redis contents can be flushed, and a flushed key silently drops the guard.
+   Related: PR #5197's own durable-continuity write has the same unguarded pattern, so one
+   mechanism should probably cover both writes.
+
+2. **What happens to sandboxes when the runner shuts down** (Slice 2). Background: the runner
+   process stops on every deploy, restart, or scale-down (a SIGTERM). At that moment some
+   conversations have a turn mid-flight and others have an idle parked sandbox. The choice, per
+   sandbox:
+   - Turn mid-flight: delete, or stop? Delete is safe: the transcript may be missing the last
+     exchange, and restarting later from a half-written state risks a wrong resume. Stop would
+     save the next turn about a second but carries that risk.
+   - Idle: stop, or delete? Stop means a rolling deploy keeps the parked disks, and the next
+     turn restarts one instead of rebuilding. Delete is simpler and forces the next turn cold.
+   The plan proposes delete for mid-flight, stop for idle, and `/kill` keeps its hard-delete
+   meaning. The cost difference is about a second per affected conversation either way; the
+   proposal favors correctness for mid-flight and warmth for idle. Confirm or override.
+
+3. **Does the session reload behave identically on a reattached same-instance sandbox?** With the
+   harness transcript mounts enabled, the transcript comes from the durable mounted files either
+   way, so it should. Confirm live in Slice 5, including the mount-hygiene case (an old file mount
+   with expired credentials surviving a stop and start; see plan.md must-fix item 4).
+
+## Answered (kept for the record)
+
+- **Warm-cap overflow behavior**: when every warm slot is taken, a finishing turn parks its
+  sandbox to stopped instead of keeping it running. Active turns are never blocked by the warm
+  cap; there is no queue and no rejection. (Mahmoud, PR review.)
+- **The live-window default and the off switch**: two minutes
+  (`AGENTA_RUNNER_DAYTONA_SESSION_IDLE_TTL_MS=120000`), and setting it to 0 disables keeping
+  sandboxes running. No separate enabled flag, and no feature flags in the finished feature; the
+  controls are env vars. (Mahmoud, PR review.)
+- **The warm cap default**: 20 (`AGENTA_RUNNER_DAYTONA_SESSION_MAX_WARM`), sized so roughly 20
+  parallel conversations can all stay warm; worst case about $3.33/hour while fully loaded,
+  bounded in time by the two-minute window. A separate variable from the local pool's
+  `AGENTA_RUNNER_SESSION_POOL_MAX`, because that one budgets host memory and this one budgets
+  billed compute. (Mahmoud, PR review.)
+- **The stop timer**: 15 minutes, env-overridable via `DAYTONA_AUTOSTOP`. The timer resets on
+  every turn, so an active conversation never hits it; it only fires after 15 minutes of
+  silence. (Mahmoud, PR review.)
+- **Auto-archive**: removed entirely, not configured around. The create call drops the
+  `autoArchiveInterval` field and the `DAYTONA_AUTOARCHIVE` override. Restoring from archive
+  (33 to 66 seconds measured) is slower than a fresh create (1.2 to 1.7 seconds). The ladder is
+  stop, then delete. (Mahmoud, PR review; measurement.)
+- **Abandonment compute budget**: measured at about 1.4 to 4 cents per crashed-runner incident
+  depending on the stop timer, plus a fraction of a cent of storage until the delete timer. No
+  separate sweeper is needed; the timers are the sweeper.
+- **Should park-to-running ship at all**: yes. Sandbox creation is about 1 second of a
+  ~15-second turn, so park-to-stopped cannot fix latency; only a sandbox that stays running
+  skips the per-turn pipeline. Park-to-running is Slice 4 of the main line.