diff --git a/.agents/skills/agent-release-gate/SKILL.md b/.agents/skills/agent-release-gate/SKILL.md new file mode 100644 index 0000000000..270b743705 --- /dev/null +++ b/.agents/skills/agent-release-gate/SKILL.md @@ -0,0 +1,70 @@ +--- +name: agent-release-gate +description: >- + Run the agent release gate — a portable, wire-level QA harness for the agent runtime. + Drives the same product endpoint the playground drives and asserts on the SSE frame + stream and real side effects, never on model prose, so it works against any deployment + (cloud or self-hosted) from three env vars. Use before an agent-workflows release, or + after changing the runner, the SDK agent adapters, the runner Docker images, or the + agent service. Triggers: "run the release gate", "QA the agent runtime", "does the + agent still work end to end", "pre-release agent QA". +allowed-tools: Read, Edit, Write, Grep, Glob, Bash +user-invocable: true +--- + +# Agent release gate + +Product-level sanity QA for the agent runtime, one layer below the playground UI. The question +is not "is every detail right" — it is "**if a user opens the product and does the obvious first +things, do they work?**" This is the gate a release passes before shipping. + +Every check asserts on the **wire** (the SSE frame types the browser sees) and on **side effects** +(the file really persisted, the revision really incremented) — never on what the model says. That +makes it deployment-agnostic: point it at any stack and the assertions still hold. + +## Run it + +Set three environment variables for the deployment under test, then run the gate: + +```bash +export AGENTA_BASE=https://your-stack.example.com # deployment origin +export AGENTA_PROJECT_ID=... # target project +export AGENTA_API_KEY=... # project API key + +uv run resources/qa_product.py --all # every cell, every journey +uv run resources/qa_product.py --cell P1 # one cell +uv run resources/qa_product.py --cell C1 --only chat # one journey +``` + +Paths are relative to this skill's directory. The deployment's vault must hold the provider keys +the cells use (Anthropic / OpenAI / OpenRouter). If the three env vars are unset the driver stops +immediately and names exactly what is missing; a legacy `--env-file ` fallback also exists. + +**Reading the result.** Each journey prints `PASS`, `FAIL`, or `SKIP` with a one-line reason, and +a per-cell markdown table lands with the full JSON in `./qa-gate-runs//` (override the +location with `AGENTA_QA_RUNS_DIR`). Runs are written to the current working directory, never into +the skill. `SKIP` is expected where a journey does not apply to a cell (for example `mcp` on any Pi +cell — user MCP is Claude-only). Any `FAIL` blocks the release until triaged. + +## When results lie + +The runtime **fails open**: a component can break, get logged, and the turn still succeeds with a +normal-looking answer. A green turn is therefore not proof on its own. Before trusting a pass, +read `resources/LESSONS.md` — every trap there produced a green test that proved nothing. The two +that bite hardest: replay conversation history byte-faithfully (tool parts included) or every turn +silently goes cold, and re-run any prior blocker-level finding after a redeploy before believing it. + +## Resources (read on demand) + +- `resources/coverage.md` — the cells (harness × sandbox × auth) and journeys (chat, mount, tool, + approve, deny, commit, warm, mcp) with a one-line meaning for each. +- `resources/LESSONS.md` — the traps. Read before writing or trusting any agent QA test. +- `resources/qa_product.py` — the gate driver (cells × journeys). +- `resources/qa_probe.py` — a one-turn wire probe: `uv run resources/qa_probe.py` confirms the + product path answers at all before running the full gate. +- `resources/qa_longctx.py` — optional long-context / Gmail / concurrent-session probes. Needs + live Gmail and GitHub Composio connections in the target project; skip it otherwise. +- `resources/seeds/` — representative green `results.json` files kept as regression-seed references. + +Release-night findings and the full evidence history are archived in +`docs/design/agent-workflows/projects/qa/` (STATUS.md, findings.md, matrix.md). diff --git a/.agents/skills/agent-release-gate/resources/LESSONS.md b/.agents/skills/agent-release-gate/resources/LESSONS.md new file mode 100644 index 0000000000..493e80d8f2 --- /dev/null +++ b/.agents/skills/agent-release-gate/resources/LESSONS.md @@ -0,0 +1,210 @@ +# QA lessons: the traps, and how to not re-learn them + +Written during the 2026-07-14 pre-release QA. Every item here cost real time to discover. Read +this BEFORE writing another agent QA test — most of these produce a **green test that proves +nothing**, which is worse than a red one. + +The one-line summary: **assert on the wire and on side effects, never on model prose — and make +your test client behave EXACTLY like the real frontend, or you are testing your own bug.** + +--- + +## 1. The test client must replay history byte-faithfully, or every turn silently goes COLD + +**The trap.** Our driver replayed each assistant turn as a text-only message +(`{role:"assistant", parts:[{type:"text",...}]}`), dropping the assistant's **tool parts**. + +The runner fingerprints the conversation over **(ordered user texts, ordered deduped tool-call +ids, user-turn count)** — `session-pool.ts:226` `historyFingerprint`, and `:252` +`expectedNextHistoryFingerprint`, which folds in the tool-call ids the runner emitted last turn. +A replay with no tool-call ids therefore **cannot match** after any tool-using turn: + +``` +[keepalive] mismatch (history) key=…; evict + cold +``` + +**Why it poisons everything.** Every turn goes cold → a fresh harness process → the runner replays +a hand-rendered transcript instead of the harness's real context. So: +- warm/cold numbers are meaningless (nothing was ever warm), +- **compaction never triggers** (the harness context never accumulates), so a long-context / + "loses information" test can pass while testing nothing at all. + +**The rule.** Echo back the **full** assistant `UIMessage.parts` — text parts *and* `tool-` +parts with `toolCallId`, `input`, `state`, `output` — exactly as the AI SDK does +(`web/packages/agenta-playground/src/state/execution/agentRequest.ts:401`). If your driver +synthesizes assistant turns, it is not testing the product. + +**The tell.** `grep 'mismatch (history)'` in the runner log. If it fires on turns your client +believes are warm, your client is the bug. + +## 2. Never assert on the model's prose. It will lie to you. + +First version of the tool test asked the agent to run `echo "QA-BASH-$((6*7+1))"` and asserted the +reply contained `QA-BASH-43`. The model **computed 43 itself and reported it without running +bash** — so a *denied* tool call still produced a "passing" reply. The wire said denied; the text +said success. + +**The rule.** Ground truth is the frame stream: +- tool executed → `tool-output-available` +- tool refused → `tool-output-error` / `tool-output-denied` +- approval raised → `tool-approval-request` +A token in the text is only ever *corroborating* evidence, and only if the model **cannot compute +it** (use a container hostname, a random file's contents — never arithmetic stated in the prompt). + +## 3. Scope tool assertions to ONE tool call, keyed by its INPUT + +A turn routinely contains several tool calls — an auto-approved read-only one alongside the gated +one. A turn-wide "did any tool run?" gives false failures. + +And you cannot key on `toolCallId`: **on approval-resume the harness RE-ISSUES the gated call under +a brand-new `toolCallId`**. You cannot key on the tool name either: Claude calls the shell +`Terminal`, Pi calls it `Bash`. **Key on the tool's `input`** (the command itself). + +## 4. `tool-input-available` carries INCOMPLETE input, and the tool name changes case mid-stream + +The frame fires repeatedly for one call, streaming a progressively-built partial input: + +``` +toolName "bash" input {"command":"echo \"QA-BASH-"} <- partial! +toolName "bash" input {"command":"echo \"QA-BASH-$(hostname"} +toolName "Bash" input {"command":"echo \"QA-BASH-$(hostname)\""} <- complete; name case flips +``` + +Take the **last** frame per `toolCallId`. Taking the first — a reasonable reading of "available" — +approves a **truncated command under the wrong name**; the runner keys approval decisions by +name+args, so the decision misses the parked gate and **the approval re-parks forever**. +(Reported as F-5: the frame name is a genuine wire-hygiene bug.) + +## 5. Approvals are IN-BAND. The REST route is a different product. + +The browser approves by re-POSTing the whole message history to `/invoke` with the tool part set to +`state:"approval-responded"`, `approval:{id, approved}`. There IS a REST endpoint +(`/api/sessions/interactions/{id}/respond`) but it is the **out-of-band Slack/trigger** path. +Testing it tests code the UI never runs. + +## 6. A paused turn "finishes" + +An approval-paused turn ends with `finish.finishReason: "other"`, not a distinct status. "The turn +ended" does NOT mean "the turn completed". Assert the reason. + +## 7. `code` tools do not exist on the product path + +The sidecar rejects them: *"Code tools are not supported by the sidecar."* They only work against +the in-process service — which is what the OLD driver (`run_matrix.py`) targets, and why copying +its scenarios into a product-path test fails instantly. The product's real tool surface is +`builtin` / `gateway` / `mcp`. + +## 8. Gateway tools: discovery output is NOT config input, and the action has no prefix + +- The action is **`FETCH_EMAILS`**, not `GMAIL_FETCH_EMAILS`. The prefixed name appears inside the + tool's own description text, which is how you get seduced into using it. Wrong name → run fails + with `Action not found: composio/gmail/GMAIL_FETCH_EMAILS (HTTP 404)`. +- `/api/tools/discover` returns the tool WITH `input_schema` + `description`; `GatewayToolConfig` + **forbids** those keys. Feeding discovery's own output back into the agent config 500s with + `extra_forbidden`. Strip to `{type, provider, integration, action, connection, name, permission}`. + (Reported as F-8 — the round trip should just work.) + +## 9. NEVER diagnose from a run that overlapped a container restart + +A full matrix run showed every cell failing with 500s (`Could not verify credentials … 404`), plus +a UI-visible `404 on /api/workflows/revisions/resolve`, plus `[sessions/persist] DROPPED … fetch +failed` and `getaddrinfo ENOTFOUND api`. **All phantoms** — another agent was recreating the +api/worker containers at that moment. Everything went green on re-run. + +Check `docker ps` uptimes before believing a failure. Re-run before reporting. + +## 10. Read-only by construction when real accounts are connected + +The project has live Gmail and GitHub connections. QA must never send mail, reply to a thread, or +write to GitHub as a side effect. Derive tools from read-only use-cases AND filter any action whose +name contains SEND/REPLY/CREATE/DELETE/UPDATE/MODIFY/TRASH/DRAFT/MERGE before it reaches an agent. + +## 11. The product fails OPEN, so absence of an error means nothing + +The recurring shape of every serious bug this pass: **a component fails, the runner logs it and +carries on.** The turn succeeds. The UI looks normal. + +- mounts 503 → run in a throwaway `/tmp` cwd, every file lost (F-1) +- Pi permission extension can't install → **run with no enforcement**; `ask` never asks, `deny` + never denies (F-3) +- Daytona's tunnel to the store fails → skip the mount, "not fatal"; files never persist (F-7) +- session records fail to POST → dropped after 3 retries, turn proceeds + +**Therefore: a passing turn is not evidence.** For every capability, verify the side effect +(the file is really there next turn; the commit really exists) and grep the runner log for +`degraded|skipped|without this mount|tunnel discovery failed|DROPPED|cold`. + +## 12. Environment-shaped bugs hide behind image differences + +Pi's permission enforcement failed only on the deployment's image, because +`PI_CODING_AGENT_DIR=/pi-agent` doesn't exist there and the runner runs as uid 1000; our EE dev +image runs as root and ships the dir. Same code, opposite behavior. **Always check the container's +user and the actual paths** (`docker exec … id; ls -ld `) before concluding the code is fine. +And note a workaround applied with `docker exec` is **lost on container recreate** — re-verify it +before every batch. + +--- + +## 13. Findings expire on redeploy — re-run blocker-level findings after the stack is rebuilt + +F-9 ("Claude harness never resumes its native session") was CONFIRMED across 72h of real traffic +and triaged as a release blocker. A deployment repair landed later the same day, pulling in recent +upstream fixes. Nobody re-ran F-9 against the rebuilt stack before trusting it — until a decisive +cold-context experiment on 2026-07-14 showed native session resume now working 4/4 runs, downgrading +F-9 to a residual resilience concern (see STATUS.md). + +**The trap.** A deployment under active repair invalidates earlier observations made against it. +Once the repair lands, the finding is stale, not necessarily wrong — but you don't know which +until you re-check. Treating "CONFIRMED" as permanent past a redeploy is how a fixed bug survives +in a triage doc as a blocker. + +**The rule.** For any blocker-level finding, record WHICH build/commit/deploy window it was +observed on. After any redeploy that touches the relevant code path, re-run the decisive experiment +before shipping a release decision on that finding — do not just re-read the old evidence. + +## 14. The v0 revision is a SEED — a committed config only persists on the SECOND commit + +Committing an agent config as a workflow revision (`POST /api/workflows/revisions/commit`) looks +like it stores your `data.parameters` immediately. It does not on the first commit. The DAO +force-nulls `data`/`flags`/`meta` for **version 0** (`api/oss/src/dbs/postgres/git/dao.py` +`_null_revision_fields`, `if revision.version == "0"`). So a fresh variant's first commit is an +empty seed; your config lands on the **second** commit (v1). A test that commits once and asserts +`data.parameters == X` fails with `KeyError: 'data'` and looks like a broken endpoint — it is not. +Commit twice (seed, then the real change) and assert v0→v1 plus the changed field surviving a +`GET /api/workflows/revisions/{id}`. Also: `data` is `extra="forbid"` — only +`{uri,url,headers,runtime,script,schemas,parameters}` are accepted. + +Second trap in the same area: this is a WORKFLOW-revision commit, NOT the in-stream +`data-committed-revision` SSE frame (which is a different mechanism — the agent committing during a +turn) and NOT a git commit. The playground's Save/Commit button hits the REST route above. + +## 15. User MCP servers are Claude-only, public-HTTPS-only, and the harness dials them + +Three things will each silently break an MCP smoke test: + +- **Pi rejects any run that declares `mcps`** (`run-plan.ts` `PI_USER_MCP_UNSUPPORTED_MESSAGE`). + User MCP needs a harness with `capabilities.mcpTools` — i.e. **Claude**. Do not smoke-test MCP on + a Pi cell; SKIP it there. +- **A local MCP server is unreachable.** The SDK resolver AND the runner both run an SSRF guard + (`assert_endpoint_url_allowed` / `validateUserMcpUrl`) that rejects `http://` and + private/loopback/metadata hosts unless `AGENTA_INSECURE_EGRESS_ALLOWED` / + `AGENTA_AGENT_MCPS_HOST_ALLOWLIST` is set (neither is, on bighetzner). Use a **public HTTPS** + server. DeepWiki (`https://mcp.deepwiki.com/mcp`, no auth) works. +- **The harness — not the runner process — opens the connection**, from the runner host on `local` + (from the sandbox on Daytona). The endpoint must be reachable from wherever the harness runs. + +The config entry is a full object, not a URL string: +`{"name","connection":{"type":"http","url":...},"policy":{"tools":{"mode":"all"}}}`. Assert on the +wire: a `tool-output-available` frame for a tool named `mcp____`. + +## The checklist for the next QA run + +1. `docker ps` — is anything restarting? If yes, wait. +2. Does the runner have its harness dirs (`/pi-agent`)? Is it root or not? +3. Drive the **product path** (`/services/agent/v0/invoke`), not the service `/invoke`. +4. Echo history **faithfully** (tool parts included), then confirm `hit-continue` in the log. +5. Assert on frames + side effects. Never on prose. +6. After every capability passes, grep the log for silent degradation. +7. Re-run anything that failed once before reporting it. +8. Before trusting an existing blocker-level finding, check whether the stack has been redeployed + since it was observed — if so, re-run the decisive experiment. diff --git a/.agents/skills/agent-release-gate/resources/coverage.md b/.agents/skills/agent-release-gate/resources/coverage.md new file mode 100644 index 0000000000..dc20b798d6 --- /dev/null +++ b/.agents/skills/agent-release-gate/resources/coverage.md @@ -0,0 +1,51 @@ +# Coverage: the cells and the journeys + +The gate is the product of two lists: **cells** (the configurations under test) and **journeys** +(the user actions run against each cell). `qa_product.py` defines both; this file is the reference. + +## Cells (harness × sandbox × auth) + +The core axis is harness × sandbox. Provider and auth mode are a sub-matrix run inside the Pi cells +only, because that is an authentication question, not a sandbox one — re-running it in all four core +cells would test the same code twice. + +| Cell | Harness | Sandbox | Model | Auth mode | Why this cell exists | +|---|---|---|---|---|---| +| C1 | `claude` | `local` | `sonnet` (alias) | subscription (OAuth) | Claude on the local sandbox; the default "use my subscription" path. A full model id gets dropped to the default on the Claude ACP path, so the gate pins the `sonnet` alias (finding F-007). | +| C2 | `claude` | `daytona` | `sonnet` | vault key | Claude in a cloud sandbox. Daytona rejects subscription auth by design, so this cell genuinely needs a funded Anthropic vault key. | +| C3 | `pi_core` | `local` | `gpt-5.6-luna` | vault key (OpenAI) | Pi on the local sandbox with a managed OpenAI key. | +| C4 | `pi_core` | `daytona` | `gpt-5.6-luna` | vault key (OpenAI) | Pi in a cloud sandbox; the remote-mount path that surfaced the silent file-loss finding (F-7). | +| P1 | `pi_core` | `local` | `openrouter/deepseek/deepseek-v4-flash` | vault key (OpenRouter) | OpenRouter as a first-class native provider. | +| S1 | `pi_core` | `local` | `gpt-5.6-luna` | subscription (Codex OAuth) | The ChatGPT/Codex subscription path via the sidecar, independent of any vault key. | +| P2 | `pi_core` | `local` | `deepseek/deepseek-v4-flash` | custom OpenAI-compatible provider | OpenRouter reached as a custom OpenAI-compatible endpoint — the path every self-hoster with a proxy or local vLLM uses, and the least-travelled one. Needs a `custom_provider` vault slug; pass `--custom-slug`. | + +The pinned models and connection modes are the gate's **fixtures**: each is chosen for a reason +(alias vs full id on Claude, subscription vs vault where the sandbox forces it, a healthy provider +for the long-context probe). The inline comments in `qa_product.py` carry the specific reason per +cell — keep them in sync if a cell changes. + +## Journeys (run in every applicable cell) + +| Journey | What it does | Passes when | +|---|---|---| +| `chat` | Create an agent, send one message. | The turn completes with a `finish` frame, not an `error`. | +| `mount` | Write a file in turn 1, read it back in turn 2. | The file survives across turns — proof the durable mount is real, not a throwaway `/tmp` cwd. | +| `tool` | Call a tool whose return bakes in an unguessable token. | The token appears in the reply, so the tool provably ran (the model cannot guess it). | +| `approve` | Raise an approval, then approve it. | The approved tool call continues via the in-band approval protocol the browser uses. | +| `deny` | Raise an approval, then deny it. | The denied path is handled cleanly (no phantom failure, no re-parking forever). | +| `commit` | Save an agent config as a new workflow revision, then fetch it back. | The changed parameter survives the round trip and the version bumps (v0 seed → v1; see LESSONS #14). Harness-agnostic — it drives the config REST API, not a turn. | +| `warm` | Run three turns, watch latency and the runner log. | Turns 2-3 are faster and the log confirms the session was genuinely **loaded**, not silently cold. | +| `mcp` | Deliver an MCP server in the agent config and call one of its tools. | A `tool-output-available` frame fires for an `mcp__*` tool. **Claude only** — Pi rejects user MCP, so this `SKIP`s on every Pi cell. Uses the public DeepWiki server by default; override with `--mcp-url`. | + +Triggers are deliberately **out of scope** for this gate. + +## Optional probes (`qa_longctx.py`) + +Separate from the gate, these need live **Gmail and GitHub Composio connections** in the target +project. Skip them if the project has none. + +| Probe | What it catches | +|---|---| +| `memory` | Plant a token, flood the context with bulky tool output across many turns, then ask for the token back. Catches compaction dropping early context. | +| `gmail` | The Gmail/GitHub gateway tools resolve and actually execute. Read-only actions only — writes (SEND/REPLY/CREATE/…) are filtered before they reach an agent. | +| `concurrent` | N sessions run at once, each holding a different token. Catches cross-session bleed a single-session test can never see. | diff --git a/.agents/skills/agent-release-gate/resources/qa_longctx.py b/.agents/skills/agent-release-gate/resources/qa_longctx.py new file mode 100644 index 0000000000..9a95284f2d --- /dev/null +++ b/.agents/skills/agent-release-gate/resources/qa_longctx.py @@ -0,0 +1,319 @@ +# /// script +# requires-python = ">=3.10" +# dependencies = ["httpx>=0.27"] +# /// +"""Long-conversation / compaction / many-tools QA. + +Three probes, all wire-asserted: + + memory A token is planted in turn 1, then the context is FLOODED (bulky Gmail payloads + + large bash output) across many turns. The last turn asks for the token back. If Pi's + compaction drops it, the token does not come back. This is the reported bug. + + gmail The Gmail (Composio gateway) tools resolve and actually execute. Read-only actions + ONLY -- GMAIL_REPLY_TO_THREAD is deliberately excluded so QA never sends mail. + + concurrent N sessions run at the SAME time, each holding a DIFFERENT token. At the end each is + asked for its own. Catches cross-session bleed (session A answering with B's token), + which a single-session test can never see. + + uv run qa_longctx.py --sandbox local --probe memory --turns 12 + uv run qa_longctx.py --sandbox daytona --probe all +""" + +from __future__ import annotations + +import argparse +import json +import pathlib +import time +import uuid +from concurrent.futures import ThreadPoolExecutor + +import importlib.util + +_spec = importlib.util.spec_from_file_location( + "qa", pathlib.Path(__file__).resolve().parent / "qa_product.py" +) +qa = importlib.util.module_from_spec(_spec) +_spec.loader.exec_module(qa) + +# Tools come from /api/tools/discover VERBATIM. Do NOT hand-write the action name: the action is +# `FETCH_EMAILS`, not `GMAIL_FETCH_EMAILS` (the integration prefix is not part of it), and a wrong +# name fails the whole run with a 500 "Action not found ... (HTTP 404)". Discovery hands back a +# ready-to-use tool object, which is also what the builder agent does — so this exercises the real +# path. +# +# READ-ONLY BY CONSTRUCTION: the use-cases below only ever resolve to read actions. QA must never +# send mail or write to GitHub from a real connected account as a side effect. +READ_ONLY_USE_CASES = [ + "read my gmail inbox emails", + "list my gmail threads", + "list my github repositories", + "read github issues", +] +# Any action whose name suggests a write is dropped, belt-and-braces, before it reaches an agent. +_WRITE_ISH = ( + "SEND", + "REPLY", + "CREATE", + "DELETE", + "UPDATE", + "MODIFY", + "TRASH", + "DRAFT", + "MERGE", +) +BASH = {"type": "builtin", "name": "bash"} + + +def discover_tools() -> list: + import httpx + + r = httpx.post( + f"{qa.BASE}/api/tools/discover", + headers={ + "Authorization": f"ApiKey {qa.KEY}", + "Content-Type": "application/json", + }, + json={"use_cases": READ_ONLY_USE_CASES}, + timeout=90.0, + ) + r.raise_for_status() + tools = [c["tool"] for c in r.json().get("capabilities", [])] + # QA FINDING F-8: /tools/discover returns a gateway tool WITH `input_schema` + `description`, + # but GatewayToolConfig forbids extra keys, so feeding discovery's own output back into the + # agent config 500s with `extra_forbidden`. The discover -> configure round trip is broken. + # Strip to the accepted key set so the rest of QA can proceed. + allowed = { + "type", + "provider", + "integration", + "action", + "connection", + "name", + "permission", + } + tools = [{k: v for k, v in t.items() if k in allowed} for t in tools] + safe = [ + t + for t in tools + if not any(w in t.get("action", "").upper() for w in _WRITE_ISH) + ] + dropped = [t["action"] for t in tools if t not in safe] + if dropped: + print(f" dropped write-capable actions: {dropped}", flush=True) + print( + f" discovered {len(safe)} read-only tools: {[t['action'] for t in safe]}", + flush=True, + ) + return safe + + +# Populated in main() after credentials resolve — discovery hits the live /api/tools/discover +# endpoint, so it cannot run at import time (that would break --help with no credentials). +GATEWAY_TOOLS: list = [] + + +# gpt-5.6-luna/openai currently returns "The agent produced no output" on this deployment (a +# regression under separate investigation), so the long-context probe runs on OpenRouter, which is +# healthy. The harness and session-pool path are identical. +def cell(sandbox: str) -> dict: + return { + "harness": "pi_core", + "sandbox": sandbox, + "model": "openrouter/deepseek/deepseek-v4-flash", + "provider": "openrouter", + } + + +def params(sandbox: str, tools: list) -> dict: + return qa.template( + cell(sandbox), + tools=tools, + instructions=( + "You are a QA assistant. Follow instructions exactly. When asked to remember " + "something, remember it for the whole conversation." + ), + permission_default="allow", + ) + + +def probe_gmail(sandbox: str) -> dict: + """Do the Gmail tools resolve AND execute?""" + s = str(uuid.uuid4()) + p = params(sandbox, GATEWAY_TOOLS + [BASH]) + t = qa.invoke( + s, + [ + qa.user_msg( + "List the subjects of the 3 most recent emails in my inbox, then list my GitHub repositories. Use the tools." + ) + ], + p, + timeout=420.0, + ) + ok = qa.tool_ran(t) and not t.errors + return { + "pass": ok, + "why": "a Gmail gateway tool executed (tool-output-available) with no error", + "tools_called": [c.get("toolName") for c in t.tool_calls], + "turn": t.summary(), + } + + +def probe_memory(sandbox: str, turns: int) -> dict: + """Plant a token, flood the context, then ask for it back.""" + s = str(uuid.uuid4()) + token = f"QA-MEM-{uuid.uuid4().hex[:12].upper()}" + p = params(sandbox, GATEWAY_TOOLS + [BASH]) + + msgs = [ + qa.user_msg( + f"Remember this exact token for the rest of our conversation: {token}. " + "Do NOT write it to any file. Just reply: OK" + ) + ] + t = qa.invoke(s, msgs, p, timeout=420.0) + msgs.append(t.assistant_message()) + if t.errors: + return {"pass": False, "why": "turn 1 (plant) errored", "turn": t.summary()} + + # Flood the context. Alternate bulky Gmail payloads with large bash output — this is what a + # real user's long, tool-heavy session looks like, and it is what triggers compaction. + filler = [ + "Fetch my 5 most recent emails with the Gmail tool and summarize each in one line.", + "Use bash to run: seq 1 800 | paste -sd, - and report the last 20 characters only.", + "List my Gmail threads, then list my GitHub repositories. Report only the counts.", + 'Use bash to run: for i in $(seq 1 60); do echo "line-$i: $(head -c 40 /dev/urandom | base64)"; done and report only the final line.', + ] + trace = [] + for i in range(turns): + q = filler[i % len(filler)] + msgs.append(qa.user_msg(q)) + t = qa.invoke(s, msgs, p, timeout=420.0) + msgs.append(t.assistant_message()) + trace.append( + {"turn": i + 2, "ms": t.ms, "tools": len(t.tool_calls), "err": t.errors[:1]} + ) + print( + f" flood turn {i + 2}/{turns + 1}: {t.ms}ms tools={len(t.tool_calls)}", + flush=True, + ) + + msgs.append( + qa.user_msg( + "What was the exact token I asked you to remember at the very start? Reply with only the token." + ) + ) + final = qa.invoke(s, msgs, p, timeout=420.0) + ok = token in final.reply + return { + "pass": ok, + "why": f"the token planted in turn 1 survived {turns} tool-heavy turns (token={token})", + "token": token, + "recalled": final.reply[:120], + "flood": trace, + "session_id": s, + } + + +def probe_concurrent(sandbox: str, n: int = 3) -> dict: + """N simultaneous sessions, each with its own token. Any cross-answer is a leak.""" + tokens = {i: f"QA-CONC{i}-{uuid.uuid4().hex[:8].upper()}" for i in range(n)} + + def one(i: int) -> dict: + s = str(uuid.uuid4()) + p = params(sandbox, GATEWAY_TOOLS + [BASH]) + tok = tokens[i] + msgs = [qa.user_msg(f"Remember this token: {tok}. Reply only: OK")] + t = qa.invoke(s, msgs, p, timeout=420.0) + msgs.append(t.assistant_message()) + msgs.append( + qa.user_msg( + "Fetch my 3 most recent emails with the Gmail tool and summarize them." + ) + ) + t = qa.invoke(s, msgs, p, timeout=420.0) + msgs.append(t.assistant_message()) + msgs.append( + qa.user_msg( + "What token did I ask you to remember? Reply with only the token." + ) + ) + t = qa.invoke(s, msgs, p, timeout=420.0) + return { + "i": i, + "expected": tok, + "reply": t.reply[:80], + "session": s, + "errors": t.errors[:1], + } + + with ThreadPoolExecutor(max_workers=n) as ex: + results = list(ex.map(one, range(n))) + + own = all(r["expected"] in r["reply"] for r in results) + # The leak check: did any session echo ANOTHER session's token? + leaks = [ + {"session": r["i"], "leaked_token_of": j} + for r in results + for j, tok in tokens.items() + if j != r["i"] and tok in r["reply"] + ] + return { + "pass": own and not leaks, + "why": f"each of {n} concurrent sessions recalled ITS OWN token and none leaked another's", + "leaks": leaks, + "results": results, + } + + +def main() -> int: + ap = argparse.ArgumentParser() + ap.add_argument("--sandbox", default="local", choices=["local", "daytona"]) + ap.add_argument( + "--probe", default="all", choices=["all", "gmail", "memory", "concurrent"] + ) + ap.add_argument( + "--turns", type=int, default=12, help="flood turns for the memory probe" + ) + ap.add_argument( + "--env-file", + help=f"credentials file (fallback when env vars are unset; default {qa.DEFAULT_ENV_FILE})", + ) + args = ap.parse_args() + + qa.resolve_credentials(args.env_file) + global GATEWAY_TOOLS + GATEWAY_TOOLS = discover_tools() + + out: dict = {} + probes = ["gmail", "memory", "concurrent"] if args.probe == "all" else [args.probe] + for name in probes: + print(f"[{args.sandbox}] {name} ...", flush=True) + try: + if name == "gmail": + r = probe_gmail(args.sandbox) + elif name == "memory": + r = probe_memory(args.sandbox, args.turns) + else: + r = probe_concurrent(args.sandbox) + except Exception as e: + r = {"pass": False, "why": f"driver exception: {type(e).__name__}: {e}"} + out[name] = r + print( + f"[{args.sandbox}] {name}: {'PASS' if r.get('pass') else 'FAIL'} — {r.get('why', '')}\n", + flush=True, + ) + + stamp = time.strftime("%Y%m%d-%H%M%S") + d = qa.RUNS / f"longctx-{args.sandbox}-{stamp}" + d.mkdir(parents=True, exist_ok=True) + (d / "results.json").write_text(json.dumps(out, indent=2)) + print(f"results: {d}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/.agents/skills/agent-release-gate/resources/qa_probe.py b/.agents/skills/agent-release-gate/resources/qa_probe.py new file mode 100644 index 0000000000..baab2a4478 --- /dev/null +++ b/.agents/skills/agent-release-gate/resources/qa_probe.py @@ -0,0 +1,172 @@ +# /// script +# requires-python = ">=3.10" +# dependencies = ["httpx>=0.27"] +# /// +"""Minimal wire probe: can we drive the product path (/services/agent/v0/invoke) at all? + +Sends one turn with an inline agent config and prints every SSE frame type seen, plus the +assistant text. If this works, the full QA driver is just scenarios on top of it. + + uv run qa_probe.py --harness pi_core --sandbox local --model gpt-5.6-luna +""" + +from __future__ import annotations + +import argparse +import json +import os +import pathlib +import sys +import uuid + +import httpx + +# Credentials come from the environment FIRST (AGENTA_BASE, AGENTA_PROJECT_ID, AGENTA_API_KEY), +# then from an env file (default below, overridable with --env-file). Resolved in main() so that +# --help works with no credentials present. +REQUIRED_CREDS = ("AGENTA_BASE", "AGENTA_PROJECT_ID", "AGENTA_API_KEY") +DEFAULT_ENV_FILE = pathlib.Path.home() / ".agenta-bighetzner.env" + +BASE = "" +PROJECT = "" +KEY = "" + + +def _read_env_file(path: pathlib.Path) -> dict: + values: dict = {} + path = pathlib.Path(path).expanduser() + if not path.exists(): + return values + for line in path.read_text().splitlines(): + line = line.strip() + if line and not line.startswith("#") and "=" in line: + k, v = line.split("=", 1) + values[k.strip()] = v.strip() + return values + + +def resolve_credentials(env_file: str | pathlib.Path | None = None) -> None: + """Populate BASE/PROJECT/KEY from the environment first, then the env file. Raises SystemExit + naming exactly which credentials are missing.""" + global BASE, PROJECT, KEY + file_values = _read_env_file(env_file or DEFAULT_ENV_FILE) + resolved: dict = {} + missing: list = [] + for name in REQUIRED_CREDS: + value = os.environ.get(name) or file_values.get(name) + if value: + resolved[name] = value + else: + missing.append(name) + if missing: + raise SystemExit( + "Missing credentials: " + + ", ".join(missing) + + f".\nSet them as environment variables or pass --env-file " + f"(default: {DEFAULT_ENV_FILE})." + ) + BASE = resolved["AGENTA_BASE"] + PROJECT = resolved["AGENTA_PROJECT_ID"] + KEY = resolved["AGENTA_API_KEY"] + + +def agent_template(harness: str, sandbox: str, model: str, provider: str) -> dict: + return { + "instructions": {"agents_md": "Be terse. Do exactly what is asked."}, + "llm": { + "model": model, + "provider": provider, + "connection": {"mode": "agenta", "slug": None}, + "extras": {}, + }, + "tools": [], + "mcps": [], + "skills": [], + "harness": {"kind": harness}, + "sandbox": {"kind": sandbox}, + } + + +def main() -> int: + p = argparse.ArgumentParser() + p.add_argument("--harness", default="pi_core") + p.add_argument("--sandbox", default="local") + p.add_argument("--model", default="gpt-5.6-luna") + p.add_argument("--provider", default="openai") + p.add_argument("--msg", default="Reply with exactly: PONG") + p.add_argument( + "--env-file", + help=f"credentials file (fallback when env vars are unset; default {DEFAULT_ENV_FILE})", + ) + args = p.parse_args() + + resolve_credentials(args.env_file) + + session_id = str(uuid.uuid4()) + url = f"{BASE}/services/agent/v0/invoke" + body = { + "session_id": session_id, + "data": { + "inputs": { + "messages": [ + { + "id": str(uuid.uuid4()), + "role": "user", + "parts": [{"type": "text", "text": args.msg}], + } + ] + }, + "parameters": { + "agent": agent_template( + args.harness, args.sandbox, args.model, args.provider + ) + }, + }, + } + headers = { + "Authorization": f"ApiKey {KEY}", + "Accept": "text/event-stream", + "x-ag-messages-format": "vercel", + "Content-Type": "application/json", + } + + print(f"session={session_id}", file=sys.stderr) + frames: list[str] = [] + text: list[str] = [] + with httpx.Client(timeout=180.0) as client: + with client.stream( + "POST", url, params={"project_id": PROJECT}, json=body, headers=headers + ) as r: + print(f"HTTP {r.status_code}", file=sys.stderr) + if r.status_code >= 400: + print(r.read().decode()[:2000]) + return 1 + for line in r.iter_lines(): + if not line or line.startswith(":"): + continue + if not line.startswith("data: "): + continue + payload = line[6:] + if payload == "[DONE]": + break + try: + frame = json.loads(payload) + except json.JSONDecodeError: + continue + t = frame.get("type", "?") + frames.append(t) + if t == "text-delta": + text.append(frame.get("delta", "")) + if t in ("error", "finish", "tool-approval-request"): + print(f" !! {t}: {json.dumps(frame)[:400]}", file=sys.stderr) + + print("\n--- frame types (in order, deduped consecutive) ---") + dedup = [f for i, f in enumerate(frames) if i == 0 or frames[i - 1] != f] + print(" -> ".join(dedup)) + print("\n--- assistant text ---") + print("".join(text).strip() or "(empty)") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/.agents/skills/agent-release-gate/resources/qa_product.py b/.agents/skills/agent-release-gate/resources/qa_product.py new file mode 100644 index 0000000000..11554bedbe --- /dev/null +++ b/.agents/skills/agent-release-gate/resources/qa_product.py @@ -0,0 +1,985 @@ +# /// script +# requires-python = ">=3.10" +# dependencies = ["httpx>=0.27"] +# /// +"""Product-path QA driver for the agent release gate. + +Drives the SAME endpoint the playground drives (`/services/agent/v0/invoke`), with the same +headers and the same in-band approval protocol the browser uses. Asserts on the wire (SSE frame +types), never on model prose. Where the model must prove something in text, an unguessable +constant is baked into a tool's return value, so a matching reply PROVES the tool ran. + + uv run qa_product.py --cell C3 # one cell + uv run qa_product.py --all # every cell + uv run qa_product.py --cell C3 --only approve # one journey + +Credentials come from the environment (AGENTA_BASE, AGENTA_PROJECT_ID, AGENTA_API_KEY), falling +back to --env-file. Results land in ./qa-gate-runs// (override with AGENTA_QA_RUNS_DIR) +as JSON + a markdown table. +""" + +from __future__ import annotations + +import argparse +import json +import os +import pathlib +import re +import time +import uuid + +import httpx + +HERE = pathlib.Path(__file__).resolve().parent +# Results land in the CURRENT working directory, never inside the skill, so repeated runs do not +# accumulate in the tree. Override with AGENTA_QA_RUNS_DIR (absolute or relative to the CWD). +RUNS = pathlib.Path(os.environ.get("AGENTA_QA_RUNS_DIR", "qa-gate-runs")).resolve() + +# Credentials: read from the environment FIRST, then fall back to an env file. The env vars are +# AGENTA_BASE (deployment origin), AGENTA_PROJECT_ID, and AGENTA_API_KEY — the same three the +# playground needs. This keeps the gate deployment-agnostic: point it at any stack by exporting +# three vars. The file fallback (default below, overridable with --env-file) is only for backward +# compatibility with the original bighetzner QA setup. +REQUIRED_CREDS = ("AGENTA_BASE", "AGENTA_PROJECT_ID", "AGENTA_API_KEY") +DEFAULT_ENV_FILE = pathlib.Path.home() / ".agenta-bighetzner.env" + +# Resolved by resolve_credentials() before any journey runs. Left empty so that --help and other +# no-network entry points work with no credentials present at all. +BASE = "" +PROJECT = "" +KEY = "" + + +def _read_env_file(path: pathlib.Path) -> dict: + values: dict = {} + path = pathlib.Path(path).expanduser() + if not path.exists(): + return values + for line in path.read_text().splitlines(): + line = line.strip() + if line and not line.startswith("#") and "=" in line: + k, v = line.split("=", 1) + values[k.strip()] = v.strip() + return values + + +def resolve_credentials(env_file: str | pathlib.Path | None = None) -> None: + """Populate BASE/PROJECT/KEY. Environment variables win; the env file only fills what the + environment did not set. Raises SystemExit with a clear, specific message naming exactly which + credentials are missing so a first-time runner knows what to set.""" + global BASE, PROJECT, KEY + file_values = _read_env_file(env_file or DEFAULT_ENV_FILE) + resolved: dict = {} + missing: list = [] + for name in REQUIRED_CREDS: + value = os.environ.get(name) or file_values.get(name) + if value: + resolved[name] = value + else: + missing.append(name) + if missing: + raise SystemExit( + "Missing credentials: " + + ", ".join(missing) + + ".\nSet them as environment variables, e.g.\n" + " export AGENTA_BASE=https://your-stack.example.com\n" + " export AGENTA_PROJECT_ID=...\n" + " export AGENTA_API_KEY=...\n" + f"or pass --env-file to a file with those lines " + f"(default: {DEFAULT_ENV_FILE})." + ) + BASE = resolved["AGENTA_BASE"] + PROJECT = resolved["AGENTA_PROJECT_ID"] + KEY = resolved["AGENTA_API_KEY"] + + +# A public, no-auth, HTTPS Streamable-HTTP MCP server used by the `mcp` journey. DeepWiki is a +# well-known free reference server (tools: read_wiki_structure / read_wiki_contents / ask_question). +# Override with --mcp-url to point at any other public server. The runner/SDK both reject non-https +# and private/loopback hosts (SSRF guard), so a LOCAL server is NOT reachable from the deployment — +# it must be a public HTTPS URL. See STATUS.md "MCP smoke test". +DEFAULT_MCP_URL = "https://mcp.deepwiki.com/mcp" +MCP_URL = DEFAULT_MCP_URL + + +def api_call(method: str, path: str, timeout: float = 60.0, **kwargs) -> httpx.Response: + """One REST call to the /api surface (the routes the playground UI drives for config/commits), + NOT the SSE /services/agent/v0/invoke turn endpoint. Auth is the same ApiKey header, and + project_id rides the query string (never the body), exactly like the browser.""" + return httpx.request( + method, + f"{BASE}/api{path}", + params={"project_id": PROJECT}, + headers={"Authorization": f"ApiKey {KEY}", "Content-Type": "application/json"}, + timeout=timeout, + **kwargs, + ) + + +# --------------------------------------------------------------------------- +# Cells: harness x sandbox (core) + provider/auth sub-matrix (Pi only). +# --------------------------------------------------------------------------- +CELLS = { + # Claude: use the `sonnet` alias — a full model id is dropped to the default on the + # Claude ACP path (QA finding F-007). + "C1": { + "harness": "claude", + "sandbox": "local", + "model": "sonnet", + "provider": "anthropic", + # SUBSCRIPTION (OAuth), not the vault key: the project's Anthropic key is out of credit, + # and "Use subscription" is what the playground defaults to anyway. + "connection": {"mode": "self_managed", "slug": None}, + }, + "C2": { + "harness": "claude", + "sandbox": "daytona", + "model": "sonnet", + "provider": "anthropic", + # VAULT KEY (mode "agenta"), NOT subscription: Daytona rejects runtime-provided + # (subscription) auth by design — "Use a managed API key … or run this harness on the + # local sandbox." C2 therefore genuinely needs a funded Anthropic key in the vault. + "connection": {"mode": "agenta", "slug": None}, + }, + "C3": { + "harness": "pi_core", + "sandbox": "local", + "model": "gpt-5.6-luna", + "provider": "openai", + }, + "C4": { + "harness": "pi_core", + "sandbox": "daytona", + "model": "gpt-5.6-luna", + "provider": "openai", + }, + # Provider sub-matrix: an auth question, not a sandbox question — Pi + local only. + "P1": { + "harness": "pi_core", + "sandbox": "local", + "model": "openrouter/deepseek/deepseek-v4-flash", + "provider": "openrouter", + }, + # S1: the Codex SUBSCRIPTION path — Pi with provider `openai-codex` (a first-class + # subscription provider slug, distinct from the vault-key `openai` provider; see + # sdks/python/agenta/sdk/agents/capabilities.py PI_SUBSCRIPTION_PROVIDERS). Auth comes from + # the subscription sidecar's ChatGPT/Codex OAuth login (~/.pi/agent/auth.json), never a + # vault key, so `self_managed` + slug None is the whole connection. + "S1": { + "harness": "pi_core", + "sandbox": "local", + "model": "gpt-5.6-luna", + "provider": "openai-codex", + "connection": {"mode": "self_managed", "slug": None}, + }, + # P2 (OpenRouter as a CUSTOM OpenAI-compatible provider) needs a `custom_provider` secret in + # the vault; `connection.slug` points at it. Set --custom-slug to run it. + "P2": { + "harness": "pi_core", + "sandbox": "local", + "model": "deepseek/deepseek-v4-flash", + "provider": "custom", + "connection": { + "mode": "self_managed", + "slug": None, + }, # slug filled from --custom-slug + }, +} + +# NOTE: `code` tools are NOT usable on the product path — the sidecar rejects them +# ("Code tools are not supported by the sidecar.", services/runner/src/tools/code.ts). They only +# work against the in-process service, which is what the OLD qa driver (run_matrix.py) targets. +# The product's real tool surface is `builtin` (bash/read/write/...), `gateway` (Composio) and +# `mcp`. So we prove tool execution with builtin bash echoing an unguessable token. +BASH_TOOL = {"type": "builtin", "name": "bash"} + +# The token MUST NOT be derivable from the prompt. An early version of this used +# `echo "QA-BASH-$((6*7+1))"` — and the model simply computed 43 and reported it WITHOUT running +# bash, so a DENIED tool call still produced a "passing" reply. The wire said denied; the prose +# lied. Lesson, and the rule for this whole driver: assert on FRAMES, and only ever use a token +# the model cannot compute. The container hostname is random per container. +BASH_PROMPT = ( + 'Use the bash tool to run exactly: echo "QA-BASH-$(hostname)-$(uname -m)" ' + "and reply with only its stdout." +) +BASH_TOKEN_RE = re.compile(r"QA-BASH-[0-9a-f]{6,}-\w+") + +# For the APPROVAL journeys the command must MUTATE. Claude Code classifies bash commands and +# auto-approves read-only ones (a bare `echo`) no matter what the permission policy says, so +# approving a read-only echo tests nothing on Claude — and a user approving an action is, by +# definition, approving a mutating one. Pi gates all bash via the extension, so a mutating +# command works as the single approval probe on BOTH harnesses. +MUTATE_PROMPT = ( + "Use the bash tool to run exactly: " + "echo WROTE > /tmp/qa-$(hostname).txt && cat /tmp/qa-$(hostname).txt " + "and reply with only its stdout." +) + + +def tool_ran(t: "Turn") -> bool: + """Did ANY tool execute? The wire, never the reply.""" + return "tool-output-available" in t.frames + + +def outcome_for_input(t: "Turn", wanted_input: dict) -> str | None: + """The outcome of the call carrying THIS input ("available"|"error"|"denied"), or None. + + Keyed by the command, NOT the toolCallId or the tool name, because on resume the harness + RE-ISSUES the gated call under a brand-new toolCallId (and Claude names it `Terminal` while + Pi names it `Bash`). Keying on either would look at the wrong call. And a turn routinely holds + several calls — an auto-approved read-only one beside the gated one — so a turn-wide check + gives false failures. + """ + for call in t.tool_calls: + if call.get("input") == wanted_input: + out = t.tool_outcomes.get(call["toolCallId"]) + if out: + return out + return None + + +def template( + cell: dict, + tools: list | None = None, + instructions: str | None = None, + permission_default: str | None = None, + mcps: list | None = None, +) -> dict: + conn = cell.get("connection") or {"mode": "agenta", "slug": None} + t = { + "instructions": { + "agents_md": instructions + or "Be terse. Do exactly what is asked, nothing more." + }, + "llm": { + "model": cell["model"], + "provider": cell["provider"], + "connection": conn, + "extras": {}, + }, + "tools": tools or [], + "mcps": mcps or [], + "skills": [], + "harness": {"kind": cell["harness"]}, + "sandbox": {"kind": cell["sandbox"]}, + } + if permission_default: + # Layer-2: the runner's permission posture. `ask` is what makes a tool call raise the + # approval dock in the product — this is the real approval mechanism a user hits. + t["runner"] = { + "kind": "sidecar", + "permissions": {"default": permission_default}, + } + return t + + +def user_msg(text: str) -> dict: + return { + "id": str(uuid.uuid4()), + "role": "user", + "parts": [{"type": "text", "text": text}], + } + + +class Turn: + """One /invoke round trip, parsed off the wire.""" + + def __init__(self) -> None: + self.frames: list[str] = [] + self.text: list[str] = [] + self.approval: dict | None = None # {approvalId, toolCallId} + self.tool_calls: list[dict] = [] # {toolCallId, toolName, input} + # Outcome per toolCallId: "available" | "error" | "denied". A turn can contain SEVERAL + # tool calls (an agent often runs an auto-approved read-only call alongside the gated + # one), so "did the tool run?" MUST be asked of the specific gated call, never of the + # turn as a whole. + self.tool_outcomes: dict[str, str] = {} + # The PAYLOAD behind each outcome (output value or errorText), keyed by toolCallId. + # Needed to replay a byte-faithful assistant UIMessage: the AI SDK ships the tool's + # output back to the server on every subsequent turn's history, and the runner's + # history fingerprint (session-pool.ts historyFingerprint) hashes tool-call ids out of + # that history. A text-only replay drops them -> mismatch -> warm session evicted. + self.tool_payloads: dict[str, dict] = {} + # Parts in the ORDER the model actually produced them (mirrors AI SDK + # `UIMessage.parts` arrival order): a list of {"kind": "text", "text": str} | + # {"kind": "tool", "id": toolCallId}, consumed by assistant_message(). + self._segments: list[dict] = [] + self.finish_reason: str | None = None + self.errors: list[str] = [] + self.committed_revision: dict | None = None + self.http_status: int = 0 + self.ms: int = 0 + + @property + def reply(self) -> str: + return "".join(self.text).strip() + + def assistant_message(self) -> dict: + """Rebuild this turn's reply as a FULL Vercel UIMessage — text AND tool parts, in the + order the model produced them — so replaying it as history is byte-faithful to what the + real frontend (AI SDK `useChat`) sends back on the next turn (`agentRequest.ts:401`). + + A text-only replay drops the assistant's tool parts, and the runner's history + fingerprint (`session-pool.ts` `historyFingerprint`) hashes the ordered, deduped + tool-call ids out of that history. Missing ids -> `mismatch (history)` on the next + turn -> the warm session is EVICTED and every following turn runs cold. See + `sdks/python/agenta/sdk/agents/adapters/vercel/messages.py` `_tool_part_blocks` for + the exact states the server accepts on ingest — this mirrors them precisely: + "output-available" + output, "output-error" + errorText, "output-denied" (no payload, + read by `_approval_decision`'s state fallback as an inline deny). + """ + by_id = {c["toolCallId"]: c for c in self.tool_calls} + parts: list[dict] = [] + for seg in self._segments: + if seg["kind"] == "text": + if seg["text"]: + parts.append({"type": "text", "text": seg["text"]}) + continue + call = by_id.get(seg["id"], {}) + part = { + "type": f"tool-{call.get('toolName') or 'tool'}", + "toolCallId": seg["id"], + "input": call.get("input") or {}, + } + outcome = self.tool_outcomes.get(seg["id"]) + payload = self.tool_payloads.get(seg["id"], {}) + if outcome == "available": + part["state"] = "output-available" + part["output"] = payload.get("output") + elif outcome == "error": + part["state"] = "output-error" + part["errorText"] = payload.get("errorText") + elif outcome == "denied": + part["state"] = "output-denied" + else: + # No outcome landed within this turn (e.g. a call still awaiting an approval + # decision) — mirror the AI SDK's in-flight tool-part state so the id still + # rides the history, without fabricating a result it never produced. + part["state"] = "input-available" + parts.append(part) + if not parts: + parts.append({"type": "text", "text": self.reply}) + return {"id": str(uuid.uuid4()), "role": "assistant", "parts": parts} + + def summary(self) -> dict: + return { + "http": self.http_status, + "ms": self.ms, + "finish": self.finish_reason, + "frames": self.frames, + "tools": [t.get("toolName") for t in self.tool_calls], + "approval": bool(self.approval), + "errors": self.errors, + "reply": self.reply[:400], + } + + +def invoke( + session_id: str, messages: list, params: dict, timeout: float = 300.0 +) -> Turn: + t = Turn() + body = { + "session_id": session_id, + "data": {"inputs": {"messages": messages}, "parameters": {"agent": params}}, + } + headers = { + "Authorization": f"ApiKey {KEY}", + "Accept": "text/event-stream", + "x-ag-messages-format": "vercel", + "Content-Type": "application/json", + } + start = time.time() + with httpx.Client(timeout=timeout) as client: + with client.stream( + "POST", + f"{BASE}/services/agent/v0/invoke", + params={"project_id": PROJECT}, + json=body, + headers=headers, + ) as r: + t.http_status = r.status_code + if r.status_code >= 400: + t.errors.append(f"HTTP {r.status_code}: {r.read().decode()[:500]}") + t.ms = int((time.time() - start) * 1000) + return t + for line in r.iter_lines(): + if not line or line.startswith(":") or not line.startswith("data: "): + continue + payload = line[6:] + if payload == "[DONE]": + break + try: + f = json.loads(payload) + except json.JSONDecodeError: + continue + ftype = f.get("type", "?") + t.frames.append(ftype) + if ftype == "text-delta": + delta = f.get("delta", "") + t.text.append(delta) + # Coalesce consecutive text-delta frames into ONE running text segment; + # a tool call between two text runs starts a NEW segment (see below), so + # this reproduces the AI SDK's interleaved part order. + if t._segments and t._segments[-1]["kind"] == "text": + t._segments[-1]["text"] += delta + else: + t._segments.append({"kind": "text", "text": delta}) + elif ftype == "tool-input-available": + # CAREFUL: this frame is emitted REPEATEDLY for one tool call, carrying a + # progressively-built PARTIAL input, and `toolName` changes case along the way + # ("bash" while streaming -> "Bash" when complete). Only the LAST frame per + # toolCallId holds the real command. Keeping the first one approves a + # truncated command under the wrong name, the runner's decision key + # (name+args) misses the parked gate, and the approval re-parks forever. + call = { + "toolCallId": f.get("toolCallId"), + "toolName": f.get("toolName"), + "input": f.get("input"), + } + is_new_call = not any( + c["toolCallId"] == call["toolCallId"] for c in t.tool_calls + ) + t.tool_calls = [ + c for c in t.tool_calls if c["toolCallId"] != call["toolCallId"] + ] + [call] + # Segment position is fixed at FIRST appearance (when the call starts), + # never moved by later partial-input updates — that's when the AI SDK + # would have inserted the tool part into UIMessage.parts. + if is_new_call: + t._segments.append({"kind": "tool", "id": call["toolCallId"]}) + elif ftype == "tool-approval-request": + t.approval = { + "approvalId": f.get("approvalId"), + "toolCallId": f.get("toolCallId"), + } + elif ftype in ( + "tool-output-available", + "tool-output-error", + "tool-output-denied", + ): + tcid = f.get("toolCallId") + if tcid: + t.tool_outcomes[tcid] = ftype.replace("tool-output-", "") + if ftype == "tool-output-available": + t.tool_payloads[tcid] = {"output": f.get("output")} + elif ftype == "tool-output-error": + t.tool_payloads[tcid] = {"errorText": f.get("errorText")} + elif ftype == "data-committed-revision": + t.committed_revision = f.get("data") + elif ftype == "error": + t.errors.append(json.dumps(f)[:300]) + elif ftype == "finish": + t.finish_reason = f.get("finishReason") + t.ms = int((time.time() - start) * 1000) + return t + + +def approval_reply(turn: Turn, approved: bool) -> dict: + """Rebuild the paused assistant message with the decision inline — exactly what the browser + does (addToolApprovalResponse -> re-POST the history). NOT the out-of-band REST route.""" + call = next( + (c for c in turn.tool_calls if c["toolCallId"] == turn.approval["toolCallId"]), + turn.tool_calls[-1] if turn.tool_calls else {}, + ) + return { + "id": str(uuid.uuid4()), + "role": "assistant", + "parts": [ + { + "type": f"tool-{call.get('toolName')}", + "toolCallId": turn.approval["toolCallId"], + "state": "approval-responded", + "input": call.get("input") or {}, + "approval": {"id": turn.approval["approvalId"], "approved": approved}, + } + ], + } + + +# --------------------------------------------------------------------------- +# Journeys +# --------------------------------------------------------------------------- + + +def j1_chat(cell: dict) -> dict: + """J1: the agent answers at all.""" + s = str(uuid.uuid4()) + t = invoke(s, [user_msg("Reply with exactly: PONG")], template(cell)) + ok = t.finish_reason == "stop" and "PONG" in t.reply.upper() and not t.errors + return { + "pass": ok, + "why": "finish=stop and reply contains PONG", + "turn": t.summary(), + } + + +def j3_tool(cell: dict) -> dict: + """J3: a tool really executed — proven by a token only a real shell can produce.""" + s = str(uuid.uuid4()) + t = invoke( + s, + [user_msg(BASH_PROMPT)], + template( + cell, + tools=[BASH_TOOL], + instructions="Use the bash tool when asked to run a command. Report only its stdout.", + permission_default="allow", + ), + ) + ok = tool_ran(t) and bool(BASH_TOKEN_RE.search(t.reply)) + return { + "pass": ok, + "why": "wire shows tool-output-available AND the reply carries a token only a real shell could emit", + "turn": t.summary(), + } + + +def _approval_flow(cell: dict, approved: bool) -> dict: + """J4: with permission default `ask`, a tool call must PAUSE with a tool-approval-request, + then resume on the user's decision — the same in-band protocol the browser uses.""" + s = str(uuid.uuid4()) + params = template( + cell, + tools=[BASH_TOOL], + instructions="Use the bash tool when asked to run a command. Report only its stdout.", + permission_default="ask", + ) + msgs = [user_msg(MUTATE_PROMPT)] + t1 = invoke(s, msgs, params) + + if not t1.approval: + return { + "pass": False, + "why": "expected a tool-approval-request frame; the gate never fired", + "turn": t1.summary(), + } + # A paused turn finishes with reason "other", not "stop". + paused_ok = t1.finish_reason == "other" + + gated_call = next( + (c for c in t1.tool_calls if c["toolCallId"] == t1.approval["toolCallId"]), + t1.tool_calls[-1] if t1.tool_calls else {}, + ) + gated_input = gated_call.get("input") or {} + msgs = msgs + [approval_reply(t1, approved)] + t2 = invoke(s, msgs, params) + outcome = outcome_for_input(t2, gated_input) + + if approved: + ok = outcome == "available" and not t2.errors + why = f"approved: the gated command executed after approval (outcome={outcome}, paused finish=other: {paused_ok})" + else: + # Denied: the gated COMMAND must never have executed. Assert the WIRE, never the reply — + # a denied model will happily hallucinate the output it never received. + ok = outcome != "available" + why = f"denied: the gated command never executed (outcome={outcome})" + return { + "pass": ok, + "why": why, + "paused_finish_other": paused_ok, + "turn_paused": t1.summary(), + "turn_resumed": t2.summary(), + } + + +def j4_approve(cell: dict) -> dict: + return _approval_flow(cell, approved=True) + + +def j4_deny(cell: dict) -> dict: + return _approval_flow(cell, approved=False) + + +def j6_warm(cell: dict) -> dict: + """J6 (latency half): three turns in one session; turns 2/3 should be faster than turn 1. + The cold/warm TRUTH lives in the runner log — this only measures. See STATUS.md F-2.""" + s = str(uuid.uuid4()) + params = template(cell) + msgs: list = [] + times = [] + for i, q in enumerate( + [ + "Reply with exactly: ONE", + "Reply with exactly: TWO", + "Reply with exactly: THREE", + ] + ): + msgs = msgs + [user_msg(q)] + t = invoke(s, msgs, params) + times.append(t.ms) + msgs = msgs + [t.assistant_message()] + if t.errors: + return {"pass": False, "why": f"turn {i + 1} errored", "turn": t.summary()} + warm_gain = times[0] - min(times[1], times[2]) + return { + "pass": warm_gain > 0, + "why": f"turn1={times[0]}ms, turn2={times[1]}ms, turn3={times[2]}ms (warm gain {warm_gain}ms)", + "session_id": s, + "times_ms": times, + } + + +def j2_mount(cell: dict) -> dict: + """J2: the agent's working directory PERSISTS across turns. + + Turn 1 writes a token to a file. Turn 2 — a separate /invoke on the same session — reads it + back. This is the journey that silently failed while mounts were 503ing: the agent ran in a + throwaway /tmp cwd, every turn looked fine, and the file was gone. So the pass condition is + the token coming back FROM DISK in turn 2, with a real tool call behind it. + """ + s = str(uuid.uuid4()) + token = f"QA-MOUNT-{uuid.uuid4().hex[:10]}" + params = template( + cell, + tools=[BASH_TOOL], + instructions="Use the bash tool when asked. Report only the command's stdout.", + permission_default="allow", + ) + msgs = [ + user_msg( + f"Use bash to run exactly: echo {token} > qa-mount.txt ; then reply with only: WROTE" + ) + ] + t1 = invoke(s, msgs, params) + if not tool_ran(t1): + return { + "pass": False, + "why": "turn 1 never executed the write", + "turn_write": t1.summary(), + } + + msgs = msgs + [ + t1.assistant_message(), + user_msg( + "Use bash to run exactly: cat qa-mount.txt and reply with only its stdout." + ), + ] + t2 = invoke(s, msgs, params) + ok = token in t2.reply and tool_ran(t2) + return { + "pass": ok, + "why": f"turn 2 read the token back from the mounted cwd (token={token})", + "turn_write": t1.summary(), + "turn_read": t2.summary(), + } + + +def j5_commit(cell: dict) -> dict: + """J5: committing an agent config as a new workflow revision — the playground's Save/Commit. + + This is a WORKFLOW-revision commit (a new version of the agent's configuration), NOT a git + commit and NOT the in-stream `data-committed-revision` frame. It drives the exact REST route + the UI's commit button hits: `POST /api/workflows/revisions/commit` + (web/packages/agenta-entities/src/workflow/api/api.ts commitWorkflowRevisionApi). + + Wire truth, not prose: after committing a changed parameter we FETCH the revision back + (`GET /api/workflows/revisions/{id}`) and assert the stored config carries the change AND the + version incremented. + + Two facts that bite (both verified in the API): + - The FIRST commit on a fresh variant is the v0 SEED: the DAO force-nulls its data/flags/meta + (`dbs/postgres/git/dao.py` `_null_revision_fields`, `if revision.version == "0"`). So a + config only persists on the SECOND commit (v1). The UI does the same seed-then-commit dance. + - `data` is `extra="forbid"` — only {uri,url,headers,runtime,script,schemas,parameters} are + accepted; the agent config goes under `data.parameters`. + + QA artifacts are namespaced `qa-commit-` and the whole workflow is archived at the end so + repeated runs don't pile up. + """ + hexid = uuid.uuid4().hex[:8] + token = f"QA-COMMIT-{uuid.uuid4().hex[:12]}" # unguessable; also gitleaks-allowlisted shape + workflow_id = None + try: + r = api_call( + "POST", + "/workflows/", + json={ + "workflow": { + "slug": f"qa-commit-{hexid}", + "name": f"QA commit {hexid}", + "flags": { + "is_custom": True, + "is_evaluator": False, + "is_feedback": False, + }, + } + }, + ) + if r.status_code != 200: + return { + "pass": False, + "why": f"create workflow HTTP {r.status_code}: {r.text[:200]}", + } + workflow_id = r.json()["workflow"]["id"] + + r = api_call( + "POST", + "/workflows/variants/", + json={ + "workflow_variant": { + "slug": f"qa-commit-{hexid}-v", + "name": f"QA commit {hexid} v", + "workflow_id": workflow_id, + } + }, + ) + if r.status_code != 200: + return { + "pass": False, + "why": f"create variant HTTP {r.status_code}: {r.text[:200]}", + } + variant_id = r.json()["workflow_variant"]["id"] + + # The committed config IS an agent config — the same shape a playground agent commits. + base_params = { + "agent": { + "instructions": {"agents_md": "seed"}, + "llm": {"model": cell["model"], "provider": cell["provider"]}, + "tools": [], + "harness": {"kind": cell["harness"]}, + "sandbox": {"kind": cell["sandbox"]}, + } + } + + def commit(parameters: dict, message: str, slug: str) -> httpx.Response: + return api_call( + "POST", + "/workflows/revisions/commit", + json={ + "workflow_revision": { + "slug": slug, + "name": f"QA commit {hexid} rev", + "message": message, + "data": { + "uri": "agenta:builtin:chat:v0", + "parameters": parameters, + }, + "workflow_id": workflow_id, + "workflow_variant_id": variant_id, + } + }, + ) + + # v0 seed (data is intentionally nulled by the API for version 0). + r = commit(base_params, "seed", f"qa-commit-seed-{hexid}") + if r.status_code != 200: + return { + "pass": False, + "why": f"seed commit HTTP {r.status_code}: {r.text[:200]}", + } + seed_version = r.json()["workflow_revision"].get("version") + + # v1: the real commit — modify one config parameter (the instructions token). + changed = json.loads(json.dumps(base_params)) + changed["agent"]["instructions"]["agents_md"] = token + r = commit( + changed, "QA commit journey: change agents_md", f"qa-commit-real-{hexid}" + ) + if r.status_code != 200: + return { + "pass": False, + "why": f"real commit HTTP {r.status_code}: {r.text[:200]}", + } + committed = r.json()["workflow_revision"] + revision_id = committed["id"] + new_version = committed.get("version") + + # Fetch the revision back and compare on the wire (never trust the commit echo alone). + r = api_call("GET", f"/workflows/revisions/{revision_id}") + if r.status_code != 200: + return { + "pass": False, + "why": f"fetch revision HTTP {r.status_code}: {r.text[:200]}", + } + fetched = r.json()["workflow_revision"] + fetched_token = ( + (fetched.get("data") or {}) + .get("parameters", {}) + .get("agent", {}) + .get("instructions", {}) + .get("agents_md") + ) + version_bumped = ( + seed_version == "0" and new_version == "1" and fetched.get("version") == "1" + ) + ok = fetched_token == token and version_bumped + return { + "pass": ok, + "why": ( + f"committed a new revision and read it back: token match={fetched_token == token}, " + f"version {seed_version}->{new_version} (bumped={version_bumped})" + ), + "workflow_id": workflow_id, + "revision_id": revision_id, + "token": token, + } + finally: + # Clean up so repeated runs don't accumulate QA workflows. + if workflow_id: + try: + api_call("POST", f"/workflows/{workflow_id}/archive") + except Exception: + pass + + +# The wire name of an MCP-delivered tool is `mcp____` (verified on DeepWiki: +# `mcp__deepwiki__read_wiki_structure`). We give the agent NO builtin tools, so any tool call it +# makes is necessarily the MCP tool — and we still key the assertion on the `mcp__` prefix. +MCP_TOOL_RE = re.compile(r"^mcp__") + + +def j7_mcp(cell: dict) -> dict: + """J7: an MCP server declared in the agent config is delivered to the harness, and one of its + tools actually executes — proven by a `tool-output-available` frame for an `mcp__*` tool. + + Two hard constraints (both verified in the runner): + - **Claude only.** Pi refuses any run that declares `mcps` + (`run-plan.ts` PI_USER_MCP_UNSUPPORTED_MESSAGE); user MCP needs a harness with mcpTools + (Claude). So this journey SKIPS on non-Claude cells. + - **Public HTTPS only.** The SDK resolver and the runner both run an SSRF guard that rejects + http:// and private/loopback/metadata hosts, so a local MCP server is unreachable from the + deployment. --mcp-url must be a public HTTPS Streamable-HTTP endpoint (default: DeepWiki). + + The harness dials the URL directly (on `local`, from the runner host), so the endpoint must be + reachable from the deployment's network. + """ + if cell["harness"] != "claude": + return { + "skip": True, + "why": f"MCP requires a Claude harness; Pi rejects any run with mcps (cell harness={cell['harness']}). Run with --cell C1.", + } + + s = str(uuid.uuid4()) + mcp = { + "name": "deepwiki", + "connection": {"type": "http", "url": MCP_URL}, + "policy": {"tools": {"mode": "all"}}, + } + prompt = ( + "Use the deepwiki MCP tool named read_wiki_structure with repoName 'facebook/react' to " + "list the wiki topics, then reply with only: DONE." + ) + t = invoke( + s, + [user_msg(prompt)], + template( + cell, + tools=[], + instructions="Use the available MCP tools when asked. Be terse.", + permission_default="allow", + mcps=[mcp], + ), + ) + + mcp_calls = [c for c in t.tool_calls if MCP_TOOL_RE.match(c.get("toolName") or "")] + mcp_ran = any( + t.tool_outcomes.get(c["toolCallId"]) == "available" for c in mcp_calls + ) + if not mcp_calls and not mcp_ran: + why = ( + f"no mcp__* tool call was made against {MCP_URL} — the harness may not have reached " + "the server, or the server exposed no tools. Check the runner log for MCP errors." + ) + else: + why = f"an mcp__* tool executed against {MCP_URL} (wire shows tool-output-available)" + return { + "pass": mcp_ran, + "why": why, + "mcp_url": MCP_URL, + "mcp_tools_called": [c.get("toolName") for c in mcp_calls], + "turn": t.summary(), + } + + +JOURNEYS = { + "chat": j1_chat, + "mount": j2_mount, + "tool": j3_tool, + "approve": j4_approve, + "deny": j4_deny, + "commit": j5_commit, + "warm": j6_warm, + "mcp": j7_mcp, +} + + +def main() -> int: + p = argparse.ArgumentParser() + p.add_argument("--cell", action="append", help="C1..C4, P1, P2") + p.add_argument("--all", action="store_true") + p.add_argument("--only", action="append", help=f"one of {list(JOURNEYS)}") + p.add_argument( + "--custom-slug", help="vault slug of the custom OpenAI-compatible provider (P2)" + ) + p.add_argument( + "--mcp-url", + help=f"public HTTPS MCP server URL for the `mcp` journey (default: {DEFAULT_MCP_URL})", + ) + p.add_argument( + "--model", + help="override the cell's model (e.g. `haiku` on a Claude cell; aliases only on Claude — F-007)", + ) + p.add_argument( + "--env-file", + help=f"credentials file (fallback when the env vars are unset; default {DEFAULT_ENV_FILE})", + ) + args = p.parse_args() + + resolve_credentials(args.env_file) + + cells = list(CELLS) if args.all else (args.cell or ["C3"]) + journeys = args.only or list(JOURNEYS) + if args.custom_slug: + CELLS["P2"]["connection"]["slug"] = args.custom_slug + if args.mcp_url: + global MCP_URL + MCP_URL = args.mcp_url + if args.model: + for cid in cells: + CELLS[cid]["model"] = args.model + + stamp = time.strftime("%Y%m%d-%H%M%S") + outdir = RUNS / stamp + outdir.mkdir(parents=True, exist_ok=True) + + results: dict = {} + for cid in cells: + cell = CELLS[cid] + results[cid] = {"config": {k: v for k, v in cell.items()}, "journeys": {}} + for jname in journeys: + print(f"[{cid}] {jname} ... ", end="", flush=True) + try: + r = JOURNEYS[jname](cell) + except Exception as e: # a crash is a result, not a reason to lose the run + r = {"pass": False, "why": f"driver exception: {type(e).__name__}: {e}"} + results[cid]["journeys"][jname] = r + verdict = "SKIP" if r.get("skip") else ("PASS" if r.get("pass") else "FAIL") + print(verdict, f"— {r.get('why', '')[:90]}") + (outdir / "results.json").write_text(json.dumps(results, indent=2)) + + lines = ["| cell | harness | sandbox | model | " + " | ".join(journeys) + " |"] + lines.append("|" + "---|" * (4 + len(journeys))) + for cid, r in results.items(): + c = r["config"] + cellstr = [ + ( + "SKIP" + if r["journeys"][j].get("skip") + else ("PASS" if r["journeys"][j].get("pass") else "FAIL") + ) + for j in journeys + ] + lines.append( + f"| {cid} | {c['harness']} | {c['sandbox']} | {c['model']} | " + + " | ".join(cellstr) + + " |" + ) + table = "\n".join(lines) + (outdir / "summary.md").write_text(table + "\n") + print("\n" + table) + print(f"\nresults: {outdir}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/.agents/skills/agent-release-gate/resources/seeds/README.md b/.agents/skills/agent-release-gate/resources/seeds/README.md new file mode 100644 index 0000000000..55cc960d3d --- /dev/null +++ b/.agents/skills/agent-release-gate/resources/seeds/README.md @@ -0,0 +1,11 @@ +# Seeds + +Representative green `results.json` from real gate runs, kept as regression-seed references — the +shape a healthy run produces, and a baseline to diff a future run against. + +- `product-C2-full-green.results.json` — `qa_product.py` on cell C2 (Claude/daytona, funded vault + key), all six journeys PASS. +- `longctx-daytona-all-green.results.json` — `qa_longctx.py` on daytona, all three probes (gmail, + memory, concurrent) PASS. + +These are captured evidence, not fixtures a test loads. Live runs write to `./qa-gate-runs/`. diff --git a/.agents/skills/agent-release-gate/resources/seeds/longctx-daytona-all-green.results.json b/.agents/skills/agent-release-gate/resources/seeds/longctx-daytona-all-green.results.json new file mode 100644 index 0000000000..5bbe284f59 --- /dev/null +++ b/.agents/skills/agent-release-gate/resources/seeds/longctx-daytona-all-green.results.json @@ -0,0 +1,1562 @@ +{ + "gmail": { + "pass": true, + "why": "a Gmail gateway tool executed (tool-output-available) with no error", + "tools_called": [ + "gmail__FETCH_EMAILS", + "github__LIST_REPOSITORIES_FOR_THE_AUTHENTICATED_USER", + "github__LIST_REPOSITORIES_FOR_THE_AUTHENTICATED_USER" + ], + "turn": { + "http": 200, + "ms": 34320, + "finish": "stop", + "frames": [ + "start", + "start-step", + "reasoning-start", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-end", + "tool-input-start", + "tool-input-available", + "tool-input-available", + "tool-input-start", + "tool-input-available", + "tool-input-available", + "tool-output-available", + "tool-output-available", + "reasoning-start", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-end", + "tool-input-start", + "tool-input-available", + "tool-input-available", + "tool-input-available", + "tool-input-available", + "tool-input-available", + "tool-input-available", + "tool-input-available", + "tool-input-available", + "tool-input-available", + "tool-input-available", + "tool-output-available", + "text-start", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-delta", + "text-end", + "finish-step", + "finish" + ], + "tools": [ + "gmail__FETCH_EMAILS", + "github__LIST_REPOSITORIES_FOR_THE_AUTHENTICATED_USER", + "github__LIST_REPOSITORIES_FOR_THE_AUTHENTICATED_USER" + ], + "approval": false, + "errors": [], + "reply": "## 3 most recent inbox emails\n\n1. **Re: [Agenta-AI/agenta] [integration] big-agents (PR #4791)**\n2. **There\u2019s been a login from a new device**\n3. **The Next AI Goldrush: Tokens, Loops, and Neofirms**\n\n## GitHub repositories\n\n**Agenta-AI:**\n- `.github`\n- `agenta`\n- `agenta-blog`\n- `agenta-challenge-afcjunior`\n- `agenta-challenges`\n- `agenta-commands`\n- `agenta-commons`\n- `agenta-core`\n- `agenta-dem" + } + }, + "memory": { + "pass": true, + "why": "the token planted in turn 1 survived 12 tool-heavy turns (token=QA-MEM-C3E52C87BBC4)", + "token": "QA-MEM-C3E52C87BBC4", + "recalled": "QA-MEM-C3E52C87BBC4", + "flood": [ + { + "turn": 2, + "ms": 13770, + "tools": 1, + "err": [] + }, + { + "turn": 3, + "ms": 18459, + "tools": 1, + "err": [] + }, + { + "turn": 4, + "ms": 48439, + "tools": 6, + "err": [] + }, + { + "turn": 5, + "ms": 15274, + "tools": 1, + "err": [] + }, + { + "turn": 6, + "ms": 29674, + "tools": 1, + "err": [] + }, + { + "turn": 7, + "ms": 14914, + "tools": 1, + "err": [] + }, + { + "turn": 8, + "ms": 50291, + "tools": 6, + "err": [] + }, + { + "turn": 9, + "ms": 17089, + "tools": 1, + "err": [] + }, + { + "turn": 10, + "ms": 27201, + "tools": 1, + "err": [] + }, + { + "turn": 11, + "ms": 18205, + "tools": 1, + "err": [] + }, + { + "turn": 12, + "ms": 33918, + "tools": 4, + "err": [] + }, + { + "turn": 13, + "ms": 19040, + "tools": 1, + "err": [] + } + ], + "session_id": "5044cc8f-80a6-476a-b979-d7fd2e6fd6e3" + }, + "concurrent": { + "pass": true, + "why": "each of 3 concurrent sessions recalled ITS OWN token and none leaked another's", + "leaks": [], + "results": [ + { + "i": 0, + "expected": "QA-CONC0-EED5E5D1", + "reply": "QA-CONC0-EED5E5D1", + "session": "17ba423b-3030-4433-8ad5-cb180ab99fce", + "errors": [] + }, + { + "i": 1, + "expected": "QA-CONC1-68F047A9", + "reply": "QA-CONC1-68F047A9", + "session": "9def2528-6585-4aa4-80cf-09f0675eff7d", + "errors": [] + }, + { + "i": 2, + "expected": "QA-CONC2-4DB4BA74", + "reply": "QA-CONC2-4DB4BA74", + "session": "bc13611b-c6c2-4cae-83ed-f4ec1416f6f5", + "errors": [] + } + ] + } +} \ No newline at end of file diff --git a/.agents/skills/agent-release-gate/resources/seeds/product-C2-full-green.results.json b/.agents/skills/agent-release-gate/resources/seeds/product-C2-full-green.results.json new file mode 100644 index 0000000000..eb86d70299 --- /dev/null +++ b/.agents/skills/agent-release-gate/resources/seeds/product-C2-full-green.results.json @@ -0,0 +1,287 @@ +{ + "C2": { + "config": { + "harness": "claude", + "sandbox": "daytona", + "model": "haiku", + "provider": "anthropic", + "connection": { + "mode": "agenta", + "slug": null + } + }, + "journeys": { + "chat": { + "pass": true, + "why": "finish=stop and reply contains PONG", + "turn": { + "http": 200, + "ms": 8178, + "finish": "stop", + "frames": [ + "start", + "start-step", + "reasoning-start", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-end", + "text-start", + "text-delta", + "text-end", + "finish-step", + "finish" + ], + "tools": [], + "approval": false, + "errors": [], + "reply": "PONG" + } + }, + "mount": { + "pass": true, + "why": "turn 2 read the token back from the mounted cwd (token=QA-MOUNT-7328b41f02)", + "turn_write": { + "http": 200, + "ms": 11115, + "finish": "stop", + "frames": [ + "start", + "start-step", + "reasoning-start", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-end", + "tool-input-start", + "tool-input-available", + "tool-input-available", + "tool-output-available", + "reasoning-start", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-end", + "text-start", + "text-delta", + "text-end", + "finish-step", + "finish" + ], + "tools": [ + "Terminal" + ], + "approval": false, + "errors": [], + "reply": "WROTE" + }, + "turn_read": { + "http": 200, + "ms": 5069, + "finish": "stop", + "frames": [ + "start", + "start-step", + "reasoning-start", + "reasoning-delta", + "reasoning-delta", + "reasoning-end", + "tool-input-start", + "tool-input-available", + "tool-input-available", + "tool-output-available", + "reasoning-start", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-end", + "text-start", + "text-delta", + "text-end", + "finish-step", + "finish" + ], + "tools": [ + "Terminal" + ], + "approval": false, + "errors": [], + "reply": "QA-MOUNT-7328b41f02" + } + }, + "tool": { + "pass": true, + "why": "wire shows tool-output-available AND the reply carries a token only a real shell could emit", + "turn": { + "http": 200, + "ms": 10436, + "finish": "stop", + "frames": [ + "start", + "start-step", + "reasoning-start", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-end", + "tool-input-start", + "tool-input-available", + "tool-input-available", + "tool-output-available", + "reasoning-start", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-end", + "text-start", + "text-delta", + "text-end", + "finish-step", + "finish" + ], + "tools": [ + "Terminal" + ], + "approval": false, + "errors": [], + "reply": "QA-BASH-50895052-fe88-451f-92f5-8c123028d780-x86_64" + } + }, + "approve": { + "pass": true, + "why": "approved: the gated command executed after approval (outcome=available, paused finish=other: True)", + "paused_finish_other": true, + "turn_paused": { + "http": 200, + "ms": 8850, + "finish": "other", + "frames": [ + "start", + "start-step", + "reasoning-start", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-end", + "tool-input-start", + "tool-input-available", + "tool-input-available", + "tool-input-available", + "tool-approval-request", + "finish-step", + "finish" + ], + "tools": [ + "Terminal" + ], + "approval": true, + "errors": [], + "reply": "" + }, + "turn_resumed": { + "http": 200, + "ms": 4958, + "finish": "stop", + "frames": [ + "start", + "start-step", + "tool-input-start", + "tool-input-available", + "tool-output-available", + "reasoning-start", + "reasoning-delta", + "reasoning-delta", + "reasoning-end", + "text-start", + "text-delta", + "text-end", + "finish-step", + "finish" + ], + "tools": [ + "Terminal" + ], + "approval": false, + "errors": [], + "reply": "WROTE" + } + }, + "deny": { + "pass": true, + "why": "denied: the gated command never executed (outcome=error)", + "paused_finish_other": true, + "turn_paused": { + "http": 200, + "ms": 7965, + "finish": "other", + "frames": [ + "start", + "start-step", + "reasoning-start", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-delta", + "reasoning-end", + "tool-input-start", + "tool-input-available", + "tool-input-available", + "tool-input-available", + "tool-approval-request", + "finish-step", + "finish" + ], + "tools": [ + "Terminal" + ], + "approval": true, + "errors": [], + "reply": "" + }, + "turn_resumed": { + "http": 200, + "ms": 1750, + "finish": "stop", + "frames": [ + "start", + "start-step", + "tool-input-start", + "tool-input-available", + "tool-output-error", + "reasoning-start", + "reasoning-delta", + "reasoning-delta", + "reasoning-end", + "text-start", + "text-delta", + "text-delta", + "text-delta", + "text-end", + "finish-step", + "finish" + ], + "tools": [ + "Terminal" + ], + "approval": false, + "errors": [], + "reply": "The bash tool execution was not permitted. Please check your permissions or hooks configuration if you'd like to authorize this command." + } + }, + "warm": { + "pass": true, + "why": "turn1=9497ms, turn2=1721ms, turn3=1453ms (warm gain 8044ms)", + "session_id": "6b22fca8-2db1-4965-9388-0e742e1ec36d", + "times_ms": [ + 9497, + 1721, + 1453 + ] + } + } + } +} \ No newline at end of file diff --git a/.claude/skills/agent-release-gate b/.claude/skills/agent-release-gate new file mode 120000 index 0000000000..56ca7e5985 --- /dev/null +++ b/.claude/skills/agent-release-gate @@ -0,0 +1 @@ +../../.agents/skills/agent-release-gate \ No newline at end of file diff --git a/.gitignore b/.gitignore index 0a3c51b685..cbd9ed7d79 100644 --- a/.gitignore +++ b/.gitignore @@ -23,6 +23,9 @@ __pycache__/ **/error-*.log **/*.logs **/*.log +# QA evidence runs keep the raw runner log alongside results.json (e.g. run.log) so a +# cold successor can corroborate findings without re-running the experiment. +!docs/design/agent-workflows/projects/qa/runs/**/run.log /.nox/ /docs/_build/ @@ -84,11 +87,13 @@ services/runner/tests/results/ .agents/* !.agents/skills/ .agents/skills/* +!.agents/skills/agent-release-gate/ !.agents/skills/write-template-playbooks/ !.claude/ .claude/* !.claude/skills/ .claude/skills/* +!.claude/skills/agent-release-gate !.claude/skills/write-template-playbooks # Temporary SDK copies created by run.sh --local diff --git a/.gitleaks.toml b/.gitleaks.toml index 89342722d0..7ba52c3ec1 100644 --- a/.gitleaks.toml +++ b/.gitleaks.toml @@ -43,6 +43,16 @@ regexes = [ # Dummy SeaweedFS STS signing key shown (commented) as a format hint in the dev # env examples. Decodes to "this-is-a-signing-key-for-mounts" — not a real secret. '''dGhpcy1pcy1hLXNpZ25pbmcta2V5LWZvci1tb3VudHM=''', + # ------------------------------------------------------ QA SYNTHETIC CANARIES + # Synthetic canary tokens the agent-runtime QA probes plant and recall to test + # memory/compaction (e.g. QA-MEM-0EF553A2C1C1, QA-CONC0-1A2B3C4D). They land in + # the runs/**/results.json evidence files — random-looking but NOT secrets. + '''QA-[A-Z]+[0-9]?-[0-9A-Fa-f]{8,}''', + # Synthetic EARLY/LATE canary tokens the cold-context QA probe plants inside a + # large tool output to test truncation/eviction across the 4000-char transcript + # cap (e.g. EARLY-3f9a1c2b..., LATE-8e0d5f6a...). Same story: land in + # runs/**/results.json + run.log evidence files — random-looking but NOT secrets. + '''(EARLY|LATE)-[0-9A-Fa-f]{16,}''', # ---------------------------------------------------------------------------- ] diff --git a/docs/design/agent-workflows/projects/qa/README.md b/docs/design/agent-workflows/projects/qa/README.md index e1ffed5ad0..8d9eb35b74 100644 --- a/docs/design/agent-workflows/projects/qa/README.md +++ b/docs/design/agent-workflows/projects/qa/README.md @@ -1,3 +1,7 @@ +> The runnable release gate graduated to the `agent-release-gate` skill +> (`.agents/skills/agent-release-gate/`). This folder is now the release-night findings archive +> (STATUS.md, findings.md, matrix.md, LESSONS moved with the skill). + # Agent-workflows QA and autohealing recipe This folder holds the manual QA program for the agent-workflows feature, the findings it diff --git a/docs/design/agent-workflows/projects/qa/STATUS.md b/docs/design/agent-workflows/projects/qa/STATUS.md new file mode 100644 index 0000000000..f99ac20033 --- /dev/null +++ b/docs/design/agent-workflows/projects/qa/STATUS.md @@ -0,0 +1,598 @@ +# Pre-release QA — STATUS + +Live status doc. Kept current so a cold successor (or a post-crash restart) can resume without +re-deriving anything. Last updated: 2026-07-14. + +## The goal + +Product-level sanity QA before the agent-workflows release. The question is not "is every detail +right" — it is "**if a user opens the product and does the obvious first things, do they work?**" +Detail bugs get caught later by the playground test suite. This is the gate. + +## The system under test + +- **Deployment**: `https://bighetzner.agenta.dev` — compose project `agenta-oss-team-*`, + deployed from `/home/team/agenta` (owned by user `team`, NOT readable by us; env changes go + through the agent that owns it). +- **Runner**: `agenta-oss-team-runner-1`, image `agenta-allharness-sidecar:latest` (both + harnesses baked; Claude Code is never distributed, this image is local-only). +- **Credentials**: `~/.agenta-bighetzner.env` (mode 600) — project API key, workspace/project id. +- **Logs**: `docker logs agenta-oss-team-{runner,api,web}-1` — we have read access. This is how + we see truths the UI hides (cold sessions, mount failures). +- **Vault keys set by Mahmoud in the project**: Anthropic, OpenAI, OpenRouter. + +## The matrix + +Core = harness x sandbox. Provider/auth is a sub-matrix run inside the Pi cells only (it is an +auth question, not a sandbox question — re-running it in all four cells tests the same code twice). + +| Cell | Harness | Sandbox | Model | +|---|---|---|---| +| C1 | `claude` | `local` | sonnet (alias; a full model id gets dropped to default on the Claude ACP path — F-007) | +| C2 | `claude` | `daytona` | sonnet | +| C3 | `pi_core` | `local` | `gpt-5.6-luna` (OpenAI) | +| C4 | `pi_core` | `daytona` | `gpt-5.6-luna` | +| P1 | `pi_core` | `local` | `openrouter/deepseek/deepseek-v4-flash` — OpenRouter as a native provider | +| P2 | `pi_core` | `local` | OpenRouter as a **custom OpenAI-compatible provider** (base_url `https://openrouter.ai/api/v1`) — the least-travelled path, most likely broken | + +## The journeys (run in every core cell) + +| | Journey | Passes when | +|---|---|---| +| J1 | Create agent, send one message | Turn completes; `finish` frame, not `error` | +| J2 | Write a file, then read it back **in a second turn** | File exists in the mount; content survives across turns | +| J3 | Call a tool | An unguessable token baked into the tool's return appears in the reply | +| J4 | Approval: **approve** one, then **deny** another | Approved path continues; denied path is handled cleanly, no phantom failure | +| J5 | Commit (save an agent config as a new revision) | The new revision, fetched back over the wire, carries the changed parameter AND the version bumped (see below — this is the playground's `POST /api/workflows/revisions/commit`, NOT the in-stream `data-committed-revision` frame) | +| J6 | Warm/cold | Turns 2-3 faster than 1; runner log confirms session **loaded**, not silently cold; conversation continues coherently after a forced cold restart | +| J7 | MCP smoke test | An MCP server declared in the agent config is delivered to the harness and one of its tools executes — a `tool-output-available` frame for an `mcp__*` tool (Claude only; see below) | + +Triggers are explicitly out of scope for this gate. + +## How we assert (never on model prose) + +1. **Wire** — the SSE frame types (`tool-approval-request`, `data-committed-revision`, `finish`). + Structural, not textual. +2. **Side effects** — the file is really in the mount; the revision really incremented. +3. **Logs** — the only source of truth for warm/cold and for silent mount failures. + +Where the model must prove something in text, we bake an unguessable constant into a tool's +return value, so a matching reply *proves* the tool ran. The model cannot guess it. + +## Key wire facts (verified in code, do not re-derive) + +- Turn: `POST {origin}/services/agent/v0/invoke?project_id=…` — **not** an `/api` route. +- Auth: `Authorization: ApiKey `; `project_id` goes in the **query string**, never the body. +- Headers: `Accept: text/event-stream`, `x-ag-messages-format: vercel`. SSE, `data: [DONE]` + terminated, `: keepalive` comment lines must be ignored. +- **There is no create-session endpoint** — the session id is minted client-side and materialized + lazily on first invoke. +- **Approvals are in-band.** The browser approves by re-POSTing the whole message history to + `/invoke` with the tool part set to `state: "approval-responded"`, `approval: {id, approved}`. + The REST route `/api/sessions/interactions/{id}/respond` is the *out-of-band* (Slack/trigger) + path and is NOT what the UI does — testing it would test the wrong code path. +- **"Commit" = a workflow revision commit, not a git commit.** There is no git surface in the + runner at all. +- Config selectors: `parameters.agent.harness.kind` (`pi_core`|`pi_agenta`|`claude`) and + `parameters.agent.sandbox.kind` (`local`|`daytona`). Flat `model`/`agents_md` at the template + root are rejected — use `llm.model` and `instructions.agents_md`. +- A **paused** turn (approval) finishes with `finishReason: "other"`, so "turn ended" does not + mean "turn completed". + +## Status + +| Item | State | +|---|---| +| Docker log access | DONE | +| Daytona env fix (other agent) | DONE — `local,daytona`, snapshot `agenta-agent-sandbox-v1`, target `eu` | +| Object-store fix (other agent) | DONE — SeaweedFS up, `mounts/sign` = 200, real durable cwd. F-1 closed. | +| Playwright testability backlog | DONE — `playwright-testability.md` | +| Programmatic driver | DONE — `scripts/qa_product.py` | +| C3/C4/P1 (Pi cells) | DONE — see matrix below. NOTE 2026-07-14 ~21:45: the **OpenAI vault key is out of quota** (C3/C4 chat now FAIL with a clean "insufficient credit" error — external, needs a top-up; see runs/20260714-214326). The error surfaces correctly, so the #5317 fix is live. | +| C1/C2 (Claude cells) | **DONE** — Anthropic key funded 2026-07-14; C2 ran the full six-journey set on the vault key, all green (see matrix below) | +| S1 (Pi + Codex subscription) | **DONE** — chat/tool/approve green (see matrix below) | +| P2 (custom OpenAI-compatible provider) | **BLOCKED — needs the raw OpenRouter key or a custom_provider slug** | +| UI end-to-end pass | PARTIAL — approval dock verified end-to-end; rest blocked on credit | +| J5 commit journey | **DONE** — PASS on P1; `commit` journey added to the driver (see below) | +| J7 MCP smoke test | **DONE** — PASS on C1 (Claude/local, subscription) against public DeepWiki MCP; `mcp` journey added (see below) | + +**RESUMED 2026-07-14 ~21:20: the deployment settled (runner healthy, SeaweedFS up, `chat` green +on P1 and C1). The 18:05 pause is lifted. The two new journeys below ran against this settled +stack.** + +## Matrix (last good run: runs/20260714-175715, Pi cells; Claude from runs/20260714-174932) + +| cell | harness | sandbox | model | chat | mount | tool | approve | deny | warm | +|---|---|---|---|---|---|---|---|---|---| +| C1 | claude | local | sonnet | PASS | — | PASS | PASS | PASS | — | +| C2 | claude | daytona | **haiku** (vault key) | PASS | **PASS** | PASS | PASS | PASS | PASS | +| C3 | pi_core | local | gpt-5.6-luna | PASS | PASS | PASS | PASS | PASS | PASS | +| C4 | pi_core | daytona | gpt-5.6-luna | PASS | **FAIL** (pre store-exposure; see C2 mount) | PASS | PASS | PASS | PASS | +| P1 | pi_core | local | openrouter/deepseek-v4-flash | PASS | PASS | PASS | PASS | PASS | PASS | +| S1 | pi_core | local | gpt-5.6-luna (**openai-codex subscription**) | PASS | — | PASS | PASS | — | — | + +### C2 full six-journey run (2026-07-14 ~21:53, funded Anthropic vault key) — `runs/20260714-215309` + +The never-tested cell. Connection `{"mode": "agenta"}` (the vault key — Daytona rejects +subscription auth by design), model alias `haiku` (accepted on the Claude ACP path exactly like +`sonnet`; probed first on chat, `runs/20260714-215252`). All six journeys PASS. + +- **mount PASS is the headline**: the redeploy exposed the store publicly + (`AGENTA_STORE_ENDPOINT_URL=https://bighetzner-store.agenta.dev` on the api container), and the + runner log shows the Daytona sandbox REALLY mounting it — `remote mounted agenta-store:mounts/… + (verified alive)` via geesefs against the public URL, `stage=mounts ms≈2000` (vs the 2ms + nothing-mounted signature of F-7), and **zero** `mount degraded` / `tunnel discovery failed` + lines in the window. **F-7 is fixed on this deployment by the store exposure.** (The fail-open + code path still exists for deployments without a public store — the hardening ask stands.) +- warm: 9497ms → 1721/1453ms, corroborated by two `[keepalive] hit-continue` lines for the + session (`6b22fca8`) — genuinely warm, not just faster. +- deny outcome is `error` (Claude surfaces a denied tool as tool-output-error), which the driver + accepts (`outcome != "available"`). + +### S1: Pi + Codex subscription (2026-07-14 ~21:53) — `runs/20260714-215322` (chat), `runs/20260714-215335` (tool+approve) + +New cell in the driver: `pi_core`/`local`, provider `openai-codex`, model `gpt-5.6-luna`, +connection `{"mode": "self_managed", "slug": null}` (the ChatGPT/Codex OAuth login in the +subscription sidecar, not a vault key). chat, tool, approve: **all PASS** — the subscription wire +path works end to end, including the approval gate. Contrast with C3/C4 (same model via the vault +`openai` key), which now fail on quota — the subscription path is independent of that key. + +Warm/cold looks healthy where measured: C3 4444ms -> 1166/1130ms; C4 12189ms -> 1699/1392ms. +(Latency only. The cold/warm TRUTH is in the runner log — see F-2.) + +**OpenRouter as a native provider (P1) is fully green.** That answers one of the two provider +questions. + +### J5 commit + J7 MCP (added 2026-07-14 ~21:20, settled stack) + +| cell | harness | sandbox | model | commit (J5) | mcp (J7) | +|---|---|---|---|---|---| +| P1 | pi_core | local | openrouter/deepseek-v4-flash | PASS (`runs/20260714-212356`) | SKIP (Pi rejects mcps) | +| C1 | claude | local | sonnet (subscription) | PASS | PASS (`runs/20260714-212400`) | + +- **commit** is harness-agnostic (it drives the config REST API, not a turn), so it runs and passes + the same in every cell; P1 is the recorded evidence. +- **mcp** requires a Claude harness and therefore SKIPs on every Pi cell (P1/C3/C4/P2) with a clear + message; C1 is the recorded pass. Evidence of the SKIP path: `runs/20260714-212423`. + +## Journey mechanics (J5 commit, J7 MCP) — the load-bearing facts + +**J5 commit — the endpoint and the seed trap.** "Commit" = save the agent config as a new workflow +revision, the playground's Save/Commit button. The driver drives the same REST route the UI does +(`web/packages/agenta-entities/src/workflow/api/api.ts` `commitWorkflowRevisionApi`): + +- Create the artifact + variant: `POST /api/workflows/` then `POST /api/workflows/variants/`. +- Commit: `POST /api/workflows/revisions/commit`, body + `{"workflow_revision": {slug, name, message, workflow_id, workflow_variant_id, + "data": {"uri": ..., "parameters": {...the agent config...}}}}`. Auth is `ApiKey`, `project_id` + in the query string. The agent config lives under `data.parameters` (for an agent workflow: + `data.parameters.agent.{instructions,llm,tools,harness,sandbox}`). +- Fetch back: `GET /api/workflows/revisions/{id}` → `workflow_revision.data.parameters` + + `.version`. +- **The trap that cost real time:** the FIRST commit on a fresh variant is the **v0 seed**, and the + DAO force-nulls its `data`/`flags`/`meta` (`api/oss/src/dbs/postgres/git/dao.py` + `_null_revision_fields`, guarded by `if revision.version == "0"`). A config only persists on the + **second** commit (v1). The UI does the same seed-then-commit dance. So the journey commits twice + (seed, then the real change) and asserts v0→v1 plus the changed `agents_md` token surviving a + fetch-back. `data` is `extra="forbid"` — only `{uri,url,headers,runtime,script,schemas,parameters}` + are accepted. QA artifacts are namespaced `qa-commit-` and the workflow is archived + (`POST /api/workflows/{id}/archive`) in a `finally`, so repeated runs leave nothing behind. + +**J7 MCP — what it takes to run, and why it is Claude-only.** The agent config accepts user MCP +servers under the template's `mcps` list. Each entry is a full `MCPServerConfig` +(`sdks/python/agenta/sdk/agents/mcp/models.py`), NOT a bare URL: +`{"name": "", "connection": {"type": "http", "url": "", "headers": {...}?, +"credentials": {...}?}, "policy": {"tools": {"mode": "all"}}}`. Only `type: "http"` +(Streamable-HTTP / SSE) is supported; there is no user `stdio`. + +Two hard constraints, both verified in the runner: + +1. **Claude only.** Pi refuses any run that declares `mcps` + (`services/runner/src/engines/sandbox_agent/run-plan.ts`, `PI_USER_MCP_UNSUPPORTED_MESSAGE`). + User MCP needs a harness with `capabilities.mcpTools` (Claude). The journey therefore SKIPs on + every Pi cell and runs on C1 (Claude/local). C1 uses **subscription** auth (the vault Anthropic + key is out of credit) — proven working on this stack. +2. **Public HTTPS only — a local MCP server is NOT reachable.** Both the SDK resolver + (`assert_endpoint_url_allowed`) and the runner (`validateUserMcpUrl`) run an SSRF guard that + rejects `http://` and private/loopback/metadata hosts unless `AGENTA_INSECURE_EGRESS_ALLOWED` + (SDK) or `AGENTA_AGENT_MCPS_HOST_ALLOWLIST` (runner) is set. The **harness** dials the URL — on + `local` from the runner host — so the endpoint must be a public HTTPS server reachable from the + deployment's network. + +**Infra to run J7:** no infra we host is needed — the journey uses a well-known free public +reference server, **DeepWiki** (`https://mcp.deepwiki.com/mcp`, no auth, tools +`read_wiki_structure` / `read_wiki_contents` / `ask_question`), reachable from bighetzner. Override +with `--mcp-url ` for any other public server. If in future you want to point at +a server that is not publicly reachable (a local one, or an intranet one), you would need to set +`AGENTA_AGENT_MCPS_HOST_ALLOWLIST`/`AGENTA_INSECURE_EGRESS_ALLOWED` on the runner AND make the URL +reachable from the runner host — neither is set today, so a local server will be rejected by the +SSRF guard. Assertion is wire-level: a `tool-output-available` frame for a tool named `mcp__*` +(the runner namespaces MCP tools `mcp____`, e.g. `mcp__deepwiki__read_wiki_structure`); +the runner log corroborates with an `[HITL] gate toolName="mcp__deepwiki__..." outcome=allow` line. + +## Triage: product bug vs deployment artifact + +The question that decides what blocks the release. "Deployment-independent" means the defect is in +code that behaves the same everywhere; it cannot be configured away. + +| # | Finding | Class | Reasoning | +|---|---|---|---| +| F-5 | `tool-input-available` streams a PARTIAL command; toolName flips case | **BUG, deployment-independent** | Pure wire behavior from `stream.py`. Same on every stack. | +| F-6 | Permission Policy control does not govern Claude's builtins | **BUG, deployment-independent** | `claude_settings.py:134` only emits rules for MCP-delivered tools, never for Claude's own Bash/Write/Edit. Not configurable. | +| F-8 | `/tools/discover` output is rejected by the agent config | **BUG, deployment-independent** | Schema contradiction: discovery attaches `input_schema`/`description`; `GatewayToolConfig` forbids extra keys. | +| F-3 | Pi permission layer fails OPEN (ask/deny become no-ops) | **BUG (fail-open) + image-specific TRIGGER** | The `try/catch` swallow in `pi-assets.ts:349` is on every deployment. The trigger here is the image (no `/pi-agent`, non-root). Expect NOT to reproduce on our root-ful dev image — which does not make it safe: a read-only rootfs, a k8s securityContext, or a custom `PI_CODING_AGENT_DIR` re-arms it. **It was live on the stack we are shipping from.** | +| F-7 | Daytona durable mount silently skipped -> files never persist | **FIXED on this deployment (store exposure, 2026-07-14); fail-open hardening still open** | Was CONFIRMED, see below. After the redeploy exposed the store publicly (`AGENTA_STORE_ENDPOINT_URL=https://bighetzner-store.agenta.dev`), C2 mount PASSES with `remote mounted … (verified alive)` in the runner log. The silent-skip code path still exists for any deployment whose store is not publicly reachable — the "fail loud" ask stands. | +| F-1 | Mounts 503 -> agent runs in a throwaway `/tmp` cwd | FIXED (config) + BUG (fail-open) | Store was simply not deployed. The fail-open half is the same defect as F-3/F-7. | +| F-9 | Claude harness resuming its native session | **DOWNGRADED / PARTIALLY RETRACTED — was a blocker, now a residual resilience risk** | See below. The 72h `mode=create`-only observation predates a redeploy that pulled recent upstream fixes; a 2026-07-14 decisive experiment against the redeployed stack shows native session resume now working (4/4 runs, both harnesses, `mode=load` + `loaded=true`). The lossy 4000-char rebuild path still exists as the fallback when native load fails. | +| F-10 | Daytona sandbox destroyed ~2s after a 120s park | **UNKNOWN — not config** | Config is byte-identical across both stacks (all Daytona vars empty -> code defaults). The runner has **no code path that deletes a parked sandbox** (`deleteSandbox` is defined and never called). Local Daytona sandboxes park correctly (`state: stopped`, autostop 15m). So the 1.8s destruction on bighetzner is a platform anomaly or a create->park race — needs a live re-test with tighter logging. | +| — | Claude + Daytona rejects subscription auth | **BY DESIGN, well surfaced** | `"Daytona sandboxes do not support runtime-provided (subscription) authentication. Use a managed API key … or run this harness on the local sandbox."` Not a bug. It does mean C2 genuinely needs a funded Anthropic key. | +| F-11 | **Every Pi provider error is swallowed into "The agent produced no output."** | **REGRESSION (2 days old), deployment-independent — CONFIRMED, one-line fix** | See below. Commit `42075a5e9f` disarmed `findSwallowedPiError`. Dead key, exhausted quota, rate limit, bad model — all surface to the user as "no output". | +| — | The OpenAI account is OUT OF QUOTA | **EXTERNAL — needs a top-up** | Pi's transcript: `"You exceeded your current quota…"`, `stopReason:"error"`, `totalTokens:0`. Affects EVERY OpenAI model, not `gpt-5.6-luna`. **Our own QA burned it.** No runaway spend. | + +## Findings + +**F-1 (CONFIRMED, blocker) — no object store => mounts 503 => the agent silently loses every file.** + +Reproduced on the first QA run (session `c43cafde`), and it is happening on *every* run: +``` +[sandbox-agent] sign HTTP 503 session=c43cafde… name=cwd — running without this mount +[sandbox-agent] mount degraded kind=session_cwd cause=sign_returned_no_mount +[sandbox-agent] harness=pi_core sandbox=local cwd=/tmp/agenta-sandbox-agent-JBGK1o +``` +API side: every `POST /sessions/mounts/sign` and `POST /mounts/agents/sign` returns **503**. + +Root cause chain: +1. `MountsService` raises `MountStorageUnavailable("Mount storage backend is not configured.")` + (`api/oss/src/core/mounts/service.py:388,405`) because the S3-compatible store is not enabled. + `StoreConfig.enabled = bool(access_key and secret_key)` (`api/oss/src/utils/env.py:1117`). +2. The bighetzner API container has **no `AGENTA_STORE_*` env vars at all** — verified via + `docker inspect`. +3. And there is nowhere for them to point: **the published `gh` compose ships no object store.** + `seaweedfs` exists only in `docker-compose.dev.yml` (OSS *and* EE); both `docker-compose.gh.yml` + files have no store service. `env.oss.gh.example:360` marks ACCESS_KEY/SECRET_KEY as required + with a `replace-me` placeholder, while `AGENTA_STORE_ENDPOINT_URL` defaults to + `http://seaweedfs:8333` — a host that does not exist in a `gh` deployment. + +Two distinct defects, and the second is the one that scares me: + +- **(a) Hosting gap.** A self-hoster using the published compose has no store unless they bring + their own S3. If bring-your-own-S3 is intended, the default endpoint pointing at a nonexistent + `seaweedfs` host is a trap. +- **(b) Silent data loss.** When signing fails, the runner **continues anyway** with a throwaway + `/tmp` cwd. The turn succeeds, the UI renders a perfectly normal answer, and every file the + agent wrote is gone. Nothing surfaces to the user. This should fail loudly or at minimum show a + degraded state. As shipped, a self-hoster's agent appears to work and silently cannot persist + anything. + +Blocks J2 and J5 in **all four cells**. Until a store exists, matrix results for those journeys +are meaningless. + +**F-3 (CONFIRMED, security blocker) — the Pi permission layer FAILS OPEN. `ask` never prompts and +`deny` never blocks.** + +Proven on C3 (pi_core/local) before the workaround below: with +`runner.permissions.default = "deny"`, the bash tool **executed anyway** (wire showed +`tool-output-available`), and with `"ask"` no approval gate ever fired. The entire Layer-2 +permission posture was a no-op. + +Root cause chain: +1. Pi's builtin permissions are enforced by an **extension** the runner installs into + `PI_CODING_AGENT_DIR` (`run-plan.ts:212 permissionPlanCouldGatePiBuiltin`). +2. Installing it does `mkdirSync(join(agentDir, "extensions"))` + (`services/runner/src/engines/sandbox_agent/pi-assets.ts:346-351`). +3. That call is wrapped in a try/catch that **swallows the failure and logs `pi extension install + skipped: …`** — then the run proceeds with no enforcement at all. **The security boundary + fails open.** +4. Trigger on this deployment: `PI_CODING_AGENT_DIR=/pi-agent` (the compose default), but + `/pi-agent` **does not exist** in `agenta-allharness-sidecar:latest` and the runner runs as + **uid 1000 (node)**, so it cannot create a directory at `/`. Our EE dev runner image runs as + **root** and ships `/pi-agent`, which is why this never showed up locally. + +Causal link verified: after `docker exec -u 0 … mkdir -p /pi-agent && chown 1000:1000 /pi-agent`, +the gate fires correctly (`[HITL] gate toolName="Bash" permission=deny outcome=deny`) and both +approve and deny pass. + +The image is only the trigger. **The defect is the fail-open.** Any deployment where that dir is +not writable — non-root user, read-only rootfs, a k8s `securityContext`, a custom +`PI_CODING_AGENT_DIR` — silently loses ask/deny with nothing surfaced to the user. The fix is to +**fail closed**: if the permission plan needs extension enforcement and the extension cannot +install, the run must error, not proceed. + +NOTE: all C-cell results below were obtained WITH the manual `/pi-agent` workaround applied to the +running container. That workaround is lost on container recreate. + +**F-11 (CONFIRMED, regression from `42075a5e9f`, one-line fix) — every Pi provider error is +swallowed and shown to the user as "The agent produced no output."** + +The OpenAI account is out of quota (external, needs a top-up — our own QA burned it). Pi recorded +the real cause on disk: +```json +{"stopReason":"error","provider":"openai","model":"gpt-5.6-luna","usage":{"totalTokens":0}, + "errorMessage":"You exceeded your current quota, please check your plan and billing details. …"} +``` +The user saw: **"The agent produced no output."** The runner log said only +`[sandbox-agent] prompt stopReason=end_turn`. + +The runner HAS a helper for exactly this — `findSwallowedPiError` +(`services/runner/src/engines/sandbox_agent/pi-error.ts`, wired at `sandbox_agent.ts:2151`). It +reads `join(piAgentDir, "sessions")` (`pi-error.ts:103`). But commit **`42075a5e9f` "fix(agent): +persist Pi transcripts in session workspaces" (2026-07-13)** moved Pi's transcripts to +`/agents/sessions/pi/` (`pi-assets.ts:31-33`, `piSessionWorkspaceDir`) and **never updated the +callsite**. On the live runner: +``` +$ docker exec … ls /pi-agent/sessions +ls: cannot access '/pi-agent/sessions': No such file or directory +``` +`readdirSync` throws -> the helper returns `undefined` -> the engine falls through to the empty-turn +path. **The error-surfacing mechanism was silently disarmed two days ago.** + +Fix: pass the transcript root, not the agent dir — `join(plan.cwd, "agents")`, so the existing +`join(piAgentDir, "sessions")` resolves. Add a regression test asserting the callsite's dir matches +`piSessionWorkspaceDir`. Note the Daytona path was never covered by this helper by design +(`pi-error.ts:18-19`), so C4 keeps swallowing until the error is surfaced over ACP. + +Consequence for users: a dead key, an exhausted quota, a rate limit, or a bad model id all present +as "the agent produced no output" — the single least actionable message we could show. + +**F-9 (DOWNGRADED / PARTIALLY RETRACTED 2026-07-14 — was "CONFIRMED on both stacks, likely THE +long-conversation bug") — the Claude harness never resumes its native session. Every turn is +rebuilt from a lossy hand-rendered transcript.** + +**Original claim, kept for the record.** Evidence from our OWN dev stack (not bighetzner), over +72h of real dogfooding traffic: + +| harness | `create_session mode=create` | `mode=load` | +|---|---|---| +| claude | **96** | **0** | +| pi_core | 21 | 50 | + +and the `[continuity] session/load attempted … harness=…` line **never once names claude** (48 +`loaded=true` events, all `pi_core`). So this is not a bighetzner artifact and not our driver's +history bug — the claude path never even *attempts* to reload its session. + +**Why this matters more than it looks.** There are two different mechanisms: +- the **keepalive pool** (an in-memory warm process, TTL 60s) — Claude DOES hit this, which is why + our C1 warm journey passes (3847ms -> 1772ms); +- **session load from disk** (`mode=load`) — resuming the harness's OWN native session. Claude + **never** does this. + +So the moment the 60s pool TTL lapses — or the runner restarts, or the pool evicts under load — a +Claude conversation does not resume. It is reconstructed from the runner's hand-rendered +transcript (`transcript.ts buildTurnText`), which **hard-truncates every tool result at 4000 chars** +and blind tail-slices the whole history at 100k. Lossy, verbatim character deletion, re-applied on +every cold turn. + +That is a very strong candidate for the reported "long conversations with lots of tool output lose +information" — and it predicts the loss is **worse on Claude than on Pi**, which matches a +tool-heavy Claude session degrading over time. Worth confirming with a long Claude conversation +that crosses the 60s pool TTL between turns. + +**What the decisive experiment showed (2026-07-14, `runs/coldctx-20260714-201002/`).** The +prediction above was directly testable: plant an EARLY token near the start of a ~7.3k-char bash +output and a LATE token past the 4000-char truncation cap, wait 75s (past the 60s keepalive pool +TTL) so the session goes cold, then ask for both tokens back in the same session via a faithful +history replay (full assistant `parts`, tool calls included — see LESSONS.md #1). If F-9 as +written were still true, `claude` should recall EARLY but lose LATE. + +It did not. Run against the redeployed bighetzner stack — 2 runs on `claude/local` (subscription +auth) and 2 runs on `pi_core/local` (`openrouter/deepseek/deepseek-v4-flash`) — **both tokens came +back in all 4/4 runs**, and the runner log confirms the session genuinely went cold and then +genuinely reloaded natively for both harnesses: + +``` +[sandbox-agent] [continuity] session/load attempted session=3aa4d192… harness=claude loaded=true +[sandbox-agent] [timing] stage=create_session ms=1480 sandbox=local/… session=3aa4d192… mode=load +``` + +(and the matching pair for `pi_core`). Claude native session resume works on this build. The +`mode=create`-only signature from the 72h dogfooding window is gone. + +**Residual risk (this is now a resilience concern, not an every-cold-turn data-loss bug).** The +lossy 4000-char rebuild path described above still exists in code and is still real — it is just +no longer the routine outcome of crossing the 60s pool TTL. It remains the fallback whenever +native load genuinely fails: sandbox recreated, a Daytona teardown, or a runner restart. Those +cases still deserve hardening (fail loud instead of silently truncating), but they are no longer +"every long Claude conversation eventually loses information" — they are "a conversation that +survives an infrastructure event may lose information," a materially smaller and rarer blast +radius. + +**Likely cause of the change.** The most plausible explanation is the redeploy the other agent ran +tonight against bighetzner, which pulled in recent upstream fixes to the runner/session-continuity +path (see the STATUS PAUSED note above — QA was explicitly paused because the deployment was +mid-repair). The original 96/96 `mode=create` observation predates that redeploy and was never +re-run against the rebuilt stack until tonight's experiment. See LESSONS.md for the general rule +this produced: re-run any blocker-level finding after a redeploy before trusting it. + +**F-8 (open, DX blocker) — `/api/tools/discover` returns a tool object that the agent config +REJECTS. The discover -> configure round trip is broken.** + +Discovery hands back a ready-looking gateway tool: +```json +{"type":"gateway","provider":"composio","integration":"gmail","action":"FETCH_EMAILS", + "connection":"gmail-79x","input_schema":{...},"description":"..."} +``` +Feeding that straight back into `parameters.agent.tools` fails the run with HTTP 500: +``` +Invalid tool configuration: [{'type':'extra_forbidden','loc':('gateway','input_schema')}, + {'type':'extra_forbidden','loc':('gateway','description')}] +``` +`GatewayToolConfig` (`sdks/python/agenta/sdk/agents/tools/models.py:105`) accepts only +`type/provider/integration/action/connection/name` (+`permission`). `input_schema` and +`description` are legal on a **client** tool but forbidden on a **gateway** tool — and discovery +attaches them anyway. + +So the obvious flow — "discover a capability, put it on your agent" — 500s. Our own builder agent +and any SDK user hits this. Fix: either have discovery emit the config-shaped object, or have +`GatewayToolConfig` ignore the two informational keys. + +Second, smaller trap in the same area: **the action name has no integration prefix.** It is +`FETCH_EMAILS`, not `GMAIL_FETCH_EMAILS`. Guessing the prefixed name (which appears inside the +tool's own description text) fails the run with +`Gateway tool resolution failed: Action not found: composio/gmail/GMAIL_FETCH_EMAILS (HTTP 404)`. + +Once the extra keys are stripped, Gmail AND GitHub gateway tools resolve and execute correctly on +`pi_core`/`local`. + +**F-7 (CONFIRMED, blocker for Daytona) — on Daytona, the durable mount is SKIPPED when the object +store is in-network, so files do not persist across turns. Silently.** + +Confirmed at the log level on the settled stack (session `226c563d`): +``` +[sandbox-agent] tunnel discovery failed: fetch failed +[sandbox-agent] [timing] stage=mounts ms=2 <- 2ms: nothing was mounted +[sandbox-agent] harness=pi_core sandbox=daytona cwd=/home/sandbox/agenta/mounts/... <- unbacked +``` +And separately: `reconnect failed sandbox=daytona/… from state 'destroyed', creating fresh` — the +Daytona sandbox is being destroyed between turns, so there is no fallback either. + +### Why this hits EVERY self-hoster, and why we could never have seen it in dev + +The full chain, all verified in the repo: + +1. A **remote** sandbox can mount the store directly only if the store is **publicly reachable** + (`mount.ts:181`). Otherwise the runner needs a **tunnel** (`mount.ts:505-537`). +2. The tunnel is **ngrok**, and ngrok is a service in **`docker-compose.dev.yml` ONLY** — both + `gh` composes (OSS and EE — the ones self-hosters actually run) have **no tunnel service**. +3. PR **#5315** ("bundle SeaweedFS store in gh.local and gh.ssl") just added the store to `gh`, + bound to **loopback** with **`traefik.enable=false`**: + ```yaml + ports: ["${AGENTA_STORE_PORT:-127.0.0.1:8333}:8333"] + labels: ["traefik.enable=false"] + # "In-network services reach it directly at seaweedfs:8333, no publish needed." + ``` + Correct for the API and runner. **A Daytona sandbox is not in-network.** +4. So on a `gh` deployment + Daytona + the bundled store: sandbox cannot reach the store -> + tunnel discovery finds nothing -> `mount.ts` **skips the mount, "not fatal"** -> the agent runs + in an unbacked directory -> **every file it writes is lost**, silently, with a normal-looking UI. + +**The fix for F-1 (bundling the store) is what makes F-7 universal.** And the reason it survived +until now: **dev ships ngrok, `gh` does not** — our development environment contains a component +our shipping environment lacks, so this class of bug is invisible to us by construction. That is +the process finding, and it is worth more than the bug. + +Options: publish the store to the sandbox (route it through Traefik with auth), ship a tunnel in +`gh`, document "Daytona requires a publicly reachable S3", or — at minimum, and regardless — +**stop failing open**: a remote sandbox that cannot attach its durable mount must ERROR, not +silently degrade to an ephemeral cwd. + +C4 (pi_core / **daytona**) failed the mount journey while C3 (pi_core / **local**) passed it, same +driver, same model. Turn 1 wrote a token to `qa-mount.txt` and reported WROTE; turn 2's `cat` came +back **empty**. + +Mechanism, from `services/runner/src/engines/sandbox_agent/mount.ts`: +- `:181` — a remote sandbox can mount the store directly only if the store is **publicly + reachable**; an in-network store needs a **tunnel**. +- This deployment's store is `AGENTA_STORE_ENDPOINT_URL=http://seaweedfs:8333` — in-network. A + Daytona sandbox out on `daytonaproxy01.eu` cannot reach that host. +- `:505` — when no tunnel is up, "the remote mount is **skipped, not fatal**". + +So the Daytona agent runs with an ephemeral cwd inside the remote sandbox, the turn succeeds, the +UI looks normal, and the files are gone at the next turn. **This is F-1 all over again — the same +fail-open shape, one layer down.** F-1 is fixed for `local`; the Daytona half was still broken +after the store landed. + +NOT YET CONFIRMED at the log level: I was about to grep the runner for the tunnel/skip lines when +QA was paused for the deployment repair. Confirm with: +`docker logs agenta-oss-team-runner-1 | grep -iE 'tunnel|remote mount|skipped'` on a fresh Daytona +run, then re-run `uv run qa_product.py --cell C4 --only mount`. + +If confirmed, the fix is one of: expose the store publicly to Daytona sandboxes, stand up the +tunnel, or — at minimum — **stop failing open**: a skipped durable mount on a remote sandbox must +surface, not silently degrade to an ephemeral cwd. + +**F-6 (open, moderate) — the generic permission Policy control does not govern Claude's builtin +tools.** The playground's Permissions section shows a **Policy** select (`allow_reads` / `allow` / +`ask` / `deny`) that is NOT conditional on harness, with the help text "Deny all — Every tool call +is refused" and "Ask — A human approves every tool call". On Claude, `runner.permissions.default` +only renders rules for tools delivered over the internal `agenta-tools` MCP server +(`sdks/python/agenta/sdk/agents/adapters/claude_settings.py:134`); it renders **nothing** for +Claude's own builtins (Bash/Terminal, Write, Edit). Those are governed only by `harness.permissions` +(the separate Claude-only control) plus Claude Code's own command classifier. + +Measured on C1 (claude/local), holding the tool fixed and varying only the policy: + +| Policy | read-only `echo` | mutating `echo > file` | +|---|---|---| +| unset (`allow_reads`) | runs, no gate | gate fires (correct) | +| `ask` | **runs, no gate** | gate fires | +| `deny` | **runs, no gate** | refused (correct) | + +So the dangerous operations ARE gated correctly on Claude — via Claude's own classifier, not via +our policy. The defect is narrower than it first looked: **a user who selects "Deny all" on Claude +still gets read-only shell commands executing**, contradicting the UI's own help text. Fix by +either making the Policy control harness-aware (hide/annotate it for Claude) or by translating the +policy into Claude builtin rules. + +**Same mechanism explains a UI-exploration finding filed as a HIGH persistence bug.** +`ui-exploration-20260714.md` bug 1 originally read "one Terminal approval becomes a permanent +grant; Policy=Ask not enforced." A follow-up investigation (2026-07-14, live probes + code trace) +found no persisted grant: approvals are answered once-only, no settings file is written, and a +mutating command re-gates every time including in new sessions. The "Terminal never asked again" +runs were read-only commands auto-approved by Claude Code's own classifier under Ask — this table's +`ask` / read-only `echo` row. Downgraded from HIGH (security persistence) to MEDIUM (policy-label +gap), same family as F-6 above; entry corrected in place with the original observation kept for the +record. + +**RETRACTED (was reported as a Claude permission bypass):** an earlier reading of this — "`ask` +silently drops the tool call on Claude" — was **model non-determinism**, not a defect: the model +simply chose not to call bash on that run. On rerun the gate fired correctly. Recorded here +because it is exactly the failure mode this driver exists to prevent, and I nearly shipped it as a +finding. + +**F-4 (open, minor) — `code` tools are accepted by the SDK and rejected at run time by the sidecar.** +`CodeToolConfig` exists in the SDK, but the product path hard-fails with "Code tools are not +supported by the sidecar." (`services/runner/src/tools/code.ts`). The playground UI does not offer +code tools, so a UI user cannot hit this — but an SDK user can configure one and only find out at +run time. Either reject at config time or support them. + +**F-5 (open, wire hygiene) — `tool-input-available` carries INCOMPLETE input, and `toolName` +changes case mid-stream.** +The frame is emitted repeatedly for a single tool call with a progressively-built partial input: +``` +toolName "bash" input {"command": "echo \"QA-BASH-"} <- partial +toolName "bash" input {"command": "echo \"QA-BASH-$(hostname"} +toolName "Bash" input {"command": "echo \"QA-BASH-$(hostname)\""} <- complete; name case flips +``` +A client that reads the first frame — a reasonable reading of "available", given +`tool-input-start` already exists for the streaming phase — approves a **truncated command under +the wrong name**. The runner keys approval decisions by name+args, so the decision then misses the +parked gate and the approval **re-parks forever** (the agent asks for the same approval on every +turn). This cost real debugging time in the driver and would cost an integrator the same. + +**F-2 (open, testability) — warm/cold is not observable from the browser.** The UI has no warm/cold +concept (`sessionStatusAtomFamily` is only running/awaiting/error/idle) and the wire does not carry +it. Only the runner log knows. Means J6 can never be a CI test until the wire carries it. See +`playwright-testability.md`. + +## Two things that need Mahmoud + +1. **RESOLVED 2026-07-14: Anthropic key funded.** C2 ran the full six-journey set on the vault key, + all green (`runs/20260714-215309`). Replaced by a NEW ask: **the OpenAI vault key is out of + quota** — C3/C4 (vault `openai` provider) fail chat with a clean "insufficient credit" error + (`runs/20260714-214326`). The Codex-subscription path (S1) is unaffected. +2. **P2 needs the raw OpenRouter key** (or a `custom_provider` secret slug created in the UI). + Secrets are encrypted at rest, so the key already in the vault cannot be read back. P2 tests + OpenRouter as a **custom OpenAI-compatible provider** (`base_url = https://openrouter.ai/api/v1`) + — the path every self-hoster with a proxy or a local vLLM uses, and the least-travelled one. + +## Phantom failures — do not trust results from a stack mid-restart + +A full matrix run at 17:51 showed every Pi cell failing with HTTP 500 (`Could not verify +credentials: … returned unexpected status code 404`, and a JSON decode error). **These were not +product bugs** — the other agent was recreating the api/cron/worker containers at that moment. +Re-running against the settled stack turned them all green. Likewise a UI-visible +`404 on /api/workflows/revisions/resolve` and a burst of +`[sessions/persist] DROPPED … after 3 retries: fetch failed` + `getaddrinfo ENOTFOUND api` in the +runner: all from the same restart window, all unreproducible afterwards. Rule for this pass: +**never diagnose from a run that overlapped a container restart — re-run first.** + +(The `DROPPED` behaviour is still worth a look on its own: session records — tool calls, results, +usage — are dropped after 3 retries and the turn proceeds regardless. Fail-open again, though the +trigger here was legitimate.) + +## Recovery + +If everything is lost: read this file, then `playwright-testability.md`, then +`scripts/qa_product.py` (the driver). The credentials are in `~/.agenta-bighetzner.env`. Nothing +in this pass mutates the deployment except through the product's own API, so there is nothing to +roll back. diff --git a/docs/design/agent-workflows/projects/qa/cleanup-plan.md b/docs/design/agent-workflows/projects/qa/cleanup-plan.md deleted file mode 100644 index 7feff16124..0000000000 --- a/docs/design/agent-workflows/projects/qa/cleanup-plan.md +++ /dev/null @@ -1,137 +0,0 @@ -# Agent-workflows QA session: workspace cleanup plan - -Date: 2026-06-21. Repo: `/home/mahmoud/code/agenta` (GitButler workspace, branch -`gitbutler/workspace`). This is a read-only survey for the user to review before any -cleanup. Nothing here is committed, staged, or modified. - -## Summary - -The uncommitted tree is dominated by one effort that is NOT ours: the in-flight -`rivet -> sandbox-agent` rename and restructure, already partly committed in the -`chore/sandbox-agent-core` lane (HEAD sits on top of it). That rename touches almost -every `services/agent/**`, the SDK adapters, the Python agent service, hosting (compose, -k8s, railway), CI, and many existing design docs. On top of it ride a frontend -`AgentChatSlice` change and GitButler's own hook backups. Our QA session produced only -four kinds of change: (1) the Composio no-auth tools fix, which is already byte-identical -in PR #4785, so the workspace copies are pure duplicates to discard; (2) the QA docs and -new design-proposal folders, most of which are newer than what PR #4779 holds or are not -in #4779 at all, so they need a docs update; (3) the F-001 system-prompt fix, which lives -inside the renamed `services/agent/src/engines/sandbox_agent.ts` and is therefore tangled -with the rename, so it belongs on the rename lane, not on PR #4778 (which still ships the -old `rivet.ts`); (4) runner Docker tweaks, most of which already landed in PR #4778. The -net cleanup is small: discard the four Composio duplicates, land the QA/proposal docs, -and route the F-001 hunk onto the rename lane. Everything else stays with its owner. - -## Key facts established by diffing (not assumed) - -- The four Composio files (`dtos.py`, `providers/composio/adapter.py`, `service.py`, - `tests/.../test_no_auth_connection.py`) are byte-for-byte identical to - `origin/fix/composio-no-auth-toolkits` (PR #4785). Verified by `git hash-object` == - `git rev-parse origin/...:` for all four. -- PR #4778 (`feat/agent-runner-engines`) still ships `services/agent/src/engines/rivet.ts` - and the old `services/agent/test/` layout. The workspace has already renamed that to - `services/agent/src/engines/sandbox_agent.ts` and moved tests to - `services/agent/tests/unit/` (the `chore/sandbox-agent-core` lane, in HEAD). So the - F-001 fix cannot land on #4778 cleanly; it must follow the rename. -- `services/agent/test/skills.test.ts` and `services/agent/test/extension-tools.test.ts` - do not exist at those paths anymore. The whole `services/agent/test/` dir is gone - (shown as deleted), replaced by `services/agent/tests/unit/**`, which is already - committed in HEAD. `extension-tools.test.ts` now lives at - `services/agent/tests/unit/extension-tools.test.ts`, rewritten for vitest. The git - status snapshot in the task prompt was stale on this point. -- The QA docs that ARE in #4779 (`qa/README.md`, `qa/matrix.md`, `qa/findings.md`, - `qa/regression-*.md`, `qa/scripts/*`) still have large workspace diffs vs #4779 - (200-360 lines each). #4779 holds an older snapshot; the workspace holds our newer QA - content. These are real content updates, not rename noise. -- The proposal folders `skills-config`, `model-config`, `harness-capabilities`, - `code-tool-sandbox`, plus `qa/implementation-plan.md` and `feature-matrix-test.md`, are - in no lane and not in #4779. They are purely uncommitted, ours, and need a home. -- `e4_local_sdk.py` (named in the task brief) does not exist under `qa/scripts/`. Only - `run_matrix.py` and `mcp_qa_server.mjs` are there. -- `code-tool-sandbox/` has two extra files beyond the brief's list: `security-review.md` - and `status.md` (both ours). -- `.agents/skills/**` is gitignored and does not appear in status; not chased. - -## Classification table (grouped by destination) - -### Discard as duplicate of PR #4785 - -| File / group | Owner | Already landed | Destination | Notes | -|---|---|---|---|---| -| `api/oss/src/core/tools/dtos.py` | ours | yes, #4785 | discard (duplicate of #4785) | hash-identical to PR branch | -| `api/oss/src/core/tools/providers/composio/adapter.py` | ours | yes, #4785 | discard (duplicate of #4785) | hash-identical | -| `api/oss/src/core/tools/service.py` | ours | yes, #4785 | discard (duplicate of #4785) | hash-identical | -| `api/oss/tests/pytest/unit/tools/test_no_auth_connection.py` | ours | yes, #4785 | discard (duplicate of #4785) | untracked here, but content == PR #4785 blob | - -### Land in a docs update (new docs PR, or update #4779) - -| File / group | Owner | Already landed | Destination | Notes | -|---|---|---|---|---| -| `docs/design/agent-workflows/qa/README.md`, `matrix.md`, `findings.md`, `regression-testing-research.md`, `regression-skill-DRAFT.md` | ours | partial: older copy in #4779 | update #4779 (or new docs PR) | workspace is newer (F-012/13/14, F-008 downgrade); 200-360 line diffs vs #4779 | -| `docs/design/agent-workflows/qa/scripts/run_matrix.py`, `mcp_qa_server.mjs` | ours | partial: older copy in #4779 | update #4779 | workspace newer; real content diffs | -| `docs/design/agent-workflows/qa/implementation-plan.md` | ours | no | new docs PR / #4779 | not in #4779 | -| `docs/design/agent-workflows/qa/runs/**` (21 json) | ours | yes, #4779 | likely discard / no-op | all 21 already in #4779; confirm no content drift before re-landing | -| `docs/design/agent-workflows/skills-config/**` | ours | no | new docs PR / #4779 | proposal folder, in no lane | -| `docs/design/agent-workflows/model-config/**` | ours | no | new docs PR / #4779 | proposal folder, in no lane | -| `docs/design/agent-workflows/harness-capabilities/**` | ours | no | new docs PR / #4779 | proposal folder, in no lane | -| `docs/design/agent-workflows/code-tool-sandbox/**` | ours | no | new docs PR / #4779 | proposal folder incl. `explainer.md`, `security-review.md`, `status.md` | -| `docs/design/agent-workflows/feature-matrix-test.md` | ours | no | new docs PR / #4779 | live-test report from the prior session; in no lane | -| `docs/design/agent-workflows/qa/cleanup-plan.md` (this file) | ours | no | new docs PR / #4779 (optional) | survey artifact; optional to commit | - -### Mixed: needs care (our hunk tangled with the rename) - -| File / group | Owner | Already landed | Destination | Notes | -|---|---|---|---|---| -| `services/agent/src/engines/sandbox_agent.ts` | mixed (rename = other, F-001 = ours) | no | F-001 hunk -> `chore/sandbox-agent-core` lane (or a PR stacked on it) | F-001 = the `writeSystemPromptLocal` / `uploadSystemPromptToSandbox` additions + the `system`/`append_system` wiring (~lines 197-236, 907-930). Cannot go to #4778 (#4778 still has `rivet.ts`). The rename body itself is OTHER. | -| `hosting/docker-compose/ee/docker-compose.dev.yml` | mixed (rename = other, MCP-flag = ours) | MCP flag: yes, in #4776 | leave the rename for its owner; our `AGENTA_AGENT_ENABLE_MCP` already in #4776 | WS replaces `agent-pi`->`sandbox-agent`, `AGENTA_AGENT_PI_URL`->`AGENTA_RUNNER_URL`, drops the RUNTIME/HARNESS/SANDBOX vars (all rename). Our MCP flag survives and is already in #4776. Nothing of ours to add. | -| `services/agent/docker/Dockerfile` | mixed mostly other | python3: yes, in #4778 | leave for rename owner | WS-vs-#4778 delta is only `USER node` (restructure), not ours. python3 already in #4778. | -| `services/agent/docker/Dockerfile.dev` | other (rename + skills COPY) | dev rebuild + python3: yes, in #4778 | leave for rename owner | WS-vs-#4778 delta is rename text + `COPY skills ./skills` (AgentaHarness), not a QA fix | - -### Leave for owner (the rivet -> sandbox-agent rename / restructure) - -All OTHER. None contain a QA-session change. These belong to the -`chore/sandbox-agent-core` lane (or the relevant feature lane) and should stay there. - -| File / group | Owner | Destination | Notes | -|---|---|---|---| -| `services/agent/src/**` except the engines above (`cli.ts`, `server.ts`, `extensions/agenta.ts`, `protocol.ts`, `responder.ts`, `tools/*.ts`, `tracing/otel.ts`), `engines/pi.ts` | other | leave (rename lane) | `runRivet`->`runSandboxAgent`, engine string `rivet`->`sandbox-agent`, env renames; `cli.ts`/`server.ts` also carry the TS-structure testability refactor | -| `services/agent/src/engines/skills.ts` (untracked) | other | leave (rename lane) | shared bundled-skill resolver (AGENTA-on-sandbox-agent) | -| `services/agent/sandbox-images/**` (untracked) | other | leave (rename lane) | Daytona runner image assets | -| `services/agent/test/*` (all deleted) | other | leave (rename lane) | old test dir removed; replaced by `tests/unit/**` (already in HEAD) | -| `services/agent/{README.md, docker/README.md, package.json, pnpm-lock.yaml, tsconfig.json}` | other | leave (rename lane) | rename + vitest/restructure | -| `sdks/python/agenta/sdk/agents/**` (`__init__.py`, `adapters/*`, `dtos.py`, `interfaces.py`, `utils/*`), `sdks/python/agenta/__init__.py` | other | leave (rename lane) | `RivetBackend`->`SandboxAgentBackend`, env renames | -| `sdks/python/agenta/sdk/agents/adapters/rivet.py` (D) -> `sandbox_agent.py` (untracked R) | other | leave (rename lane) | the SDK side of the rename (the two `R` entries in unassigned) | -| `sdks/python/oss/tests/pytest/unit/agents/**` | other | leave (rename lane) | rename-driven test updates + AGENTA-on-sandbox-agent assertion | -| `services/oss/src/agent/{app.py, config.py, secrets.py}`, `services/oss/tests/.../test_select_backend.py` | other | leave (rename lane) | `select_backend` collapses to always `SandboxAgentBackend`; env + test rewrite | -| `hosting/**` (compose `.gh.yml` + env examples, all k8s helm/values incl. new `sandbox-agent-{deployment,service}.yaml`, all railway scripts + `sandbox-agent/Dockerfile`) | other | leave (rename lane) | adds the `sandbox-agent` service + `AGENTA_RUNNER_URL` contract | -| `.github/workflows/{12,42,43}-*.yml` | other | leave (rename lane) | adds runner unit-test job + `agenta-sandbox-agent` image build/deploy | -| `docs/docs/self-host/guides/04-deploy-on-railway.mdx`, `08-custom-agent-runner-images.mdx` (untracked) | other | leave (rename lane) | sandbox-agent runner self-host docs | -| Modified existing `docs/design/agent-workflows/*.md` (`README.md`, `adapters/{agenta,claude-code,pi}.md`, `architecture.md`, `ground-truth.md`, `implementation-review.md`, `meeting-alignment.md`, `ports-and-adapters.md`, `pr-stack.md`, `protocol.md`, `sessions.md`, `status.md`, `sdk-local-tools/*`) | other | leave (rename lane) | dominated by rivet->sandbox-agent rename; a couple cite the QA matrix but are not QA-authored | -| `docs/design/agent-workflows/provider-model-auth/**`, `typescript-structure/**` (untracked) | other | leave for owner | separate design workspaces, not ours | -| `web/oss/src/components/AgentChatSlice/state/sessions.ts` | other | leave (AgentChatSlice lane) | `crypto.randomUUID()`->`generateId()` import swap | -| `.husky/{post-checkout, pre-commit}` (M), `.husky/{post-checkout, pre-commit}-user` (untracked) | other (GitButler) | leave for GitButler | `GITBUTLER_MANAGED_HOOK_V1`; the `*-user` files are GitButler's backups of the originals | -| `.gitignore` (M) | other | leave (rename lane) | adds `services/agent/test-results/` and `coverage/` ignores | - -## Recommended cleanup actions (for user approval) - -1. Discard the four Composio files. They are byte-identical to PR #4785: revert - `api/oss/src/core/tools/{dtos.py, providers/composio/adapter.py, service.py}` to HEAD - and delete the untracked `api/oss/tests/pytest/unit/tools/test_no_auth_connection.py`. - Confirm #4785 is the surviving copy first. -2. Land the QA + proposal docs. Either refresh PR #4779 with the newer workspace versions - of the in-#4779 qa docs and scripts, and add the not-yet-in-#4779 items - (`qa/implementation-plan.md`, the four proposal folders, `feature-matrix-test.md`), or - open one new docs PR for all of it. The 21 `qa/runs/**` json files are already in - #4779; only re-land them if their content drifted. -3. Route the F-001 system-prompt fix onto the rename lane. Stage only the - `writeSystemPromptLocal` / `uploadSystemPromptToSandbox` hunks of - `services/agent/src/engines/sandbox_agent.ts` onto `chore/sandbox-agent-core` (or a PR - stacked on it), since #4778 still ships `rivet.ts` and cannot take it cleanly. Do not - touch the rename body. -4. Leave everything in the "leave for owner" group untouched. It is the rename/restructure - lane, the frontend AgentChatSlice lane, GitButler hooks, or other people's design docs. - Our runner Docker fixes (python3, dev rebuild) and our MCP compose flag already landed - in #4778 and #4776; the only remaining workspace deltas on those files are the rename, - which is not ours. -5. Optional: decide whether to commit this `cleanup-plan.md` itself with the docs in step 2 - or leave it as a local survey artifact. diff --git a/docs/design/agent-workflows/projects/qa/playwright-testability.md b/docs/design/agent-workflows/projects/qa/playwright-testability.md new file mode 100644 index 0000000000..398433d629 --- /dev/null +++ b/docs/design/agent-workflows/projects/qa/playwright-testability.md @@ -0,0 +1,135 @@ +# Making the agent UI testable + +The agent playground has no test hooks. A `grep` for `data-testid` across the whole frontend +returns 15 hits, and none are in the agent surface: `AgentChatSlice/`, `pages/agent-home/`, +`pages/agents/`, and `@agenta/playground` contain zero. + +Exactly two agent affordances have a stable accessible name today: + +- the chat input — `aria-label="Chat message"` (`packages/agenta-ui/src/RichChatInput/RichChatInput.tsx`) +- the send button — `aria-label="Send"`, which morphs into `aria-label="Stop"` while a turn is + streaming (`packages/agenta-ui/src/RichChatInput/plugins/SendButton.tsx`) + +Everything else is reachable only through English UI copy or Tailwind classes. A Playwright suite +built on those breaks the first time someone rewords a button or restyles a status icon, and — +worse — some of it breaks *silently green*, which is the failure mode that matters. Details in +"Traps" below. + +This document is the backlog to fix that. It has two tiers, and the distinction is load-bearing. + +## The two tiers + +**Tier 1 — always on.** Test IDs and state attributes. These cost nothing in production, leak +nothing, and must NOT be behind a flag. Gating them means the DOM you test is not the DOM you +ship, and the flag becomes its own source of "green in CI, broken in prod." + +**Tier 2 — behind a flag.** Internal state that has no business in a production DOM: session and +run ids, warm/cold, turn timings, wire frames. This tier exists because of a real gap, not a +convenience: **the UI has no concept of warm vs cold**. `sessionStatusAtomFamily` models only +`running / awaiting / error / idle`. So no number of test IDs will let Playwright assert the +warm/cold journey — that assertion has no DOM source at all today. Tier 2 gives it one. + +## Tier 1: attributes to add + +Ranked by what a QA suite actually needs. `data-testid` names an element; `data-state` / +`data-*` make its *state* machine-readable, so a test never has to match a CSS class or a +sentence of copy. + +### Blocking — the suite cannot be trusted without these + +| # | Affordance | Component | Add | +|---|---|---|---| +| 1 | Approval dock | `oss/src/components/AgentChatSlice/components/ApprovalDock.tsx` | `data-testid="approval-dock"` + `data-state="open\|collapsed"` on the wrapper; `data-testid="approval-approve"` and `data-testid="approval-deny"` on the buttons; `data-approval-id` and `data-tool-name` on the dock | +| 2 | Tool call in transcript | `oss/src/components/AgentChatSlice/components/ToolActivity.tsx` | `data-testid="tool-call"` + `data-tool-name={name}` + `data-status="pending\|success\|error"` on each row | +| 3 | Assistant / user message | `oss/src/components/AgentChatSlice/components/AgentMessage.tsx` | `data-testid="chat-message"` + `data-role="user\|assistant"`; on the failure bubble `data-testid="run-error"` | +| 4 | Session status | `oss/src/components/AgentChatSlice/components/SessionTagBar.tsx` (and `SessionRail.tsx`) | `data-testid="session-status"` + `data-status="running\|awaiting\|error\|idle"` — replaces matching `.bg-colorWarning` | +| 5 | Commit / revision approval body | `oss/src/components/AgentChatSlice/components/approvals/CommitRevisionApproval.tsx` | `data-testid="commit-approval"`, and `data-testid="commit-changes-summary"` on `AgentChangesSummary` | + +### Needed to reach the above (config controls) + +All three selectors live behind accordions in a drawer, so a test must open the section before +the control is even in the DOM. Add an ID to the section headers too. + +| # | Affordance | Component | Add | +|---|---|---|---| +| 6 | Harness selector | `packages/agenta-entity-ui/src/DrillInView/SchemaControls/HarnessSelectControl.tsx` | `data-testid="harness-select"` + `data-value={kind}` (`pi_core` / `pi_agenta` / `claude`); on the card variant, `data-testid="harness-option-{kind}"` | +| 7 | Sandbox selector | `packages/agenta-entity-ui/src/DrillInView/SchemaControls/EnumSelectControl.tsx` as hosted by `.../agentTemplate/useModelHarness.tsx` | `data-testid="sandbox-select"` + `data-value="local\|daytona"`. The enum labels are auto-derived by `formatEnumLabel`, so the visible text is NOT a stable hook | +| 8 | Model picker | `packages/agenta-entity-ui/src/DrillInView/SchemaControls/GroupedChoiceControl.tsx` | `data-testid="model-select"` + `data-value={model}` | +| 9 | Config sections | `ConfigAccordionSection`, `SectionDrawer` | `data-testid="config-section-{title-slug}"` | +| 10 | Create agent | `oss/src/components/pages/agent-home/components/AgentComposer/index.tsx`; `oss/src/components/pages/agents/AgentsTableSection.tsx` | `data-testid="create-agent"` on both (the agents-list one currently collides with other "Create" buttons) | + +### Nice to have + +Session tab controls already carry aria-labels (`New session`, `Rename session`, `Close +session`) and are fine as-is. The chat input and send button are fine as-is — but the existing +spec at `web/oss/tests/playwright/acceptance/agent-chat/tests.ts` uses +`getByRole("textbox").last()`, which should become +`getByRole("textbox", {name: "Chat message"})` today, for free. + +## Tier 2: the debug state bridge + +Gated behind one flag. Proposed: `NEXT_PUBLIC_AGENTA_TEST_HOOKS=1` (dev and CI only; never +enabled on a production deploy). + +When on, render a single hidden node that mirrors the state the DOM otherwise hides: + +```html +