docs(agent-workflows): plan runner self-hosting cleanup

docs/design/agent-workflows/projects/runner-selfhosting-cleanup/README.md

@@ -0,0 +1,77 @@
+# Runner self-hosting cleanup
+
+Status: DESIGNED v2 after owner review round 1, awaiting second-round review
+Date: 2026-07-13
+Target branch: `big-agents`
+
+This project makes the agent runner understandable from its deployment configuration alone. It replaces shared and overlapping environment variables with one runner-owned provider contract, removes automatic subscription credential upload, and documents explicit operator-owned subscription mounts. A capability endpoint, a bootstrap manifest, and fail-loud mount semantics are deferred open issues, not release work.
+
+## Outcome
+
+A self-hosted operator should be able to answer these questions by reading one Compose service or one Helm values block:
+
+1. Where is the runner?
+2. Which sandbox providers can it use?
+3. Which provider is the default?
+4. Which credentials belong to the runner itself?
+5. Which files are intentionally copied into a run?
+6. Which combinations of sandbox, harness, and authentication are supported?
+
+The normal deployment uses one runner service. That runner can support both local and Daytona runs at the same time. Harness choice remains per agent. Sandbox choice remains per run, constrained by the providers the operator enabled.
+
+## Decisions
+
+- The runner is a deployment-level service, not one sidecar per harness or sandbox provider.
+- The operator enables sandbox providers explicitly. Unset means local only, not every provider known to the binary.
+- The enabled-provider list is also the deployment security posture: `daytona` only means no user code ever runs inside the runner container. That is the right posture for any multi-tenant or exposed deployment.
+- API and web get provider availability from the same environment value, set once per deployment by the hosting templates. A discovery endpoint is deferred (RSH-7).
+- Daytona configuration belongs under the runner namespace. Code-evaluator Daytona configuration is separate.
+- The runner never discovers an operator's home directory or automatically uploads Pi or Claude OAuth state.
+- Subscription authentication is an explicit local self-hosting setup: a read-write Compose volume mount taught in a tutorial. The harness runs directly out of the mount so it can refresh its own OAuth token; there is no per-run copy and no bootstrap manifest in version 1 (RSH-4).
+- Runtime customization (extra binaries, certificates, dependencies) happens through operator-built images and the shipped Daytona snapshot scripts, not runner configuration.
+- Remote subscription authentication is unsupported until the relevant provider explicitly approves the integration and Agenta defines a safe product contract.
+- Pi installation is detected and repaired by the runtime. There is no deployment-wide "Pi is installed" boolean.
+- Mount failure behavior stays best-effort in version 1, with one structured warning when a durable mount degrades. The fail-loud contract is deferred (RSH-11).
+- The runner receives a narrow environment and no static Agenta API key. Per-run caller credentials authorize callbacks and trace export. It does not inherit a shared application environment or database and cryptographic secrets.
+
+## Reading order
+
+- [Context](./context.md) explains today's architecture and why the current variables are confusing.
+- [Research](./research.md) records repository evidence and the review of PRs #5274, #5285, and #5286.
+- [Interface](./interface.md) defines the target environment, Helm, provider-availability, subscription-mount, and failure contracts.
+- [Plan](./plan.md) splits the cleanup into reviewable implementation PRs.
+- [QA](./qa.md) defines the environment, harness, and credential matrix.
+- [Documentation plan](./documentation-plan.md) maps the final public docs through Diátaxis, with per-page dispositions for the existing self-host tree.
+- [Open issues](./open-issues.md) parks work that should not block the cleanup.
+- [Status](./status.md) is the owner review checklist and progress log.
+
+## In scope
+
+- Runner provider selection and configuration.
+- Runner-specific Daytona names and validation.
+- Runner-to-Services routing and authentication.
+- Local subscription setup through explicit read-write mounts.
+- Removal of automatic Daytona OAuth upload.
+- Removal of deployment-wide runtime facts and hidden policy switches.
+- A structured warning when a durable mount degrades.
+- Compose, Helm, Railway, self-hosting docs, a `hosting/README.md` layout map, and the local 8280 QA deployment.
+- One shared provider-availability value read by API and runner.
+
+## Out of scope
+
+- Implementing the Docker or E2B providers.
+- Sharing a personal subscription between users or tenants.
+- Remote subscription support.
+- A runner capability-discovery endpoint (RSH-7).
+- A declarative bootstrap-asset manifest, scripts, VPN hooks, or network overlays (RSH-4).
+- Fail-loud required-mount semantics (RSH-11).
+- Reworking code-evaluator execution in the same change.
+- Backward-compatible aliases for unpublished pre-production variables.
+- A custom-role migration bridge. Default roles and tests still need the final mount permissions.
+
+## Related work
+
+- [PR #5274](https://github.com/Agenta-AI/agenta/pull/5274) added mount and snapshot changes that exposed the immediate configuration bugs.
+- [PR #5285](https://github.com/Agenta-AI/agenta/pull/5285) proposes Docker and a sandbox allowlist. This project adopts its capability-based provider direction but replaces the permissive cross-layer allowlist contract.
+- [PR #5286](https://github.com/Agenta-AI/agenta/pull/5286) adds useful runner authentication and provider-key narrowing. This project rejects its broad Helm `commonEnv` injection and its compatibility escape hatch.
+- [Runner self-hosting explainer](../runner-selfhosting-explainer/README.md) contains the detailed as-built explanation that led to this cleanup.

docs/design/agent-workflows/projects/runner-selfhosting-cleanup/context.md

@@ -0,0 +1,99 @@
+# Context
+
+## The current architecture
+
+The agent runner is a long-lived Node service. Services sends it a run request, the runner chooses a sandbox provider, and a sandbox-agent daemon launches the selected harness.
+
+~~~text
+Services API
+  -> runner HTTP service
+       -> sandbox provider: local or Daytona
+            -> sandbox-agent daemon
+                 -> harness: Pi or Claude
+~~~
+
+The two choices are independent:
+
+- The harness belongs to the agent configuration.
+- The sandbox provider belongs to the run, with a deployment default.
+
+One runner process can therefore execute Pi locally, Claude locally, Pi in Daytona, and Claude in Daytona. We do not need separate deployment services for those combinations. Additional runner replicas are a scaling and ownership concern, not a harness-selection requirement.
+
+## Why the current setup is hard to explain
+
+Configuration grew in several directions at once:
+
+- The backend locates the runner with `AGENTA_RUNNER_INTERNAL_URL`.
+- The runner defaults a sandbox with `SANDBOX_AGENT_PROVIDER`.
+- Daytona uses ambient `DAYTONA_*` variables shared with code evaluation.
+- Agent-only overrides such as `DAYTONA_SNAPSHOT_AGENT` sit beside shared defaults.
+- `AGENTA_AGENT_SANDBOX_PI_INSTALLED` asks the operator to describe a runtime fact that the runner can detect.
+- `AGENTA_SESSION_HARNESS_MOUNTS` exposes an internal implementation choice as a deployment policy.
+- Self-managed Pi authentication causes the runner to discover its own Pi login and upload it into Daytona.
+- The API, Services, runner, and browser each carry pieces of provider availability.
+
+This makes a Compose file look like several partially overlapping products. An operator cannot tell which variables are read by the runner, which belong to the code evaluator, which describe policy, and which are only workarounds for one snapshot.
+
+## The live failure that exposed the boundary
+
+The local deployment on port 8280 had two runner-like services:
+
+- the normal Compose runner, which had the Daytona API key;
+- a subscription sidecar, which held local harness login state but had no Daytona API key.
+
+Services was pointed at the subscription sidecar. A request that selected Daytona reached that runner and failed with:
+
+~~~text
+Authentication credentials not found. Set DAYTONA_API_KEY, or both
+DAYTONA_JWT_TOKEN and DAYTONA_ORGANIZATION_ID.
+~~~
+
+Nothing was wrong with the user's Daytona account. The request reached a runner deployment that was not configured for Daytona. The setup made it easy to split capabilities accidentally.
+
+The target local QA deployment is one trusted development runner with both providers enabled and explicit read-write local subscription mounts. This is a development convenience, not a production multi-tenant topology. A local harness shares the runner container and can inspect other same-user processes through `/proc`.
+
+## Trust boundaries
+
+### Local provider
+
+The daemon and harness are child processes inside the runner container. The working directory is a starting directory, not a confinement boundary. The harness can read files and environment visible to the container user.
+
+Consequences:
+
+- The runner container must receive only secrets that a local harness is allowed to reach.
+- Helm must not inject database, auth, cryptographic, or bucket-wide store credentials into the runner.
+- Subscription files mounted into the runner are available to trusted local agents. This is opt-in and single-tenant.
+
+### Daytona provider
+
+The runner stays in the deployment. The daemon and harness execute in a Daytona sandbox. Only material the runner explicitly sends should cross that boundary:
+
+- the resolved model credential for the run;
+- workspace and instruction files;
+- scoped durable-mount credentials;
+- explicitly configured runtime files;
+- runner-owned support binaries and configuration.
+
+The runner's Daytona organization credential provisions the sandbox. It is infrastructure control-plane material and must never enter the sandbox environment.
+
+## Product principles
+
+### Configuration describes intent
+
+Operators configure facts they own: enabled providers, a default, credentials, lifecycle values, and mounted subscription inputs. The runner detects runtime facts such as whether Pi is already installed.
+
+### One owner per value
+
+A Daytona API key used to provision agent sandboxes belongs to the runner. A Daytona key used by code evaluation belongs to the code evaluator. A shared ambient variable is not a simplification when the two consumers have different images, snapshots, permissions, and lifecycles.
+
+### Explicit data movement
+
+The runner should never infer that a missing model API key means "copy whatever login I happen to have." Runtime authentication is a declared mode, and subscription state is a declared deployment mount.
+
+### Fail before side effects
+
+Unknown providers, disabled providers, missing provider credentials, and contradictory defaults fail at startup or before sandbox creation. There is no silent provider fallback. Mount behavior stays best-effort in version 1, with a structured warning on durable-to-ephemeral degradation (the fail-loud contract is RSH-11).
+
+### Pre-production cleanup
+
+These configuration surfaces are not a stable public API. The implementation should remove obsolete variables directly instead of carrying aliases, deprecation branches, or insecure compatibility modes.