(bug) Railway preview acceptance suites failing repo-wide (all branches, incl. release/*)

[mmabrouk]

The Railway preview chain (workflow 14) is currently failing on every branch that runs it, including branches with no application code changes.

Observed on 2026-08-02/03 (run start times UTC):

• release/v0.108.0 — failure (21:04)
• release/v0.107.1 — failure (20:54)
• feat/codex-harness — failure (20:54)
• feat/daytona-secrets-v2 — failure (22:40)
• wp1-wrapper-images / wp2-template-as-code / wp3-clone-preview-flow (docs/CI-only stacks for (chore) Railway previews: clone a template environment per PR instead of creating 13 services imperatively #5650) — failures across repeated fresh runs

Failure signatures vary by run, on identical app code:

• tests / run-web-tests (acceptance): 2 playground tests fail on a 20s predicate timeout ("saves changes as a new variant"), e.g. run 30772928164.
• tests / run-api-tests (acceptance): 45 tracing tests fail with ingest_traces → 500 "Failed to queue spans for processing", e.g. run 30767675393 (attempt 2).
• deploy / deploy: occasional deploy-phase failure, e.g. run 30773491965.

Because branches with zero app-code diffs fail with different signatures on different rolls, this looks like preview-environment instability (tracing queue path, deploy timing) rather than any single PR's regression. Related: #5671 (setup re-runs break on the already-mounted Postgres volume) and #5650 (the clone-based preview redesign that replaces this pipeline).

[linear-code]

AGE-4062

[mmabrouk]

Root-cause signature isolated (2026-08-03, serialized single-chain runs on the #5650 stack):

The preview API worker fails to boot: the redis Python client raises socket.gaierror: [Errno -2] Name or service not known during application startup (TargetServerAttributeNotMatched retry path), so gunicorn workers exit ("Worker failed to boot") and deploy fails after the retry budget (~16-18 min). Runs where workers partially boot instead show the ingest_traces 500s ("Failed to queue spans for processing"), the same redis path at request time.

So the whole #5673 failure zoo reduces to: the API cannot resolve its redis hostname inside legacy preview environments, consistently since the 2026-07-30..08-01 merge window. Suggested next step: pull the full startup traceback from a live preview API deployment to see the exact hostname being resolved, then diff it against what configure.sh sets for the api service (REDIS_URI* handling) and against the redis client config changes in the v0.107 wave.

Evidence runs: 30775055881 (deploy fail 18m, job 91569116027), 30775055065, 30775056158. Note the volume-add re-run failure (#5671) is now fixed in the #5650 stack, so it no longer masks this.

[mmabrouk]

Correction and completion of the root cause after reading the deploy job's embedded Railway service log tails (run 30775055881):

The socket.gaierror is asyncpg → Postgres, not redis. The Postgres service log ends with a clean shutdown ("terminating connection due to administrator command … database system is shut down") and never starts again: a cancel-in-progress interrupted a chain mid-Postgres-redeploy, leaving the service with no active deployment. Railway's internal DNS stops resolving a service with no active deployment, so alembic and the API then fail name resolution. The next run's "redeploy Postgres" step tolerates "No deployment found" (a fix meant for fresh projects) and silently accepts the dead Postgres, so no amount of re-running heals the project, the same partial-state pattern as #5671, one layer up.

Why repo-wide since ~2026-07-30: the sessions-storage merge window made API startup fail fast when the DB is unreachable, so any preview project that ever had a chain cancelled mid-deploy (every active PR does this on force-pushes) is now permanently broken until the project is deleted and recreated (close/reopen the PR).

Suggested durable fix for the legacy path: after the tolerated Postgres redeploy, verify the service has an ACTIVE deployment and recover (or fail loudly naming the cause) instead of proceeding into DNS failures. The clone-based redesign (#5650) is immune by construction: clones are fresh every time and deploys are batched.

[mmabrouk]

Final piece: with fresh projects and serialized chains, setup, deploy, API tests, and SDK tests all pass. The one remaining red job, tests / run-web-tests (acceptance), fails consistently on the v0.107 agent-platform feature tests: agent-skills/skill-folder-upload.spec.ts (the "Skills" UI text never becomes visible), prompt-registry/index.ts (no v1 revision label), and agent-chat-attach-send (attachments flow). These features' UI does not render in OSS preview environments, most plausibly because the agent feature flags are intentionally not enabled there, so the tests added in the 2026-07-30..08-01 wave cannot pass against previews by construction.

So #5673 decomposes into exactly two causes, both now proven:

1. Dead-Postgres corpse projects from chains cancelled mid-redeploy (heal: close/reopen; durable fix: verify an ACTIVE Postgres deployment after the tolerated redeploy). Fixed operationally; (bug) Legacy Railway preview setup fails on re-run: 'A volume is already mounted' treated as throttling #5671's volume fix removes the other re-run blocker.
2. v0.107 agent-feature acceptance tests that cannot pass in flag-off preview environments — needs a product/test decision: enable the flags for previews, tag these tests out of the preview selection, or accept the red until then.

[mmabrouk]

Page-snapshot evidence from the failure artifacts (run 30779592246) settles the web-suite question: the app is healthy in previews. In every failing spec the test's "create a new agent app" step silently produced a Prompt-type app (the snapshots show the prompt-app menu and no Skills section, e.g. "e2e-skill-upload… Prompt", "E2E Agent Prompt"), because agent-type apps are not available in flag-off OSS preview environments. The specs then assert agent-UI elements against a prompt app and time out.

So the fix is test hygiene, not product code: the agent-platform specs (agent-skills, agent-chat attachments, and the prompt-registry cases that bootstrap an "E2E Agent" app) should detect agent-type availability and skip cleanly when the environment does not provide it, which is standard flag-gated test behavior. That one change makes the preview suite meaningful again for every branch.