Skip to content

(bug) A transient Railway API failure during preview bootstrap leaves the PR environment permanently broken #5566

Description

@mmabrouk

When the Railway preview bootstrap hits a transient Railway API failure while creating one of the services, it still reports "Bootstrap completed" and the setup job goes green. The preview environment is then permanently missing that service, and every subsequent deploy of the PR fails deterministically — making an infra flake look like a real failure in the PR's changes.

Concrete case (2026-07-30, PR #5560):

  1. Bootstrap ran for PR feat: turn the session-storage rework on by default #5560 and created project agenta-oss-pr-5560, but the cron service was never created. hosting/railway/oss/scripts/bootstrap.sh creates each service with add_service_image() (~line 110), which runs railway add ... >/dev/null 2>&1 || true, so the failed creation was invisible. Bootstrap then printed "Bootstrap completed" (~line 170) without checking that all expected services exist.
  2. Every deploy afterwards failed at the same spot: hosting/railway/oss/scripts/configure.sh, upsert_service_varsService 'cron' not found, exit 1. The PR's test suite never ran.
  3. "Re-run failed jobs" cannot repair it: the setup job succeeded, so it is never re-run, and bootstrap never gets another chance to create the missing service. Only "Re-run all jobs" or a new push re-executes bootstrap.

This cost two wasted reruns on PR #5560, which only flips feature-flag defaults and touches nothing under hosting/railway/ — the failure was pure infra.

Evidence: both failing jobs' logs are byte-identical — run 30489344489, jobs 90705250067 and 90835834824, both ending in Service 'cron' not found.

Fix plan:

  • Stop swallowing failures in add_service_image in bootstrap.sh (or at minimum log each failed railway add loudly instead of >/dev/null 2>&1 || true).
  • After bootstrap, verify the created service list matches the expected 13 services and fail the setup job with a clear message when any are missing.
  • Make the deploy job's "Service '' not found" error in configure.sh say that the environment is missing a service and that the fix is to re-run ALL jobs (not just failed ones) so bootstrap can repair it.

Metadata

Metadata

Assignees

Labels

Bug ReportSomething isn't workingciChanges to CI configuration files and scriptsdevops

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions