You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When the Railway preview bootstrap hits a transient Railway API failure while creating one of the services, it still reports "Bootstrap completed" and the setup job goes green. The preview environment is then permanently missing that service, and every subsequent deploy of the PR fails deterministically — making an infra flake look like a real failure in the PR's changes.
Bootstrap ran for PR feat: turn the session-storage rework on by default #5560 and created project agenta-oss-pr-5560, but the cron service was never created. hosting/railway/oss/scripts/bootstrap.sh creates each service with add_service_image() (~line 110), which runs railway add ... >/dev/null 2>&1 || true, so the failed creation was invisible. Bootstrap then printed "Bootstrap completed" (~line 170) without checking that all expected services exist.
Every deploy afterwards failed at the same spot: hosting/railway/oss/scripts/configure.sh, upsert_service_vars — Service 'cron' not found, exit 1. The PR's test suite never ran.
"Re-run failed jobs" cannot repair it: the setup job succeeded, so it is never re-run, and bootstrap never gets another chance to create the missing service. Only "Re-run all jobs" or a new push re-executes bootstrap.
This cost two wasted reruns on PR #5560, which only flips feature-flag defaults and touches nothing under hosting/railway/ — the failure was pure infra.
Evidence: both failing jobs' logs are byte-identical — run 30489344489, jobs 90705250067 and 90835834824, both ending in Service 'cron' not found.
Fix plan:
Stop swallowing failures in add_service_image in bootstrap.sh (or at minimum log each failed railway add loudly instead of >/dev/null 2>&1 || true).
After bootstrap, verify the created service list matches the expected 13 services and fail the setup job with a clear message when any are missing.
Make the deploy job's "Service '' not found" error in configure.sh say that the environment is missing a service and that the fix is to re-run ALL jobs (not just failed ones) so bootstrap can repair it.
When the Railway preview bootstrap hits a transient Railway API failure while creating one of the services, it still reports "Bootstrap completed" and the setup job goes green. The preview environment is then permanently missing that service, and every subsequent deploy of the PR fails deterministically — making an infra flake look like a real failure in the PR's changes.
Concrete case (2026-07-30, PR #5560):
agenta-oss-pr-5560, but thecronservice was never created.hosting/railway/oss/scripts/bootstrap.shcreates each service withadd_service_image()(~line 110), which runsrailway add ... >/dev/null 2>&1 || true, so the failed creation was invisible. Bootstrap then printed "Bootstrap completed" (~line 170) without checking that all expected services exist.hosting/railway/oss/scripts/configure.sh,upsert_service_vars—Service 'cron' not found, exit 1. The PR's test suite never ran.This cost two wasted reruns on PR #5560, which only flips feature-flag defaults and touches nothing under
hosting/railway/— the failure was pure infra.Evidence: both failing jobs' logs are byte-identical — run 30489344489, jobs 90705250067 and 90835834824, both ending in
Service 'cron' not found.Fix plan:
add_service_imageinbootstrap.sh(or at minimum log each failedrailway addloudly instead of>/dev/null 2>&1 || true).configure.shsay that the environment is missing a service and that the fix is to re-run ALL jobs (not just failed ones) so bootstrap can repair it.