[5566] fix(ci): Fail the Railway preview bootstrap when a service is missing - #5588
Conversation
…#5566) A transient Railway API failure during 'railway add' was silently swallowed, so the setup job went green while the environment was permanently missing a service, and every later deploy failed deterministically. Bootstrap now verifies the environment's actual service list after creation, retries anything missing (RAILWAY_SERVICE_VERIFY_ATTEMPTS/_DELAY), and exits non-zero naming the missing services so a plain re-run repairs the environment. The same silent swallow in ensure_volume is now a hard error, and configure.sh tells the user that 'Re-run all jobs' is the fix when it hits a missing service. Claude-Session: https://claude.ai/code/session_01Qitvc9dCiunLishsJYRzbp
dosubot
Bot
added
the
size:L
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
@coderabbitai review |
📝 WalkthroughSummary by CodeRabbit
WalkthroughRailway bootstrap now tracks and verifies expected services, retries missing service creation, and fails on volume errors. Volume setup runs only after service verification. Configure diagnostics now instruct operators to rerun all jobs when a service remains missing. ChangesRailway bootstrap recovery
Estimated code review effort: 3 (Moderate) | ~20 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
✅ Action performedReview finished.
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: edac381c-e3eb-4e90-a7cc-d4e266de2445
📒 Files selected for processing (2)
hosting/railway/oss/scripts/bootstrap.shhosting/railway/oss/scripts/configure.sh
| if printf '%s' "$output" | grep -qi "not found"; then | ||
| printf "Service '%s' is missing from the Railway environment — bootstrap likely failed to create it.\n" "$service" >&2 | ||
| printf "Fix: use 'Re-run all jobs' (re-running only failed jobs skips the green setup job, so bootstrap never gets a chance to recreate the missing service).\n" >&2 | ||
| fi |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Do not report success after this missing-service error.
_cli_set_vars returns 1 here, but both fallback branches in upsert_service_vars discard that status and return 0 (Lines 219-220 and 229-231). A service can remain unconfigured while the job succeeds. Return the CLI helper’s status instead.
Proposed fix
- _cli_set_vars "$service" "$@"
- return 0
+ _cli_set_vars "$service" "$@"
+ return $?Apply this in both fallback branches.
…le-set failures (#5566) Railway throttles back-to-back volume adds ("Whoa there pal"), which railway_call's generic rate-limit matcher does not recognize. ensure_volume now re-checks the volume list before each of up to RAILWAY_VOLUME_ADD_ATTEMPTS attempts, so a half-succeeded add is never duplicated on the same mount path. upsert_service_vars now propagates the CLI helper's failure status instead of swallowing it, per CodeRabbit's review. Claude-Session: https://claude.ai/code/session_01Qitvc9dCiunLishsJYRzbp
|
good catch @coderabbitai — applied in both branches. |
|
Tip For best results, initiate chat on the files or code changes.
If you found this review helpful, would you consider giving us a shout-out on X? Thank you for using CodeRabbit! You are interacting with an AI system. |
1 participant
In the #5560 preview environment, bootstrap reported success even though the
cronservice was never created: a transient Railway API failure duringrailway addwas swallowed by>/dev/null 2>&1 || true, so the setup job went green while the environment was permanently missing a service. Every later deploy then failed deterministically with "Service 'cron' not found", and "Re-run failed jobs" could never repair it because the green setup job never re-runs.The fix
railway addcalls, bootstrap reads the environment's actual service list fromrailway status --jsonand re-attempts creation for anything missing. After 3 attempts (tunable viaRAILWAY_SERVICE_VERIFY_ATTEMPTS/RAILWAY_SERVICE_VERIFY_DELAY) it exits non-zero and names the missing services, so the setup job goes red and a plain re-run repairs the environment.ensure_volumehad the same silent-swallow pattern; a failed volume create is now a hard error.configure.shnow explains that "Re-run all jobs" is the fix when it hits a missing service, instead of leaving the reader with an unrecoverable-looking red deploy.What stays tolerant, and why
Service creation is still tolerant of "already exists", so re-bootstrapping a healthy environment remains a no-op. Environment create, domain create, and the read-side guards keep their existing tolerance too: they are either idempotent by nature or checked downstream, and hard-failing them would break the re-bootstrap path this fix is meant to protect.
Verification
bash -non both scripts.railwayfunctional test covering three scenarios: healthy re-bootstrap (exit 0), transient failure that self-heals on retry (exit 0), and persistent failure (exit 1 with the missing service named).Closes #5566
https://claude.ai/code/session_01Qitvc9dCiunLishsJYRzbp