Fix CREATE TYPE race that can crash the worker on first boot - #436
Fix CREATE TYPE race that can crash the worker on first boot#436vedarolap wants to merge 3 commits into
Conversation
On a fresh Postgres volume, the worker and UI API both call metadata.create_all() at startup. SQLAlchemy's enum creation is check-then-create rather than atomic, so both processes can see the job_status enum as missing and both issue CREATE TYPE, crashing the loser with a UniqueViolation. Serialize schema creation behind a Postgres advisory lock so only one process creates the schema at a time. Fixes roostorg#432
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughPostgreSQL schema creation is serialized with an advisory lock, wired into storage initialization, and covered by a concurrent fresh-database regression test. The changelog documents the ChangesPostgreSQL schema initialization
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@osprey_worker/src/osprey/worker/lib/storage/tests/test_postgres.py`:
- Around line 49-55: After each timed join in the thread coordination block,
explicitly assert that the thread is no longer alive using thread.is_alive().
Keep the existing errors assertion, so timeout failures are reported immediately
while normal thread exceptions remain covered.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 8886eed6-4471-4b14-8fbb-6de60c340b4f
📒 Files selected for processing (3)
CHANGELOG.mdosprey_worker/src/osprey/worker/lib/storage/postgres.pyosprey_worker/src/osprey/worker/lib/storage/tests/test_postgres.py
| threads = [threading.Thread(target=_create_schema) for _ in range(2)] | ||
| for thread in threads: | ||
| thread.start() | ||
| for thread in threads: | ||
| thread.join(timeout=10) | ||
|
|
||
| assert not errors, errors |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Fail the test explicitly if a thread times out.
In Python, thread.join(timeout=10) returns silently without raising an exception when the timeout expires. If this concurrency test were to deadlock, the threads would hang, join would return silently, and errors would remain empty. The test would pass the assert not errors check and then fail confusingly during the drop_database teardown (because the hung threads hold active connections).
Checking thread.is_alive() after joining ensures that deadlocks are reported clearly as test failures rather than teardown errors.
🐛 Proposed fix to handle thread timeouts
threads = [threading.Thread(target=_create_schema) for _ in range(2)]
for thread in threads:
thread.start()
for thread in threads:
thread.join(timeout=10)
+ if thread.is_alive():
+ raise TimeoutError("Schema creation thread timed out and may be deadlocked")
assert not errors, errors📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| threads = [threading.Thread(target=_create_schema) for _ in range(2)] | |
| for thread in threads: | |
| thread.start() | |
| for thread in threads: | |
| thread.join(timeout=10) | |
| assert not errors, errors | |
| threads = [threading.Thread(target=_create_schema) for _ in range(2)] | |
| for thread in threads: | |
| thread.start() | |
| for thread in threads: | |
| thread.join(timeout=10) | |
| if thread.is_alive(): | |
| raise TimeoutError("Schema creation thread timed out and may be deadlocked") | |
| assert not errors, errors |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@osprey_worker/src/osprey/worker/lib/storage/tests/test_postgres.py` around
lines 49 - 55, After each timed join in the thread coordination block,
explicitly assert that the thread is no longer alive using thread.is_alive().
Keep the existing errors assertion, so timeout failures are reported immediately
while normal thread exceptions remain covered.
2 participants
Summary
On a fresh Postgres volume,
osprey-workerand the UI API both callmetadata.create_all()at startup. SQLAlchemy's enum creation is check-then-create rather than atomic, so on a fresh volume both processes can see thejob_statusenum as missing and both issueCREATE TYPE, crashing the loser with:This is exactly the newcomer path, since
demo.shremoves old volumes on every run.postgres.create_schema(engine)and wrapped it in a Postgres advisory lock (pg_advisory_lock/pg_advisory_unlock), so only one process creates the schema at a time. By the time a second process acquires the lock,create_all's own existence checks make it a no-op.create_schema()from two threads at the same time via a barrier, to simulate the two-process race. Verified this test fails reliably (5/5 runs) against the pre-fix code path and passes reliably (5/5 runs) with the fix.Closes #432
Test plan
test_create_schema_survives_concurrent_callers_on_a_fresh_database) added inosprey_worker/src/osprey/worker/lib/storage/tests/test_postgres.pyruff check/ruff format --checkpass on changed filesmypypasses on changed filestest_bulk_action_task.py, etc.) still pass against a local Postgres instanceSummary by CodeRabbit