feat(ci): xdist backend test workflow#110776
Merged
mchen-sentry merged 11 commits intomasterfrom Mar 18, 2026
Merged
Conversation
github-actions
bot
added
the
Scope: Backend
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
Autofix Details
Bugbot Autofix prepared fixes for both issues found in the latest run.
- ✅ Fixed: Missing pipefail lets bootstrap failure go undetected
- Added pipefail option to set command (changed 'set -e' to 'set -eo pipefail') to ensure bootstrap failures in the pipeline are properly detected.
- ✅ Fixed: Experimental branch triggers will disable production CI
- Restored production CI triggers by replacing experimental branch names with 'master' and re-adding pull_request event triggers.
Or push these changes by commenting:
@cursor push 4594d2aaa4
Preview (4594d2aaa4)
diff --git a/.github/workflows/backend.yml b/.github/workflows/backend.yml
--- a/.github/workflows/backend.yml
+++ b/.github/workflows/backend.yml
@@ -3,8 +3,9 @@
on:
push:
branches:
- - mchen/tiered-xdist-v2
- - mchen/tiered-xdist-v2-clean
+ - master
+ pull_request:
+ types: [opened, synchronize, reopened, labeled]
workflow_dispatch:
# Cancel in progress workflows on pull_requests.
@@ -292,7 +293,7 @@
- name: Bootstrap per-worker Snuba instances
if: env.XDIST_PER_WORKER_SNUBA == '1'
run: |
- set -e
+ set -eo pipefail
XDIST_N=${XDIST_WORKERS:-3}
SNUBA_IMAGE=$(docker inspect snuba-snuba-1 --format '{{.Config.Image}}')
SNUBA_NETWORK=$(docker inspect snuba-snuba-1 --format '{{range $k, $v := .NetworkSettings.Networks}}{{$k}}{{end}}')This Bugbot Autofix run was free. To enable autofix for future PRs, go to the Cursor dashboard.
mchen-sentry
force-pushed
the
mingchen/di-1713-xdist-workflow
branch
2 times, most recently
from
8fd3571 to
37e0b03
Compare
mchen-sentry
force-pushed
the
mingchen/di-1713-xdist-workflow
branch
from
37e0b03 to
a089adf
Compare
Contributor
Backend Test FailuresFailures on
|
joshuarli
reviewed
Mar 16, 2026
joshuarli
approved these changes
Mar 16, 2026
Member
joshuarli
left a comment
joshuarli
left a comment
There was a problem hiding this comment.
it's a little messy (can't really help it tbh) but lgtm
![sentry[bot]](https://avatars.githubusercontent.com/in/12637?s=60&v=4){kind=small}
mchen-sentry
force-pushed
the
mingchen/di-1713-xdist-workflow
branch
3 times, most recently
from
a78ec33 to
fefad84
Compare
mchen-sentry
force-pushed
the
mingchen/di-1713-xdist-workflow
branch
from
c6b08ca to
5ba73e4
Compare
mchen-sentry
force-pushed
the
mingchen/di-1713-xdist-workflow
branch
from
5ba73e4 to
c36d35a
Compare
Enable pytest-xdist on master pushes and workflow_dispatch while keeping the existing single-process selective testing for pull requests. Changes: - Add XDIST_WORKERS, XDIST_PER_WORKER_SNUBA, PYTHONHASHSEED env vars gated on github.event_name != 'pull_request' - Bootstrap per-worker Snuba instances with proper PID tracking and health checks for parallel job error detection - Append xdist flags to PYTEST_ADDOPTS (preserving --reruns from setup-sentry) and run via make test-python-ci - Add 20-minute timeout wrapper to catch xdist hangs - Skip checkout/setup-sentry/shard-calculation when selective testing is not active (static 22-shard fast path) - Dump per-worker Snuba logs on failure for debugging
mchen-sentry
force-pushed
the
mingchen/di-1713-xdist-workflow
branch
from
c36d35a to
9336f89
Compare
- Rewrite Snuba bootstrap to use devservices compose network instead of --network host (which silently ignores -p port mappings) - Inspect running snuba-snuba-1 for image and network instead of pulling latest - Per-worker ClickHouse databases for full isolation - Fix docker stop to target snuba-snuba-1 (correct devservices name) - Change PYTHONHASHSEED from '1' to '0' for canonical deterministic hashing - Add PostgreSQL lock_timeout=30s under xdist to prevent indefinite lock waits
signal.signal() and signal.setitimer() only work in the main thread. Under xdist, execnet workers run test code in a non-main thread, causing ValueError. These tests still run on every PR (single-process).
setdefault() returns object from mypy's perspective; indexed assignment on object is not supported.
30s was too aggressive under xdist, causing retries that inflated shard durations.
30s was too aggressive under xdist, causing retries that inflated shard durations.
Ruff reformatted the line so the ignore ended up on the string literal instead of the actual indexed assignment, making mypy report both an error and an unused-ignore.
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
PYTHONHASHSEED="" is parsed by CPython's strtoul as 0, which disables hash randomization. Use 'random' to explicitly enable the default random hash seed on PR runs where xdist is disabled.
29 consecutive green runs on the stress-test branch; no longer need the soft-failure guard.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Enable pytest-xdist in the backend CI workflow with per-worker Snuba isolation. Depends on #DI-1712 (xdist infra) being merged first.
xdist is disabled for PRs, only affects merges in sentry.