@@ -4,20 +4,16 @@ This dashboard is a maintainer aid, not a transactional notification system.
44Some rare timing and notification edge cases are intentionally accepted to keep
55the implementation understandable and operationally cheap.
66
7- ## Central Workflow
7+ ## Central workflow
88
99- Scheduled backfills run from ` open-telemetry/shared-workflows ` instead of
1010 each target repository hosting its own workflow.
1111- Target repositories only need GitHub App access and an entry in
1212 ` repositories.json ` .
13- - The top-level workflow resolves target repositories, then calls a reusable
14- per-repository workflow for each target. The per-repository workflow separates
15- read-only calculation from a serialized per-repository publisher, so one
16- repository's update failure does not block delivery for repositories whose
17- updates succeeded.
18- - The top-level repository matrix runs one repository at a time. Backfills do
19- not benefit enough from cross-repository parallelism to justify extra
20- aggregate API and LLM demand.
13+ - The top-level workflow validates targets and enqueues work. Independent live
14+ and maintenance drains process up to four repositories concurrently; each
15+ repository remains serial within a wave. A failure in one repository does
16+ not prevent the others from processing.
2117- Calculated state and delivery intent for each target repository live on a
2218 worker-owned branch under
2319 ` otelbot/pull-request-dashboard-state/<repository> ` , with files still
@@ -26,7 +22,7 @@ the implementation understandable and operationally cheap.
2622 ` otelbot/pull-request-dashboard-delivery/<repository> ` . Publishers read a
2723 detached snapshot of accepted worker state and never push to the worker
2824 branch.
29- - The publishing job renders markdown from accepted dashboard state and the
25+ - The combined processor renders markdown from accepted dashboard state and the
3026 target repository's current open PR list.
3127- The dashboard issue is discovered dynamically by title and label, so target
3228 repositories do not need to store issue numbers in config.
@@ -36,32 +32,26 @@ the implementation understandable and operationally cheap.
3632 contribution comes from, so without this the CI columns of those PRs would
3733 only refresh on the hourly backfill. The webhook bridge cannot resolve the
3834 commit itself, because its GitHub App is installed only on ` shared-workflows ` .
39- - The central workflow, canary queue worker, and stable queue dispatcher
35+ - Both channels' processors
4036 paginate each target repository's open PRs and compare head SHAs. The
4137 commit-to-PR endpoint can omit fork heads or other PRs sharing the commit.
42- Each matching open PR is refreshed or dispatched before a head-SHA claim is
38+ Each matching open PR is refreshed before a head-SHA claim is
4339 acknowledged; a head with no open PR needs no refresh.
4440
45- ## Staged Rollout
41+ ## Staged rollout
4642
4743- Dashboard changes reach repositories in two stages. Canary repositories run
48- the workflow and its scripts from the commit that triggered the run ; every
49- other repository runs them from a promoted release commit, hash pinned with
50- the release tag as a comment . A regression therefore shows up on a small set
44+ processor scripts from the immutable commit that triggered the drain ; every
45+ other repository runs them from a promoted release commit. ` stable-release.json `
46+ records its SHA and release . A regression therefore shows up on a small set
5147 of repositories before it reaches the whole fleet.
5248- Staging the code rather than gating individual changes behind feature flags
5349 also covers the changes nobody thought were risky, which is where regressions
5450 actually come from. Feature flags remain available for a single genuinely
5551 risky behavior.
56- - ` jobs.<id>.uses ` cannot take an expression, so the channel cannot come from a
57- matrix value. Each entry path instead has one job per channel. A job-level
58- ` if ` cannot read ` env ` either, so the targeted canary job repeats the canary
59- list inline and the targeted stable job reads that job's skip rather than
60- repeating it again; ` test_rollout.py ` keeps the copy in sync.
61- - The reusable workflow loads the dashboard scripts through GitHub's
62- self-repository action syntax. The action resolves from the same commit as
63- the reusable workflow, so pinned stable jobs cannot mix released workflow
64- YAML with scripts from the commit that triggered the run.
52+ - The drain loads an immutable archive and a separate dependency environment
53+ per code ref. It never substitutes main's evaluation or state codecs for
54+ stable code, and each target repository gets an isolated Git clone.
6555- Repository configuration is deliberately not staged. ` repositories.json ` is
6656 always read from the commit that triggered the run, so opting a repository in
6757 or changing its settings takes effect immediately in both channels. The cost
@@ -70,14 +60,13 @@ the implementation understandable and operationally cheap.
7060- Promotion is a pull request that pins the ref, rather than a moving tag or
7161 branch. A manually dispatched workflow prepares that pull request only after
7262 an operator names the release that has soaked on canaries. The version that
73- each repository runs is then visible in the workflow file , a rollback is a
63+ each repository runs is then visible in the release configuration , a code rollback is a
7464 revert, and the reference stays hash pinned the way every other action
7565 reference in this repository is.
76- - Queue drains are coordinators for both rollout channels. They process canary
77- claims with code from the default branch and hand each coalesced stable claim
78- to the targeted workflow, which invokes the promoted repository workflow.
79- The drain acknowledges stable work only after GitHub accepts that dispatch,
80- so no worker needs to read or migrate another release's dashboard state.
66+ - Queue drains process both channels through calculation, accepted-state
67+ persistence, delivery, and publication. Neither channel acknowledges a
68+ dispatch as completed work. Accepted state and delivery receipts retain their
69+ existing formats and version guards.
8170- A rollback stops a bad change rather than restoring delivery. Downgraded code
8271 reads newer delivery state as empty, so delivering from it would repeat
8372 reminders and re-review requests already sent; the delivery version check
@@ -87,7 +76,7 @@ the implementation understandable and operationally cheap.
8776 versions use ` open_thread ` for any unresolved thread and do not persist the
8877 pending action needed to reconstruct the separate author-action state.
8978
90- ## Queue and Workflow Concurrency
79+ ## Queue and workflow concurrency
9180
9281- In owned execution, live PR/head refreshes and maintenance backfills/reminder
9382 sweeps have independent dispatcher leases and runners. A maintenance batch
@@ -97,53 +86,40 @@ the implementation understandable and operationally cheap.
9786 ownership during classification. State compare-and-swap and PR-slot acceptance
9887 still reconcile overlapping calculations; dispatcher heartbeats renew delivery
9988 ownership and stale lease generations cannot release a replacement's lease.
100- - In legacy ` canary ` and ` all ` queue modes, the Netlify bridge coalesces webhook
101- refreshes by target repository and PR before it starts GitHub Actions. One
102- singleton drain claims a bounded batch. An event that arrives during
89+ - The Netlify bridge coalesces webhook
90+ refreshes by target repository and PR before it starts GitHub Actions. Each
91+ lane's drain claims a bounded batch. An event that arrives during
10392 processing marks the item dirty and schedules one follow-up pass.
104- - Direct refreshes in ` off ` mode, and non-canary refreshes in ` canary ` mode, use
105- GitHub Actions concurrency groups by target repository and PR. GitHub Actions
106- keeps at most one running and one pending run in each group; a newer pending
107- run replaces the older pending run without canceling the run already in
108- progress.
10993- Coalescing is safe because each refresh loads current PR state from GitHub.
11094 Intermediate states can go unobserved, but the surviving run reflects the
11195 state that exists when it executes.
11296- Submitted reviews can coalesce with generic PR refreshes because the live PR
11397 status comment is rendered from current accepted dashboard state rather than
114- a review-specific event. Manual runs remain separate because they can refresh
115- large repositories that webhook-driven runs intentionally skip.
116- - Actions concurrency bounds pending direct jobs per target; it does not
117- debounce direct workflow dispatch. Every accepted direct webhook still
118- creates a workflow run. Queue modes avoid that dispatch pressure for queued
119- repositories by coalescing events in Netlify first.
120- - Publishers use one concurrency group per target repository. GitHub preserves
121- the running publisher but may replace an older pending publisher with a newer
122- one even when ` cancel-in-progress ` is false. Each backfill transaction records
98+ a review-specific event. Backfills use separate maintenance capacity because
99+ they refresh a repository rather than one pull request.
100+ - Events coalesce in Netlify before a drain dispatch. Enqueue workflow runs use
101+ independent concurrency groups so pending runs cannot replace accepted requests.
102+ - Delivery leases serialize side effects per repository. Each backfill transaction records
123103 a monotonic full-publish generation on the worker branch. A targeted
124104 publisher normally limits status-comment and Slack delivery to its PR, but
125105 drains repository-wide work when it observes a generation without a delivery
126106 receipt. The publisher records the receipt only after full delivery and the
127- dashboard issue both succeed. The queue drain follows the same rule.
107+ dashboard issue both succeed.
128108 A full queue publication satisfies the batch's pending issue update; another
129109 issue publication is needed only after a later active delivery.
130110 Webhooks without an outstanding full-publish generation remain targeted,
131111 so unrelated PR events do not repeatedly fan out. Status-comment rollout
132112 remains bounded to 50 per untargeted delivery; a webhook publisher puts its
133113 triggering PR first in that batch.
134- - Hourly failure reporting uses workflow results rather than outstanding
135- generations. A canceled publisher can leave work for a replacement that is
136- still pending or running, so cancellation alone does not trigger a failure
137- issue. Delivery, issue publication, and receipt errors fail the publishing
138- job; genuine matrix failures take precedence over cancellation. Subsequent
139- publishers retry outstanding generations, including on hourly runs. Pending
140- generations have no separate age-based alert.
114+ - Recovery reports retries, dead letters, queue age, and ownership per lane.
115+ The stalled-run watchdog watches only webhook deployment. Delivery, issue
116+ publication, and receipt errors retry the combined item.
141117
142118## GitHub Actions Instead Of Netlify For Scheduled Backfills
143119
144120- Scheduled backfills are batch jobs: they read repository PR lists, call REST
145121 and GraphQL, run Copilot classification, and update git-backed state. A
146- follow-up publishing job renders and publishes the issue body from accepted
122+ combined processor renders and publishes the issue body from accepted
147123 state.
148124- GitHub Actions provides clearer logs, concurrency controls, artifacts, and
149125 normal retry/cancel behavior for that workload.
0 commit comments