Skip to content

Commit 544059e

Browse files
traskCopilot
andauthored
Retire legacy pull request dashboard execution (#467)
* Retire legacy pull request dashboard execution Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e92d9406-b51d-4058-b607-9d4cf2d137b9 * Clarify legacy cleanup does not require pausing owned workers Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e92d9406-b51d-4058-b607-9d4cf2d137b9 --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e92d9406-b51d-4058-b607-9d4cf2d137b9
1 parent 6993106 commit 544059e

35 files changed

Lines changed: 681 additions & 3648 deletions

‎.github/CODEOWNERS‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,6 @@ github-actions-queue/** @open-telemetry/shared-workflows-approvers @trask
2727
pull-request-dashboard/** @open-telemetry/shared-workflows-approvers @trask
2828
.github/scripts/pull-request-dashboard/** @open-telemetry/shared-workflows-approvers @trask
2929
.github/workflows/pull-request-dashboard.yml @open-telemetry/shared-workflows-approvers @trask
30-
.github/workflows/pull-request-dashboard-repo.yml @open-telemetry/shared-workflows-approvers @trask
3130
.github/workflows/pull-request-dashboard-deploy-webhook.yml @open-telemetry/shared-workflows-approvers @trask
3231

3332
# Survey on Merged PR

‎.github/renovate.json5‎

Lines changed: 0 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -24,14 +24,6 @@
2424
"schedule": [
2525
"before 8am on Tuesday"
2626
]
27-
},
28-
{
29-
// This self-reference is the stable PR dashboard rollout control.
30-
// Promote it only after the matching release has soaked on canaries.
31-
"matchManagers": ["github-actions"],
32-
"matchPackageNames": ["open-telemetry/shared-workflows"],
33-
"matchFileNames": [".github/workflows/pull-request-dashboard.yml"],
34-
"enabled": false
3527
}
3628
],
3729
"labels": [

‎.github/scripts/pull-request-dashboard/CONTEXT.md‎

Lines changed: 11 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -61,10 +61,12 @@ delivery receipts retain their existing Git compare-and-swap transactions.
6161

6262
## Execution ownership
6363

64-
`PR_DASHBOARD_EXECUTION_MODE` selects `legacy`, `paused`, or `owned`. An unset
65-
variable means `legacy`; deploying this code does not opt production into a
66-
new execution path. GitHub Actions and the Netlify bridge must agree on the
67-
mode. `WEBHOOK_SETUP.md` describes the cutover and rollback.
64+
`PR_DASHBOARD_EXECUTION_MODE` must explicitly select `paused` or `owned`.
65+
The Netlify bridge rejects missing or invalid values. Its deployment override
66+
can keep the bridge paused independently of GitHub's repository setting.
67+
`WEBHOOK_SETUP.md` describes deployment and the pause procedure for incompatible
68+
ownership changes or manual maintenance. Removing retired legacy execution
69+
does not require pausing compatible owned workers.
6870

6971
In `owned` mode, webhooks, manual PR and head-SHA refreshes, hourly and manual
7072
backfills, and author-reminder write sweeps enter the existing Blob queue.
@@ -109,8 +111,8 @@ Release retries are idempotent and cannot release a newer lease generation.
109111
`execution_code.py` loads scripts from immutable commit archives into temporary
110112
directories without replacing the worker checkout. Each ref has its own Python
111113
environment and runtime dependencies. Canaries use the drain's triggering
112-
commit on main. Stable code comes from the existing promoted pins in
113-
`pull-request-dashboard.yml`; `promote_rollout.py` remains the promotion
114+
commit on main. Stable code comes from the immutable SHA and release in
115+
`stable-release.json`; `promote_rollout.py` remains the promotion
114116
authority. Repository policy comes from the current worker checkout in both
115117
channels. Stable repositories do not run main's evaluation or state codecs.
116118

@@ -150,10 +152,9 @@ but cannot eliminate that external-success/receipt-loss window.
150152
`paused` accepts work but prevents activation, claims, and successor dispatch.
151153
Already claimed work can finish under its existing heartbeat. Recovery logs each
152154
lane's queue counts, retries, dead letters, pending backfills and sweeps, dispatcher
153-
expiry, and the oldest unfinished timestamp. In `owned` mode the watchdog
154-
only monitors webhook deployment, not dashboard processing. Legacy publisher
155-
jobs and matching remain gated for migration and rollback; they must not run
156-
alongside queue-owned writers.
155+
expiry, and the oldest unfinished timestamp. The watchdog only monitors webhook
156+
deployment, not dashboard processing. All production dashboard writers use
157+
queue ownership; direct-dispatch execution has no supported rollback path.
157158

158159
Other CLI writers are maintenance tools, not independent production workers.
159160
Before invoking `dashboard.py`, `delivery.py`, `publish_dashboard.py`, or a

‎.github/scripts/pull-request-dashboard/RATIONALE.md‎

Lines changed: 34 additions & 58 deletions
Original file line numberDiff line numberDiff line change
@@ -4,20 +4,16 @@ This dashboard is a maintainer aid, not a transactional notification system.
44
Some rare timing and notification edge cases are intentionally accepted to keep
55
the implementation understandable and operationally cheap.
66

7-
## Central Workflow
7+
## Central workflow
88

99
- Scheduled backfills run from `open-telemetry/shared-workflows` instead of
1010
each target repository hosting its own workflow.
1111
- Target repositories only need GitHub App access and an entry in
1212
`repositories.json`.
13-
- The top-level workflow resolves target repositories, then calls a reusable
14-
per-repository workflow for each target. The per-repository workflow separates
15-
read-only calculation from a serialized per-repository publisher, so one
16-
repository's update failure does not block delivery for repositories whose
17-
updates succeeded.
18-
- The top-level repository matrix runs one repository at a time. Backfills do
19-
not benefit enough from cross-repository parallelism to justify extra
20-
aggregate API and LLM demand.
13+
- The top-level workflow validates targets and enqueues work. Independent live
14+
and maintenance drains process up to four repositories concurrently; each
15+
repository remains serial within a wave. A failure in one repository does
16+
not prevent the others from processing.
2117
- Calculated state and delivery intent for each target repository live on a
2218
worker-owned branch under
2319
`otelbot/pull-request-dashboard-state/<repository>`, with files still
@@ -26,7 +22,7 @@ the implementation understandable and operationally cheap.
2622
`otelbot/pull-request-dashboard-delivery/<repository>`. Publishers read a
2723
detached snapshot of accepted worker state and never push to the worker
2824
branch.
29-
- The publishing job renders markdown from accepted dashboard state and the
25+
- The combined processor renders markdown from accepted dashboard state and the
3026
target repository's current open PR list.
3127
- The dashboard issue is discovered dynamically by title and label, so target
3228
repositories do not need to store issue numbers in config.
@@ -36,32 +32,26 @@ the implementation understandable and operationally cheap.
3632
contribution comes from, so without this the CI columns of those PRs would
3733
only refresh on the hourly backfill. The webhook bridge cannot resolve the
3834
commit itself, because its GitHub App is installed only on `shared-workflows`.
39-
- The central workflow, canary queue worker, and stable queue dispatcher
35+
- Both channels' processors
4036
paginate each target repository's open PRs and compare head SHAs. The
4137
commit-to-PR endpoint can omit fork heads or other PRs sharing the commit.
42-
Each matching open PR is refreshed or dispatched before a head-SHA claim is
38+
Each matching open PR is refreshed before a head-SHA claim is
4339
acknowledged; a head with no open PR needs no refresh.
4440

45-
## Staged Rollout
41+
## Staged rollout
4642

4743
- Dashboard changes reach repositories in two stages. Canary repositories run
48-
the workflow and its scripts from the commit that triggered the run; every
49-
other repository runs them from a promoted release commit, hash pinned with
50-
the release tag as a comment. A regression therefore shows up on a small set
44+
processor scripts from the immutable commit that triggered the drain; every
45+
other repository runs them from a promoted release commit. `stable-release.json`
46+
records its SHA and release. A regression therefore shows up on a small set
5147
of repositories before it reaches the whole fleet.
5248
- Staging the code rather than gating individual changes behind feature flags
5349
also covers the changes nobody thought were risky, which is where regressions
5450
actually come from. Feature flags remain available for a single genuinely
5551
risky behavior.
56-
- `jobs.<id>.uses` cannot take an expression, so the channel cannot come from a
57-
matrix value. Each entry path instead has one job per channel. A job-level
58-
`if` cannot read `env` either, so the targeted canary job repeats the canary
59-
list inline and the targeted stable job reads that job's skip rather than
60-
repeating it again; `test_rollout.py` keeps the copy in sync.
61-
- The reusable workflow loads the dashboard scripts through GitHub's
62-
self-repository action syntax. The action resolves from the same commit as
63-
the reusable workflow, so pinned stable jobs cannot mix released workflow
64-
YAML with scripts from the commit that triggered the run.
52+
- The drain loads an immutable archive and a separate dependency environment
53+
per code ref. It never substitutes main's evaluation or state codecs for
54+
stable code, and each target repository gets an isolated Git clone.
6555
- Repository configuration is deliberately not staged. `repositories.json` is
6656
always read from the commit that triggered the run, so opting a repository in
6757
or changing its settings takes effect immediately in both channels. The cost
@@ -70,14 +60,13 @@ the implementation understandable and operationally cheap.
7060
- Promotion is a pull request that pins the ref, rather than a moving tag or
7161
branch. A manually dispatched workflow prepares that pull request only after
7262
an operator names the release that has soaked on canaries. The version that
73-
each repository runs is then visible in the workflow file, a rollback is a
63+
each repository runs is then visible in the release configuration, a code rollback is a
7464
revert, and the reference stays hash pinned the way every other action
7565
reference in this repository is.
76-
- Queue drains are coordinators for both rollout channels. They process canary
77-
claims with code from the default branch and hand each coalesced stable claim
78-
to the targeted workflow, which invokes the promoted repository workflow.
79-
The drain acknowledges stable work only after GitHub accepts that dispatch,
80-
so no worker needs to read or migrate another release's dashboard state.
66+
- Queue drains process both channels through calculation, accepted-state
67+
persistence, delivery, and publication. Neither channel acknowledges a
68+
dispatch as completed work. Accepted state and delivery receipts retain their
69+
existing formats and version guards.
8170
- A rollback stops a bad change rather than restoring delivery. Downgraded code
8271
reads newer delivery state as empty, so delivering from it would repeat
8372
reminders and re-review requests already sent; the delivery version check
@@ -87,7 +76,7 @@ the implementation understandable and operationally cheap.
8776
versions use `open_thread` for any unresolved thread and do not persist the
8877
pending action needed to reconstruct the separate author-action state.
8978

90-
## Queue and Workflow Concurrency
79+
## Queue and workflow concurrency
9180

9281
- In owned execution, live PR/head refreshes and maintenance backfills/reminder
9382
sweeps have independent dispatcher leases and runners. A maintenance batch
@@ -97,53 +86,40 @@ the implementation understandable and operationally cheap.
9786
ownership during classification. State compare-and-swap and PR-slot acceptance
9887
still reconcile overlapping calculations; dispatcher heartbeats renew delivery
9988
ownership and stale lease generations cannot release a replacement's lease.
100-
- In legacy `canary` and `all` queue modes, the Netlify bridge coalesces webhook
101-
refreshes by target repository and PR before it starts GitHub Actions. One
102-
singleton drain claims a bounded batch. An event that arrives during
89+
- The Netlify bridge coalesces webhook
90+
refreshes by target repository and PR before it starts GitHub Actions. Each
91+
lane's drain claims a bounded batch. An event that arrives during
10392
processing marks the item dirty and schedules one follow-up pass.
104-
- Direct refreshes in `off` mode, and non-canary refreshes in `canary` mode, use
105-
GitHub Actions concurrency groups by target repository and PR. GitHub Actions
106-
keeps at most one running and one pending run in each group; a newer pending
107-
run replaces the older pending run without canceling the run already in
108-
progress.
10993
- Coalescing is safe because each refresh loads current PR state from GitHub.
11094
Intermediate states can go unobserved, but the surviving run reflects the
11195
state that exists when it executes.
11296
- Submitted reviews can coalesce with generic PR refreshes because the live PR
11397
status comment is rendered from current accepted dashboard state rather than
114-
a review-specific event. Manual runs remain separate because they can refresh
115-
large repositories that webhook-driven runs intentionally skip.
116-
- Actions concurrency bounds pending direct jobs per target; it does not
117-
debounce direct workflow dispatch. Every accepted direct webhook still
118-
creates a workflow run. Queue modes avoid that dispatch pressure for queued
119-
repositories by coalescing events in Netlify first.
120-
- Publishers use one concurrency group per target repository. GitHub preserves
121-
the running publisher but may replace an older pending publisher with a newer
122-
one even when `cancel-in-progress` is false. Each backfill transaction records
98+
a review-specific event. Backfills use separate maintenance capacity because
99+
they refresh a repository rather than one pull request.
100+
- Events coalesce in Netlify before a drain dispatch. Enqueue workflow runs use
101+
independent concurrency groups so pending runs cannot replace accepted requests.
102+
- Delivery leases serialize side effects per repository. Each backfill transaction records
123103
a monotonic full-publish generation on the worker branch. A targeted
124104
publisher normally limits status-comment and Slack delivery to its PR, but
125105
drains repository-wide work when it observes a generation without a delivery
126106
receipt. The publisher records the receipt only after full delivery and the
127-
dashboard issue both succeed. The queue drain follows the same rule.
107+
dashboard issue both succeed.
128108
A full queue publication satisfies the batch's pending issue update; another
129109
issue publication is needed only after a later active delivery.
130110
Webhooks without an outstanding full-publish generation remain targeted,
131111
so unrelated PR events do not repeatedly fan out. Status-comment rollout
132112
remains bounded to 50 per untargeted delivery; a webhook publisher puts its
133113
triggering PR first in that batch.
134-
- Hourly failure reporting uses workflow results rather than outstanding
135-
generations. A canceled publisher can leave work for a replacement that is
136-
still pending or running, so cancellation alone does not trigger a failure
137-
issue. Delivery, issue publication, and receipt errors fail the publishing
138-
job; genuine matrix failures take precedence over cancellation. Subsequent
139-
publishers retry outstanding generations, including on hourly runs. Pending
140-
generations have no separate age-based alert.
114+
- Recovery reports retries, dead letters, queue age, and ownership per lane.
115+
The stalled-run watchdog watches only webhook deployment. Delivery, issue
116+
publication, and receipt errors retry the combined item.
141117

142118
## GitHub Actions Instead Of Netlify For Scheduled Backfills
143119

144120
- Scheduled backfills are batch jobs: they read repository PR lists, call REST
145121
and GraphQL, run Copilot classification, and update git-backed state. A
146-
follow-up publishing job renders and publishes the issue body from accepted
122+
combined processor renders and publishes the issue body from accepted
147123
state.
148124
- GitHub Actions provides clearer logs, concurrency controls, artifacts, and
149125
normal retry/cancel behavior for that workload.

0 commit comments

Comments
 (0)