ci(integration-test): promote qp-topic into the required suite - #1235
Merged
Conversation
The qp-topic scenario drives the shared qp-topic-measure composite — the post-label half of the paid labeling run — over canned inputs, token-free and credential-free, so requiring it costs a promotion nothing and closes the gate's blind spot on the labeling path. Four synchronized sites move together (a workflow-shape test pins each coupling): REQUIRED_SCENARIOS in the gate script, the `all` matrix in integration-test.yml, the dispatch-commands help list promote.yml prints on a freshness failure, and the docs. The required set goes 8 -> 9; the `all` matrix goes 7 -> 8 legs, with collect still riding the run as its own environment-free job. Also retires drift the change makes visible: docs/testing.md still called collect outside the gate and gave a required-run count of seven, both stale since collect joined the set, and the qp-topic leg's own rationale comment argued from its non-required status. The comment's justification inverts — the environment binding the leg never reads is exactly what makes its run title's `@ staging` suffix a deployment-gated provenance claim, which is how freshness reads every required matrix leg. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Aug 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Artifact served
The promotion gate's coverage of the qp-topic labeling path.
qp-topicexercises the sharedqp-topic-measurecomposite — the post-label half of the paid labeling run: the no-output guard, thefedcourts qp-topicspublication gate, and the publish-and-validate path. Until now that path could regress and still promote. The scenario spends no tokens and binds no credentials, so requiring it costs a promotion nothing.The scenario has a string of green live runs behind it — a standalone
qp-topicrun and several whole-suitescenario=allruns — which is the evidence its introduction deliberately deferred the gate promotion for.What changed
Four synchronized sites, each pinned by an existing workflow-shape test (
tests/test_workflow_promote.pyfails on any partial update):scripts/promotion-gate.sh—qp-topicadded toREQUIRED_SCENARIOS(8 → 9 entries)..github/workflows/integration-test.yml—{"scenario": "qp-topic", "engine": "claude-code"}added to theallmatrix (7 → 8 legs;collectstill rides the run as its own environment-free job, not a leg). Thescenariodispatch input already listedqp-topic..github/workflows/promote.yml—qp-topicadded to the per-scenario dispatch commands the freshness failure prints.docs/testing.md,docs/pipeline.md— the "outside the required suite" qualifiers dropped and the counts corrected.New true counts: 9 required runs = 8
allmatrix legs + the collect job, with engine-smoke counted once per each of three engines.Freshness matches the new leg without further change: for an engine-less entry the gate builds
^integration-test: qp-topic /and requires the@ staging$suffix, which the leg's titleintegration-test: qp-topic / claude-code @ stagingsatisfies.Pre-existing drift retired in the same paragraphs
docs/testing.mdstill said "The collect scenario is outside the gate" and gave the required-run count as "seven … five of the six real scenarios". Both were already stale against the pre-diff 8-entryREQUIRED_SCENARIOS—collectjoined the suite earlier. The sentence is dropped (its still-true half, that collect binds no environment and so no deployment-branch policy can refuse it, already lives in the scenario inventory above it) and the count rewritten.integration-test.ymlargued from its non-required status — a premise this commit deletes. The justification inverts rather than merely losing a word: the environment binding the leg never reads is exactly what makes its run title's@ stagingsuffix a deployment-gated provenance claim, which is how freshness reads every required matrix leg. A collect-style environment-free job would have weakened the evidence.Checks run
Python surface untouched (docs, workflows, one shell line), so the gate stages were scoped accordingly:
uv run pytest tests/test_workflow_promote.py tests/test_workflow_collect.py tests/test_workflow_cell_invariants.py— 50 passeduvx zizmor@1.26.1 --persona=regular .github/workflows .github/actions— no findingsactionlint1.7.12 withshellcheck -e SC2016— cleanshellcheck -e SC2016 scripts/promotion-gate.sh— cleanReview
workflow-revieweranddocs-reviewerboth ran on the diff. No blockers from either. Both raised the same recommended finding — the now-contradictory qp-topic rationale comment — which is fixed above usingworkflow-reviewer's stronger framing.docs-reviewer's nits (the "every scenario" phrasing, which would have read as including thealloption, and ragged reflow at the edit seams) andworkflow-reviewer's nit on the environment parenthetical indocs/pipeline.mdare all applied.workflow-reviewerindependently confirmed againstgh run listthat aqp-topic / claude-code @ stagingrun has concluded success from the staging branch, and that the last greenallpredating this change cannot leak into the gate — freshness pins onhead_sha, so a run at the promoted head executes the 8-leg matrix at that head.Closes #1210 — note this does not fire from a
stagingmerge; the orchestrator closes the issue by hand once this lands.