chore: promote staging to main - #968
Merged
Merged
Conversation
Feature PRs reaching main around the promotion path was convention only. The main-base required check fails any PR to main whose head is neither staging nor a bot-owned lane (collect run branches, the cleanup sweep, the metrics-refresh and cert-backtest PRs); rulesets cannot constrain a PR's source branch, so the policy lives as a check, the paths pattern. Dependabot retargets to staging for the same reason. Docs gain the operator's walkthrough of the promotion flow and the required-check ordering caveat. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ci: jail merge routing to main behind the staging promotion path
… evaluator, not per (evaluator, predictor) The per-case derivation multiplied evaluate by "3 evaluators x 3 preds", but no (evaluator, predictor) cell exists: `matrix.evaluate_matrix` builds an evaluator x case x event matrix and one judge invocation grades every predictor for its event, writing one `evaluation.json` per predictor but a single `usage.json` per (judge, run_id). Three engines cross-evaluated is 6 cells per case, not 12, so the evaluate column ran ~3x high. Re-cut every figure that hangs off the cell count, and refresh the measured means from the 413-cell ledger: per case ~$13 at the held planning rate of $2.12/cell (~$11 at measured means), full cert gate ~$70K, 14-court ceiling ~$570K, provider table and the funding ladder recomputed. The planning rate stays deliberately above the $1.86 measured blended mean, and the evaluate means are labeled as one graded event. The corrected ladder puts full cert coverage inside the initial-funding step rather than beyond it, so salience-as-spend-control is now stated as a bootstrapping argument specifically. Also drops two stale claims — that no event has been evaluated, and that only the predict column is being spent — and clarifies the evaluate fan-out in the README trigger table, which read as (evaluator x predictor) and is the phrasing the bad derivation came from. Artifact: the salience gate's capacity N, whose sizing this math drives. Closes #859 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Each of these was learned from a failed or wasted run and was written down nowhere on a committed surface, so every new session rediscovered it: - an interactive session's token is refused on workflow_dispatch, so composing the command for the maintainer is the step, not dispatching - the deterministic writers commit to main with no gate, so a writer bug reddens the data stage on every open PR at once; check main first - there is one CourtListener account and one credential, so extra accounts or a second credential are not a throughput lever - a specific SCOTUS docket is investigable with no S3 and no token: the local corpus resolves docket_number, and the per-docket JSON is public Also adds a pointer (not a copy) in the workflow-traps list to the fresh-remote-tip branching requirement. The worked reasoning stays in collect-run where it is used; the trap is cross-cutting enough that a new workflow author pushing a branch built during a long matrix should meet it in the list they are already reading. Artifact: none directly — this serves the agents that produce every artifact, by making the next session start where the last one ended. Closes #861 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…l-grain docs(budget): correct the per-case cost model — evaluate fans out per evaluator, not per (evaluator, predictor)
docs: record four operational lessons that lived only in session memory
…ourt index `evaluate_backlog` opened by materializing every SCOTUS row into a dict and then filtering it down to the few thousand holding a resolved event. Peak memory was a function of the corpus — 589,794 rows today, and the row count is an asserted-monotonic invariant, so it only grows. Drive from the resolved-event set instead and fetch each candidate's row. Only a case with a resolved event can be owed a grading, so the inverted loop reads the same rows the filter would have kept and skips building the index nothing else used. Both iteration orders are case_id-ascending and the candidates are re-sorted before the cap applies, so the queue is unchanged. Measured against the real corpus: completes at 52 MB peak RSS, where the previous shape was OOM-killed twice in a container with ~3 GB available. Wall time is dominated by the resolved-events query and is unchanged in character. Production was surviving this, so the fix removes a latent risk rather than a live outage — but the failure mode was a silent kill mid-step. Two tests pin the behavior a fixture-sized suite cannot otherwise see: `iter_rows` is made fatal, so a future whole-court walk fails loudly rather than regressing quietly, and a resolved event whose `cases` row is absent stays a skip now that a row lookup replaced a dict membership test. Artifact: the evaluate cell fan-out — this is the seam that guarantees no committed prediction is left ungraded. Closes #867 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…he split Under the corpus-split mode `cases.opinion_text` is NULL — the body lives in the per-case content store — so `query --full` emitted an empty body instead of reading the store. Hydrate in `corpus.prior_payload`, the single shaper both the CLI's local/ranged path and the query service's handler call, so the two stay byte-identical and a credential-free cell gets a body from the credentialed sidecar with no client change. The read goes through the existing `PayloadReadSource` protocol, keeping the inversion that corpus never imports casestore, and adds `casestore.read_opinion_text`. Gated narrowly: only when `full` is asked for, only when the retained `has_opinion` bit says a body exists, and only when the column is empty — so the default path never leaves the index and an opinion-less prior costs no store request. With the mode off the shaper is untouched, which is what keeps the CLI parity gate byte-identical. Degrades rather than fails. Rows are shaped and emitted one at a time, so a raising transport would truncate the JSON-lines stream mid-result; an unreadable store now logs and yields no body, matching the write side's best-effort posture. `read_opinion_text` returns the single field rather than a whole row on purpose: `opinion_text` is written only through `upsert_rows`, which always re-mirrors, so for this field the store is never staler than the blob — a guarantee whole-row reconstitution does not have, since the direct-UPDATE writers do not mirror. Two preconditions are documented rather than assumed: `has_opinion` is unset across the committed corpus and no `case.json` backfill exists, so this reader is correct but latent until both are fixed. Also drops a phantom `--include-opinion` flag from the docs — it exists nowhere in the code. Artifact: the priors an agent cell retrieves — `--full` is how a cell reads an opinion body rather than a summary. Closes #601 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
fix(evaluate): derive the backlog from resolved events, not a whole-court index
feat(query): route --full's opinion body to the content store under the split
Every inference-cost control was ex ante — capacity N, the per-run cell cap, the sweep cap, the per-cell attempt caps — each bounding one decision or one run, none reading what had actually been spent. They composed into a per-run limit with no per-period limit above it: across the day's scheduled windows, spend was bounded only by how many cells happened to be owed, the quantity that becomes large at a long conference. `spend.check_spend` sums the committed usage.json ledger over a trailing window and reports whether a configured ceiling is reached. Both plan seams consult it before minting a matrix, so a breach empties the matrix and the queued work is deferred: the predict queue and the evaluate backlog both re-derive from committed state on a later cycle. Evaluate is gated too — the ceiling governs total inference spend, and a grading costs a cell like a forecast does. Off by default (a ceiling of 0), which also short-circuits before the ledger is read: a cost control that wedges the pipeline when misconfigured is worse than none, so adopting it is a deliberate config edit. Two limits are documented rather than papered over. The ledger lags — a cell's usage.json arrives only when its run's collect PR merges — so the figure is a floor on spend inside the window, never a live balance, and both the error line and the step summary say so. And the gate is deliberately blunt: it does not attribute, forecast, or pro-rate, because it is the control that has to hold when a different control has already failed. Artifact: every predict and evaluate cell — this bounds what the fleet may spend before it spends it. Closes #860 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…o shim 1.1.0 ships `courtlistener/mcp/assets/`, so the missing-assets workaround is gone, and its session store falls back to an in-process TTL dict when no Redis URL is set, so the fakeredis pre-seed is gone with it. Verified from the published wheel rather than assumed. Two things the shim cleanup cannot do, both verified against 1.1.0's source: - The HTTP sidecar bypass STAYS. `create_http_app()` still raises without `REDIS_URL` and still forces `auth=build_auth()`, neither of which fits a loopback sidecar whose clients send no credential. It is trimmed to the three lines that do the work — build the server, serve HTTP on localhost. - `http_sidecar_launch`'s release guard is re-keyed, not deleted. It refused any package but the old pin, so bumping the manifest alone would have failed every CI cell at sidecar launch. The bypass still reaches into the release's internals, so the guard still earns its place; its message now names what a future bump must re-check. stdio drops to the release's own entry point with no `python -c` payload and no extra dependency, so `fakeredis` leaves the launch entirely (it was never a declared dependency — only a `uvx --with` string). Worth knowing before the cells run: 1.1.0 validates tool arguments strictly. Unknown keys and out-of-enum endpoint ids are hard errors where 1.0.0 ignored them, and `call_endpoint` now rejects a top-level `fields`. A sloppy retrieval call fails loudly instead of quietly returning nothing — better behaviour, but a behaviour change no unit test can see, which is why the engine smokes matter here. Recorded in the manifest comment where a prompt author will meet it. The pin is hashed into every actor's process version, so this is intentionally a new process digest. The frozen set is still empty, so nothing needs re-blessing. Artifact: the priors an agent cell retrieves — this is the retrieval surface every predict and evaluate cell reaches through. Closes #868 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
feat(cost): add an ex-post spend backstop over the measured usage ledger
chore(mcp): pin the CourtListener client to 1.1.0 and retire the stdio shim
…angs The seed walk never ran on 2026-07-27. STS was briefly unreachable, and `configure-aws-credentials` retried AssumeRole 8 times at ~6m45s each — a TCP connect timeout, not a prompt failure — until the job's 55-minute cap killed it. Every later step reports `skipped`, so the window bought zero cursor progress, and 55 minutes of runner time went to hanging. That inverts this pipeline's own rule. `config/tracking.yaml` says a degraded upstream must degrade the run "instead of hanging it into the CI timeout and losing the whole window's work" — implemented thoroughly for upstream data sources, and not at all for the CI steps that reach a cloud provider. Unbounded, the action's 12 retries outlast every job budget in the repo, so the job is always killed mid-retry rather than returning an error a workflow could act on. `action-timeout-s: 120` at all seven call sites. The action's own input rather than a step-level `timeout-minutes` because three of the seven are inside composite actions, whose steps cannot carry one — so this is the only bound that reaches them, and using it everywhere keeps the seven identical. 120s is chosen against a real cost, not picked for headroom: the action does not clear its timer on the error path, so with any timeout set, every failing assume-role now burns that full budget and ends on "Action timed out" instead of its own message. A larger value would double the cost of every credential misconfiguration to buy back only the tail retries that fire after ~100s of an already-failing STS. The guard also stops guessing. It said "the seed (historical) walk ended `cancelled`" for a run where the walk never started, which is what sent triage at the walker. It now reads the run's own job list and names the step that stopped it — verified against the real failed run, where it resolves to the credentials step. Artifact: the corpus itself — run-seed and run-pull's windows are what keep it current, and a hung window produces nothing. Closes #883 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ci: bound the assume-role step so a credential outage degrades, not hangs
"The maintainer merges" was written when every PR targeted main, so one rule covered two different concerns: do not touch the pre-registration record, and let a human see code before it lands. The staging split separated those, and the rule did not follow. Staging carries no data, is never a published record, and is guarded by required checks plus the main-base jail downstream. Holding its PRs to the promotion bar costs a maintainer round-trip per change without protecting anything the gate does not already protect. Four classes still wait for the maintainer even into staging, chosen where a green gate is weakest evidence and the risk is highest: the workflow and action permission surface, the security posture, the promotion gate itself, and the predictor/evaluator registries. Two things the rule now says out loud because they were true and unwritten. The branch rulesets require zero approvals on both branches, so nothing enforced ever stopped an agent merging — the discipline is convention, and saying so is more honest than implying a guardrail that does not exist. And with the maintainer out of the staging loop, the reviewer subagents stop being a courtesy and become the only review, so an unstated self-review has to be stated. Also corrects a convention that had quietly stopped working: `Closes #<n>` fires only from the default branch, so a staging PR never closes its issue and the backlog reports finished work as outstanding for a whole promotion cycle. Observed on four merged PRs. Artifact: none directly — it governs how every other change reaches the pipeline, and an instruction that describes behaviour the repo no longer has is one an agent follows confidently and wrongly. Closes #869 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
staging never owns data, so it falls behind main as the writers and bot lanes commit there — 10 commits behind as this lands. promote refuses a batch until staging contains main, and doing that catch-up at batch time is what costs: the merge moves staging's head, integration freshness is per-SHA, and every scenario green at the old head is instantly stale. A promotion therefore paid a full re-dispatch-and-re-approve cycle for a merge whose content is already-gated main history joined with already-gated staging history. A daily job opens a PR (base staging, head main) and arms auto-merge. It goes through the staging ruleset like any other change rather than around it, so this adds no bypass. At batch time staging already contains main and the freshness evidence still stands. Its own workflow file rather than a job on promote: promote is deliberately read-only and dispatch-only, and a scheduled job that opens and merges a PR is both a different trigger class and a different risk class — the case where a task earns its own surface instead of eroding a property another workflow was built to have. Three things the design turns on, none of them obvious: - The dev App, not the data App. The data App is a bypass actor on `main: require PR`; an unattended scheduled job must not hold the one identity that can push to main without review, and this needs no main write at all. - A merge commit, never a squash. A squash lands a commit with no parent link to main's tip, so the ancestry check fails and the next run reopens the same PR forever. - `--auto` errors when a PR is already mergeable, which is common here because the head sha is main's own and may already carry a green gate from its push run. It falls back to merging outright; the ruleset is enforced server-side, so an unsatisfied check still refuses. The sync defers while a promotion PR is open, so it never invalidates a batch in flight — the failure it exists to prevent, unattended. An already- open sync PR warns rather than passing green, since a silent success would hide a red gate while staging drifts. Artifact: every promotion batch, and so everything downstream of one — nothing merged reaches production until a batch lands. Closes #862 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
docs(agents): let agents merge to staging, keep main maintainer-only
ci: sync main into staging on a schedule, not at promotion time
…reakage Two questions the promotion regime left open on the dashboard. WHAT IT REPORTS. Scheduled runs check out the default branch, so run-ops executes the ops-report that has reached main and reads the tree actually running the pipeline — not the newer one staged for the next batch. That is the right answer, and it was accidental rather than chosen, so it is now written in the workflow header: a dashboard describing code that is not executing would be the wrong artifact, and main is the pre-registration record. The lag is narrower than it looks, since substance and spend read data/ and metrics/, which the writers commit to main directly and which are current either way; only what reflects code or config trails a batch. HOW IT READS PROMOTE. promote is level-triggered: each dispatch either names an unsatisfied gate and exits 1 with the fix in its summary, or hands back the promotion command. So a promotion sequence accumulates failures on the way to succeeding, and the health table renders that as `promote | failure | 0% (0/1)` — which reads as a broken workflow. It is the design working. The row stays, because a genuinely broken gate must remain visible, and the table footnotes it instead. The alternative — excluding it — would hide real breakage in the one workflow standing between staging and the published record. One worry checked and dismissed rather than fixed: the new always-skipped required checks cannot distort health, because `summarize_health` groups by workflow and `main-base` / `promotion-gate` are jobs within ci, not runs. And mechanically nothing about run-ops needed changing for the regime at all — it pushes to the orphan ops-metrics branch and updates an issue, never opening a PR to main. Artifact: the ops dashboard — a health table that cries wolf about its own promotion gate is one nobody reads. Closes #863 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ting ops: report the promoted state, and stop reading promote's gates as breakage
…e cohort Every existing selection test uses an artificial N of 1-3 over a handful of rows. That proves the mechanics and never touches the caps actually configured, so the September long conference would have been the first time the cap bound in production. Four tests over a realistic ~276-petition cohort at the real caps: the realized count is N plus carve-outs (211 at N=200, so the guaranteed-floor semantics are pinned, not assumed); every at-or-above-floor case survives a cohort far larger than N, which is the design's load-bearing promise and cannot be checked where the cap never bites; the fill is score-ordered rather than arrival-ordered; and two passes over identical input select an identical set, which is what makes the replay a pure read. Also records what the production corpus actually says, because it changes how the cap should be read. The gate IS armed — 24,857 rows scored under sal-v1, 3,516 latched — and the OT2026 long-conference cohort is accumulating: 352 petitions distributed for it, of which 239 are IFP and leave at Tier 0, leaving 113 eligible and all 113 funded. The cap is not binding and will not bind until the conference draws roughly three times its current volume. That two-thirds IFP share is the surprise worth writing down. A raw distribution count is a ceiling on the funded slice, not an estimate of it, and the doc previously invited reading the two as the same number — which would have overstated the release's size and cost by a factor of three. Artifact: the September cert release — this is the gate that decides which petitions it covers, and what it costs. Closes #858 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…essure test(salience): exercise the configured caps against a long-conference cohort
The cert back-test scores lift over the always-deny floor and per-band segment skill. The generic cross-court back-test scores raw accuracy and Brier with no baseline at all, and nothing said whether that was a decision or an oversight. It is a decision, and the reason is the set's composition rather than the triviality of its predictors. Measured over the corpus: of the machine-readable events it replays, about four in five are courts-of-appeals rows, and about nine in ten of those are dismissals. So the pooled label is not one quantity — `granted` means cert was granted on a SCOTUS row and a motion was granted on a circuit docket, and the harness averages them. That rules out the per-court base rate the issue floated as the one option generalizing the segment baseline. It would dress an incoherent comparison as a rigorous one, and describe a population the pipeline never forecasts: prediction scope is SCOTUS-only, and the circuit rows exist for retrieval context. The circuit rows also carry known text and metadata defects, so a lift computed on them would rest on evidence already reported unreliable. Recorded in both places a reader arrives from: the artifact catalogue, which now says what the number is not, and the module docstring, where the next implementer would otherwise re-open the question. Artifact: metrics/backtest.json — an unlabelled accuracy figure invites being read as forecasting skill, which is the one thing it cannot support. Closes #675 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…mposition The cross-court composition figures argue against a POOLED or per-court baseline. They say nothing about a SCOTUS-scoped one adjusted for the salience filter, which is the option a reader would actually propose — so leading with them left the conclusion unsupported for the case that matters. Reordered to the argument that does the work. For two floors that are trivial by construction, skill against a base rate is a closed-form function of that rate rather than a measurement, and the statpack already publishes the rate. Beyond that the population is measurably covered: all but a few per cent of this harness's SCOTUS rows are also in the cert back-test's set, which already reports the salience-adjusted baseline at the paid and carve-out-core scopes with per-band segment skill inside them. The rows unique to this harness are the non-modern-cert regimes the cert instrument excludes on purpose. The cross-court point stays, demoted to what it actually supports: why a per-court rate is a non-starter, separately from the SCOTUS question. Also records the asymmetry the decision leaves behind — the continuously refreshed artifact is the one without a skill signal, and the one with a skill signal is maintainer-triggered because it spends tokens. The remedy, if continuous tracking is ever wanted, is a scheduled cert-scoped run rather than a baseline on the cross-court floor. Artifact: metrics/backtest.json — a decision recorded with the wrong reason is one the next reader will reopen.
…overall Raw accuracy on this set is close to meaningless alone. A constant predictor scores its slice's base rate exactly, so `constant-denied` posting 0.909 granted-accuracy reads as excellent when 0.909 IS the floor. Nothing in the artifact told a reader that. Each entry now carries the floor and the lift over it, overall and per court, using the same convention the cert back-test already uses so the two are read the same way. The per-court cut is the one that matters, and the real corpus shows why: `prior-vote` scores 0.131 on SCOTUS against a 0.706 floor — lift -0.576, far worse than always saying denied, on the one population the pipeline predicts. The pooled lift is -0.110, because ca4 supplies 5,736 of 8,536 events at a 0.011 floor and drowns it. A pooled number here is effectively ca4's. Lift is presentational; ranking stays on accuracy then Brier. Promoting a pooled floor to the headline would rank on a figure that mixes outcome vocabularies — `granted` is cert granted on a SCOTUS row and a motion granted on a circuit docket — which is the one thing the cross-court shape genuinely rules out. The entry-level fields are optional, matching `mean_brier_score`: the committed artifact predates them and must still parse, and `null` honestly means the writing version did not compute the floor rather than asserting a zero. The weekly metrics refresh fills them. Artifact: metrics/backtest.json — an accuracy figure with no floor beside it invites reading a constant as a strong predictor, which is the failure this removes.
…ision feat(backtest): report the always-deny floor and lift, per court and overall
…e and in the agents' view The band rate is pooled over prior Terms, but *how many* prior Terms was an unstated default rather than a decision — and it is load-bearing. Per-Term high-band grant rates run 25.8%-48.0% across OT2017-OT2025; anchored at an OT2026 petition the band reads 37.1% (n=1006) unbounded, 33.8% (n=609) at five Terms and 43.7% (n=71) at one. That is a ~10-point spread in the number a forecast's Brier skill is scored against and in the prior a cell is told to anchor on. Two windows, because there are two: `salience.base_rate_lookback_terms` bounds the in-code baseline (the cert back-test's per-band skill, and the definition the evaluator is told to match), and `statpack.markdown_terms` bounds the Term table in `metrics/statpack.md` — the surface every agent that needs a prior actually reads, forward and replayed alike. Both ship at today's values, so no published number moves. The window is a Term-YEAR band, not a slice of the pack's rows: a Term absent from the statpack, or present only as a zero-row cursor entry, shortens the sample rather than pulling an older Term in to refill the slot. A rank slice would let the walker's coverage move every published skill number with no diff — which is the invisible shift this change exists to prevent. The two windows coincide at nine Terms and part once the pack passes ten, so both per-Term captions now state the rendered window; the prompts tell agents that a truncated caption bounds their pool and to record the divergence in `flags.json`. `metrics/statpack.md` gains that caption clause at the next metrics refresh — a sentence, not a number. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…, not a reviewer rule The `staging` deployment environment moves from an any-branch deployment policy plus a required-reviewer rule to deployment branches restricted to `staging` with no reviewer — the same shape as `prod`, one branch lower. This is the written model catching up to that settings change. The two controls gate different things. A branch policy is a code-provenance gate: only code that passed a PR plus `gate` and `paths` on the staging ruleset can bind the environment, enforced every time with nobody present. A per-run approval is a moment-of-use gate that asserts nothing about the code — the approval UI shows a workflow name and a ref, not a diff. Against the arrangement that exists the approval is redundant: no workflow declares `actions: write`, neither App holds an Actions scope, and the repo-scoped token agents hold is refused on `workflow_dispatch`, so dispatching is already maintainer-only and, with self-review permitted, the approval is a second click on the same decision by the same person. It is not redundant against a second write-access human, who could merge to `staging` unilaterally (the ruleset requires zero approving reviews), so that is recorded as an explicit revisit trigger alongside the first dispatch-capable token and the first workflow that would bind `staging` on a non-dispatch trigger. The concessions are stated rather than glossed: the staging ruleset requires no workflow linter, so a workflow change that reads a secret is caught by no required check — the control there is AGENTS.md's maintainer-wait on `.github/workflows/**` and `.github/actions/**`, convention rather than ruleset. And the blast radius is bounded on integrity, not confidentiality or spend: the role is read-only with no write path to the corpus, but it reads and lists the gated corpus, so the exposure is corpus read and model spend. What this gives up is dispatching integration scenarios from an arbitrary PR branch. Changed read seams are now validated once a change is on `staging` rather than before merge; the promotion gate still requires the seven integration runs green at the staging head, and `promotion-gate` is a required check on `main`, so nothing broken reaches `main` either way. Restoring an arbitrary-branch lane, if wanted, means a separate environment with its own keys, trust statement and reviewer — not widening this one, and it costs no workflow change since `deploy-environment` is a free-form string. Also resolves a contradiction the change surfaced: one paragraph claimed both S3 roles' OIDC trust is scoped to `prod` alone, while the staging integration runs demonstrably assume the read-only role. The read-write role names `prod`; the read-only role names both. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(corpus): read counsel through the optional-column contract The ranged and service backends read the published blob as-is — no connect migration runs — so counsel, like every column an older remote blob can lack, must read through the tolerant path. A direct record read raised KeyError against the production index and failed six of the seven promotion freshness scenarios. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(corpus): close the raw handle on the failure path too Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ronment
A scenario=all dispatch fans the promotion gate's whole required suite out
of one run — a plan job computes the {scenario, engine} matrix, the scenario
job fans out over it fail-fast-off, and freshness accepts one green
whole-line "integration-test: all @ staging" title at the head sha (runs
filtered to the staging head branch; per-scenario suffixes end-anchored).
The deploy-environment input defaults to auto and resolves from the
dispatching branch: main to prod, staging to staging, any other branch to
its own name, which binds an empty environment — no role variables, no
keys; an explicit value still wins. Single-scenario dispatches are
behavior-identical: one matrix leg carrying the inputs unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
deploy-environment becomes a closed choice (auto/prod/staging): run titles render it verbatim and feed the promotion gate's freshness matching, so every dispatcher-controlled title component must be a server-validated vocabulary — a free-text value on a green environment-free collect dispatch could otherwise forge per-scenario evidence from one crafted title line. Defense in depth in the gate script: start-anchored per-scenario prefixes, end-anchored environment suffix, head_branch pinned to staging, and titles preserving a newline excluded before matching. A test pins all three title inputs as choices so the channel cannot silently reopen. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ci(integration-test): an all scenario and branch-resolved deploy environment
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
chore(analytics): drop the unused weaker-or-equal band map
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First promotion batch since the move to the staging branch and environment: 136 commits / 60+ merged PRs. The pre-promotion batch below is fully landed; freshness evidence runs against staging's head at dispatch time.
What this batch carries
Promotion regime & CI routing — the machinery this PR itself runs on: the
main-basemerge-routing jail (#853), scheduled main→staging sync (#862, #886, #898), promoted-state reporting (#887), merge-authority split — agents merge to staging,mainis maintainer-only (#885, #917), branch policy as the staging environment's gate (#896), merge-commit rule scoping (#899), required-status-check pre-flight (#925), and the one-shotscenario=allfreshness suite with branch-resolved deploy environments and closed-choice title hardening (#987). Note:sync-staging.ymlfirst becomes operative when this PR merges; the sync for this batch is the documented by-hand path.Interim docket (new stream) — applications as a third numbering stream: discovery (#958), scope events (#959), salience signals and the escalation ladder (#960), polling (#961), the non-cert scope spellings fix (#957), and re-polling of unresolved applications with persisted escalation signals (#980).
Historical refresh (new) — the exact set the #967 re-walk needs on
main: thecounselcolumn (#962 — must be live before the walk; read through the optional-column contract, #985), keep every decided petition + full re-walk capability (#964), per-stream re-open (#965),run-seeddispatch input (#966).Salience & backtest — caps exercised against a long-conference cohort (#888), stated lookback window (#895), always-deny floor and per-court lift (#889), prior-vote population fix (#906), replay truncation by date (#947), segment baseline scoping (#908), the as-of projection layer and salience-gate replay over past Terms (#982 — the quantitative pre-#963 baseline), and its
run-backtestdispatch mode (#984).Claim taxonomy & outcome decomposition — taxonomy pre-registration (#934), frozen cert-stage
Outcome.signals(#935), merits-stage groundwork (#937, #944), frozenPredictionContext+base_rate_basis(#945, #946), leave-one-out evaluator agreement (#938), claim-withdrawal reasoning correction (#941).Analytics & metrics — court-facing docket pack (#930), paid-only cert cuts (#949), GVR comparability caveat (#950), scope-manifest refresh wiring (#955), docket-number annotation normalization (#954), grant-family series + statpack caveat (#978), duplicate-row dedupe CLI + trued-up caption (#981) with its
run-seedmaintenance step (#983).Cost, budget & infra — ex-post spend backstop (#880), per-case cost model correction (#865), bounded assume-role step (#884), MCP pin + stdio shim retirement (#882),
--fullopinion routing (#879), measured-ledger budget refresh with the interim-docket reserve (#977).Engines & observability — codex web search (#915), web-search-vs-MCP signal (#912), tool-usage rollup (#909), vote vocabulary + merits judgment axis (#948).
Evaluate durability — backlog derived from resolved events (#877), operational lessons (#866).
Ops — pipeline-runs dashboard with failure-only run logs (#979).
CLI / dev hygiene — #920, #926, #927, #914, #901, #916.
Pre-promotion batch status
run-backtestsalience-gate replay mode — ci(run-backtest): salience-gate replay mode #984run-seeddedupe maintenance step — ci(run-seed): dedupe live-minted duplicates in the maintenance tail #983run-pullpipeline-runs dashboard, failure-only run logs — ci(run-pull): a pipeline-runs dashboard and failure-only run logs #979 (closed ops: stop opening a run-log issue per day — keep the open/closed state as the failure signal #951)scenario=all+ branch-resolved deploy environment — ci(integration-test): an all scenario and branch-resolved deploy environment #987Code scanning: of the 13 CodeQL comments on this PR, one was real (#988); nine are false-positive multi-line prose/fixture strings and three flag the statically-cyclic but deliberately call-time-deferred
asof↔cert_backtestimport (documented at the import site) — those twelve are for dismissal in the security tab.Why now
Three open issues independently block on this promotion: #967 (the OT2017–2025 re-walk can only dispatch from
main), #907 (the process-digest freeze only takes effect on the default branch), and #923 (the scope-manifest regeneration rides metrics-refresh). The OT2026 first-conference milestone closes through those.Runbook (maintainer)
run:*trigger issues; no queued/in-progress predict/evaluate/backtest runs.sync-staging.ymlisn't onmainyet):git switch staging && git merge --no-edit origin/main && git push origin staginggh workflow run integration-test.yml --ref staging -f scenario=all(env auto-resolves; spends three engine-smoke cells on the staging keys).gh workflow run promote.yml --ref staging(staging's ref this once, so the gate script that accepts theallrun is the one that executes; after this promotion the plain dispatch works frommain).promotion-gatecheck immediately before merging; merge with a merge commit, never squash.