fix(validate): refuse a committed vote score off a merits event

Resolving reviewer blockers on the stage gate.

`vote_accuracy` is the evaluator's own field to write on a real cell — the
harness computes it only in the stub and replay runners — so gating
`pipeline.evaluate` and the leaderboard aggregate still left an agent free to
commit a scored cert vote. The leaderboard refused to average it, but the
artifact was wrong. `validate`'s `vote_accuracy_only_on_merits_events` closes
that: it reads the stage off the committed `event.yaml`, exactly as
`check_merits_predictions` does for the judgment half, and refuses the record
rather than only the figure derived from it. It passes over every committed
evaluation today.

`docs/outcome-decomposition.md` still described the gate as owed by a future
vote channel, contradicting `decision-model.md`; restated as current design.

The SCDB entry's legacy-host claim was wrong. `scdb.wustl.edu` resolves and
serves over plain HTTP — it only refuses TLS — and carries a live CC BY-NC 3.0
US `rel="license"` badge whose prose caption is HTML-commented out. The entry
now states that rather than declining to look, and the conclusion is unchanged:
neither host states terms for the release that would be imported, so the terms
are unknown rather than permissive.

`metrics/README.md` and `decision-model.md` claimed the ranked board's vote
mean "is always null", which rested on data well-formedness rather than on
construction; both now state the guarantee the code makes. `scores_votes`
delegates to `declares` so the equality-not-identity rule holds by
construction. A test pins that the aggregate keys on the cell's own event
rather than the block it landed in. `test_scoring` keeps its coherent cert
fixture instead of being repointed at a merits id to preserve an assertion the
new evaluate tests cover properly.

Refs #1090

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Author: modelmirror

docs/data-sources.md

@@ -33,33 +33,43 @@ CourtListener roles once funding allows — see *The planned end-state* in
 [data-pipeline.md](data-pipeline.md). Adopting it requires reviewing that
 agreement's terms alongside the licenses below.
 
-A second is planned and not yet adopted: the **Supreme Court Database**
-(SCDB) — the standing academic coding of every Supreme Court decision, and the
-only realistic route to per-Justice merits votes at scale. It is the channel
-`docs/decision-model.md` names as one that could populate `Outcome.votes` with
-the provenance block no docket text supports today. Its terms are why it is not
-adopted, and the finding there is a negative one:
-
-- **No license is stated.** The project's current home is
-  [`scdb.la.psu.edu`](https://scdb.la.psu.edu/), and none of its homepage,
-  *About*, *Documentation*, *Data*, current-release, or *Cite Us* pages carries
-  a license, a terms-of-use statement, or anything about redistribution. What
-  they carry is a university copyright footer — "Copyright ©2026 The
-  Pennsylvania State University" — and a citation request. A CC BY-NC license
-  is widely attributed to this source second-hand and may well have been
-  declared on the legacy Washington University host, but that host
-  (`scdb.wustl.edu`) serves plain HTTP only and refuses TLS, so it could not be
-  read here and nothing about it is asserted. **Treat the terms as unknown, not
-  as permissive.**
-- **That is the blocker, and NC is why it matters.** If the license is a
-  NonCommercial one, the question is not whether this pilot is commercial —
-  unfunded research over public records, publishing no paid product, is the
-  easy case — but that NC binds downstream reuse of everything derived under it,
-  and this pipeline is built as a durable evaluation harness rather than one
-  paper. An adoption decision has to answer that for the project's intended
-  future. Since the site does not answer it either way, adopting this channel
-  means **getting the terms in writing from the maintainers first**, not
-  inferring them from a badge on a host that no longer resolves.
+A second such channel is planned and not yet adopted: the **Supreme Court
+Database** (SCDB) — the standing academic coding of every Supreme Court
+decision since the 1946 Term, and the only realistic route to per-Justice
+merits votes at scale. It is the channel [decision-model.md](decision-model.md)
+names as one that could populate `Outcome.votes` with the provenance block no
+docket text supports today. Its terms are why it is not adopted, and they are
+split across two hosts that do not agree:
+
+- **The current home states no license at all.**
+  [`scdb.la.psu.edu`](https://scdb.la.psu.edu/) is where the data is now
+  published, and none of its homepage, *About*, *Documentation*, *Data*,
+  current-release, or *Cite Us* pages carries a license, a terms-of-use
+  statement, or anything about redistribution. What they carry is a university
+  copyright footer — "Copyright ©2026 The Pennsylvania State University" — and
+  a citation request.
+- **The legacy host carries a badge and no sentence.** `scdb.wustl.edu` still
+  resolves and still serves the old site over plain HTTP (it refuses TLS, so it
+  is unreachable to any HTTPS-only client). That page carries a live
+  `rel="license"` badge linking
+  [CC BY-NC 3.0 US](https://creativecommons.org/licenses/by-nc/3.0/us/) —
+  Attribution-NonCommercial, with no ShareAlike and no NoDerivatives term. The
+  sentence that would have named the license in prose is **HTML-commented out**
+  and renders to nobody, so an image link is the whole of the declaration.
+- **Neither states terms for the release we would actually import.** A badge on
+  the superseded host is not a licence grant for the Penn State-published 2025
+  release, and the publishing host says nothing. **Treat the terms as unknown
+  rather than permissive**, and note that the more restrictive reading is the
+  safe one precisely because the permissive-looking evidence is the stale half.
+- **That is the blocker, and NC is why it matters.** Taking the badge at face
+  value, the question is not whether this pilot is commercial — unfunded
+  research over public records, publishing no paid product, is the easy case —
+  but that NC binds downstream reuse of everything derived under it, and this
+  pipeline is built as a durable evaluation harness rather than one paper. An
+  adoption decision has to answer that for the project's intended future. Since
+  the publishing host answers it neither way, adopting this channel means
+  **getting the terms in writing from the maintainers first**, not inferring
+  them from a commented-out caption on a host the project has moved off.
 - **Attribution is specific and versioned.** The project asks to be cited with
   its full author list and the exact release, because the data is corrected and
   extended in place: "Please be sure to include the specific Version Number;
@@ -97,13 +107,17 @@ reuse across years, and the pair covers the corpus as it stands — SCDB
 publishes a docket-organized cut of both its case-centered and justice-centered
 files, so the join is against a shipped organization rather than a
 reconstruction. The U.S. Reporter citation (`usCite`) join is the more precise
-one and is deliberately **not** primary: it reaches only ~161 corpus rows today,
-and its coverage is gated on the opinion-cluster enrichment backlog (*Pull
-cadence* below) rather than on anything SCDB controls. It stays a confirmation
-path, not the key. Justice names normalize to the **entry-printed surnames**
-already in use — the `judgment.opinion_author` vocabulary — rather than to
-SCDB's justice-name or numeric justice-id variables, so a single spelling serves
-both the docket-derived authorship recital and any imported vote list.
+one and is deliberately **not** primary: it reaches only the corpus rows whose
+`citations` column is populated — a small minority, since the column fills only
+as the opinion-cluster enrichment backlog (*Pull cadence* below) works through
+the cert-granted slice, not from anything SCDB controls. It stays a confirmation
+path, not the key. Justice names normalize to the **entry-printed surnames** the
+authorship parser already reads — `pipeline.judgment.opinion_author`'s
+vocabulary — rather than to SCDB's justice-name or numeric justice-id variables,
+so a single spelling serves both the docket-derived authorship recital and any
+imported vote list. That parser is advisory today and takes one name token, so
+it would have to be hardened for multi-token surnames before it could be the
+normalization target in fact rather than in intent.
 
 Two layers of rights apply, and they are different:
 
@@ -152,7 +166,7 @@ The automated consumer stays within CourtListener's published API limits by desi
   site has no metered API; the client is simply polite (browser user-agent,
   ~1 request/second, backoff on errors).
 - **SCDB would spend none either.** It publishes no API — access is bulk file
-  download only, in CSV and Stata, offered as case-centered and
+  download only, including CSV and Stata, offered as case-centered and
   justice-centered cuts (each organized by citation, by docket, or by
   issue/legal provision). So that channel's cost is a release pin and a
   re-download when the release moves, not a request budget, and it competes

docs/decision-model.md

@@ -487,8 +487,9 @@ gate enforces "merits-stage event ⇒ the scored prediction carries a judgment"
 from the committed `event.yaml`, the two halves meeting because a prediction
 does not carry its event's stage. The block is scored by `vote_accuracy`
 alone: over the Justices the outcome record actually names, under
-`vote_provenance` — never over what the predictor attempted, and never
-entering any total beyond that per-cell fraction. Today the merits outcome
+`vote_provenance` — never over what the predictor attempted. Beyond that
+per-cell fraction it enters one aggregate only, the merits block's
+`mean_vote_accuracy`, and no ranked total anywhere. Today the merits outcome
 writer records **no** votes, deliberately: the terminal docket entry's
 authorship recital names at most the opinion's author and never the
 participating count `VoteProvenance` requires as the aggregation denominator,
@@ -516,9 +517,13 @@ cannot be shown *not* to be cert. The consequence is that an ingestion channel
 populating `Outcome.votes` at the cert stage — noted dissents from denial are
 published on the order list and are the obvious candidate — changes nothing
 about what is scored. That is what makes the rule structural rather than a
-property of what a particular record contains, and the tests that fail when the
-gate is removed sit beside the scorer and the board (`tests/test_evaluate.py`,
-`tests/test_leaderboard.py`).
+property of what a particular record contains. A third seam covers the one the
+first two cannot: `vote_accuracy` is the evaluator's own field to write, so
+`validate`'s `vote_accuracy_only_on_merits_events` refuses to let a scored vote
+be *committed* off a merits event, rather than only refusing to aggregate it.
+The tests that fail when any of the three is removed sit beside the scorer, the
+board, and the gate (`tests/test_evaluate.py`, `tests/test_leaderboard.py`,
+`tests/test_replay.py`, `tests/test_validate.py`).
 
 **`judgment_correct` is descriptive, not a score.** The exact-match bit on the
 full vocabulary (`Evaluation.judgment_correct`) reports

docs/outcome-decomposition.md

@@ -755,10 +755,11 @@ scores rather than before it is declared.
 
 The per-Justice prohibition is structural, not stylistic: nothing in the
 mechanical family records or
-resolves a per-Justice dissent, and nothing writes one into `Outcome.votes`,
-because `docs/decision-model.md` pre-registers that an individual cert vote is
-never scored and a vote-populating channel owes a stage gate this claim does
-not pass through.
+resolves a per-Justice dissent, and nothing writes one into `Outcome.votes`.
+`docs/decision-model.md` pre-registers that an individual cert vote is never
+scored, and `pipeline.moments.scores_votes` enforces it — vote scoring is
+admitted only on a declared merits moment, so a channel that populated cert
+votes would not put this claim through a scored path.
 
 **`cert-v1` is superseded where it is declared, and kept where it is the
 fallback.** A version id names what a cell was *asked*, and the declaration is

docs/predicted-artifacts.md

@@ -458,9 +458,9 @@ in prose.
   prediction name the right label on the stage's own axis: the disposition at
   cert and interim, the judgment at merits), `brier_score` on the stage's
   binary, `judgment_correct` on a merits cell only, `vote_accuracy` over the
-  Justices both sides name — also on a merits cell only, because an individual
-  cert vote is never scored (`docs/decision-model.md`) —
-  `reasoning_quality`, a structured `leakage`
+  Justices both sides name — also on a merits cell only, since that is the one
+  stage whose votes are scored at all and an individual cert vote never is
+  (`docs/decision-model.md`) — `reasoning_quality`, a structured `leakage`
   assessment over the harness-captured retrieval log, and the evaluator's own
   independent `big_case` read. `claim_scores`, `base_rate_salience_version`,
   and `process_version` are the harness's, never the evaluator's word.

src/fedcourtsai/cli.py

@@ -1522,7 +1522,9 @@ def leaderboard(
     Deterministic and offline: aggregates the newest committed
     ``evaluation.json`` per (case, event, predictor, evaluator) under ``data/``
     into one best-first standing per predictor — accuracy, mean
-    Brier score, mean vote accuracy, a reasoning-quality summary, and counts,
+    Brier score, mean vote accuracy (declared merits moments only — an
+    individual cert vote is never scored, so the ranked board carries no vote
+    mean), a reasoning-quality summary, and counts,
     each reported **per stratum** (forward forecasts vs retrospective cells vs
     procedural mootness-basis cells, never blended and with only the timing
     strata ranked; see the ``Leaderboard`` schema). The ranked board is the

src/fedcourtsai/pipeline/moments.py

@@ -232,9 +232,12 @@ def scores_votes(event_id: str) -> bool:
     :func:`spec_for` documents is the wrong one for a prohibition; a caller
     reading a vocabulary loses nothing by falling back, while a guard that
     falls back scores exactly the ids it cannot vouch for.
+
+    Expressed through :func:`declares` rather than re-reading the table, so the
+    equality-not-identity rule that function documents holds here by
+    construction instead of by inspection.
     """
-    spec = _BY_EVENT_ID.get(event_id)
-    return spec is not None and spec.stage == Stage.merits
+    return declares(event_id, Stage.merits)
 
 
 def first_moment(stage: Stage) -> Moment | None:

src/fedcourtsai/validate.py

@@ -25,7 +25,10 @@
   prose document a ``prediction.json`` names resolves to a file beside it; every
   committed claims block is one the claim scorer will not silently void; and
   every merits-stage event's scored (latest-per-predictor) prediction carries
-  its ``judgment`` — the stage-aware half of the merits prediction contract.
+  its ``judgment`` — the stage-aware half of the merits prediction contract —
+  and no evaluation carries a ``vote_accuracy`` off a merits event, since an
+  individual cert vote is never scored and that field is the evaluator's own to
+  write.
 
 The verdict is a pure function of its inputs (corpus, ledger, baseline,
 tracked courts, as-of date), with no clock or network, so it is deterministic and
@@ -59,6 +62,7 @@
     CorpusScopeAudit,
     CorpusValidation,
     Disposition,
+    Evaluation,
     EventKind,
     LedgerValidation,
     PredictableEvent,
@@ -98,6 +102,7 @@
 CHECK_PREDICTION_DOCS = "prediction_docs_exist"
 CHECK_PREDICTION_CLAIMS = "prediction_claims_scoreable"
 CHECK_MERITS_PREDICTIONS = "merits_predictions_carry_judgment"
+CHECK_SCORED_VOTES = "vote_accuracy_only_on_merits_events"
 CHECK_STALE_UNPARSED_GRANTS = "no_stale_unparsed_grants"
 
 # How long after its cert grant a merits proceeding may sit with neither a
@@ -814,6 +819,58 @@ def check_merits_predictions(data_root: Path) -> CorpusCheck:
     return _check(CHECK_MERITS_PREDICTIONS, problems, checked=checked)
 
 
+def check_scored_votes(data_root: Path) -> CorpusCheck:
+    """No committed evaluation may carry ``vote_accuracy`` off a merits event.
+
+    An individual cert vote is never scored (``docs/decision-model.md``), and
+    ``pipeline.moments.scores_votes`` enforces that wherever the harness computes
+    the figure. But on a real cell ``vote_accuracy`` is the *evaluator's* field to
+    write, like ``brier_score`` — the harness stamps only ``claim_scores`` and the
+    base-rate basis record — so the computed gate cannot speak for an agent that
+    wrote the number itself. The leaderboard refuses to aggregate such a value,
+    which keeps it out of every published total; this check refuses to let it be
+    committed at all, so the prohibition holds on the artifact and not merely on
+    the figures derived from it.
+
+    Read against the committed ``event.yaml`` rather than the moments register:
+    an ``evaluation.json`` does not carry its event's stage, and the ledger's
+    stage stamp is what a reader of the artifact sees. That makes this the same
+    shape as :func:`check_merits_predictions` — the schema holds what it can
+    self-contained, and the half needing the event definition lives here. A file
+    that does not parse is ``validate_ledger``'s concern (schema law) and is
+    skipped.
+
+    The directory test precedes the parse for the reason
+    :func:`check_merits_predictions` gives: most events carry no evaluations, and
+    ``validate data`` runs once per cell in both fan-outs.
+    """
+    problems: list[str] = []
+    checked = 0
+    for event_file in _ledger_files(data_root, "*/*/events/*/event.yaml"):
+        evaluations_root = event_file.parent / "evaluations"
+        if not evaluations_root.is_dir():
+            continue
+        try:
+            event = PredictableEvent.model_validate(yaml.safe_load(event_file.read_text()))
+        except (OSError, ValueError, ValidationError):
+            continue
+        if event.stage == Stage.merits:
+            continue
+        for path in sorted(evaluations_root.glob("*/*/*/evaluation.json")):
+            try:
+                evaluation = Evaluation.model_validate(json.loads(path.read_text()))
+            except (OSError, ValueError, ValidationError):
+                continue
+            checked += 1
+            if evaluation.vote_accuracy is not None:
+                problems.append(
+                    f"evaluation {path}: carries vote_accuracy on "
+                    f"{event.stage or 'stage-less'}-stage event {event.event_id!r} — "
+                    f"a vote is scored only on a merits event"
+                )
+    return _check(CHECK_SCORED_VOTES, problems, checked=checked)
+
+
 # --- referential integrity (git-only subset, for the PR gate) ------------------
 
 
@@ -885,6 +942,7 @@ def run_ledger_referential_checks(data_root: Path) -> list[CorpusCheck]:
         check_prediction_docs(data_root),
         check_prediction_claims(data_root),
         check_merits_predictions(data_root),
+        check_scored_votes(data_root),
     ]
 
 
@@ -915,6 +973,7 @@ def _run_checks(
         check_prediction_docs(data_root),
         check_prediction_claims(data_root),
         check_merits_predictions(data_root),
+        check_scored_votes(data_root),
     ]
     return CorpusValidation(
         ok=all(c.passed for c in checks),

tests/test_evaluate.py

@@ -18,7 +18,7 @@
 
 import pytest
 
-from fedcourtsai import corpus
+from fedcourtsai import corpus, ids
 from fedcourtsai.pipeline.base_rates import (
     INTERIM_BASE_RATE_MIN_RESOLVED,
     MERITS_BASE_RATE_MIN_PARSED,
@@ -44,6 +44,7 @@
     BaseRateBucket,
     Disposition,
     Engine,
+    EventKind,
     Judgment,
     JusticeVote,
     Outcome,
@@ -1036,15 +1037,24 @@ def test_vote_accuracy_is_denied_by_default_off_the_register() -> None:
     Denial is the default rather than the cert stage being named: the register
     is the only authority on an event's stage, so an id it does not declare —
     an entry-pinned event, a record older than the table — is one that cannot be
-    shown *not* to be cert. Interim moments are declared and unscored for the
-    plainer reason that the stage forecasts no votes.
+    shown *not* to be cert.
     """
-    assert not scores_votes("evt-appeal-disposition")
-    prediction, outcome = _voting_pair("evt-appeal-disposition")
+    undeclared = ids.event_id(EventKind.appeal.value, "disposition")
+    assert not scores_votes(undeclared)
+    prediction, outcome = _voting_pair(undeclared)
     assert vote_accuracy(prediction, outcome) is None
+
+
+def test_vote_accuracy_is_not_scored_at_the_interim_stage() -> None:
+    """Declared, and still unscored — for a plainer reason than the cert rule.
+
+    An interim moment is on the register, so this is not the deny-by-default
+    fallback: the stage simply forecasts no votes, and admitting only merits
+    moments is what makes that hold without a second rule.
+    """
     for spec in moments_for(Stage.interim):
-        interim_prediction, interim_outcome = _voting_pair(spec.event_id)
-        assert vote_accuracy(interim_prediction, interim_outcome) is None, spec.event_id
+        prediction, outcome = _voting_pair(spec.event_id)
+        assert vote_accuracy(prediction, outcome) is None, spec.event_id
 
 
 def test_realized_band_rate_reads_a_retired_versions_alt_segments_block() -> None: