Skip to content

CONFORMANCE.md

Latest commit

 

History

History
102 lines (84 loc) · 6.61 KB

CONFORMANCE.md

File metadata and controls

102 lines (84 loc) · 6.61 KB
Raw

Conformance policy — the corpus is the original authority

The conformance/ corpus is the original authority on what a proofbundle verifier must accept and reject. When an implementation and the corpus disagree, the corpus wins until a governed change (below) says otherwise. This file is the policy; conformance/README.md documents the mechanics (case format, kinds, how to run).

Definitions

Core vector — a case authored in this repository, generated by the reference implementation or hand-constructed from a primary source (RFC test vector, spec example). Core vectors define the format's behaviour, including its failure behaviour.

External vector — a case produced by a second, independent implementation (for example MarkovianProtocol/audit-anchor) and vendored here as pure data. External vectors prove cross-implementation agreement. They are never edited to make them pass; they are recorded with exactly the properties they have, including expected-fail findings.

Non-negotiable rules

  1. Digest pinning. Every vendored artifact is pinned by content digest in the case's manifest. A verification run recomputes the digests; a mismatch fails the run. A case without pins is not accepted into the corpus.
  2. Attribution. Every external vector names its producing implementation and author in attribution, permanently. Removing or rewording attribution is a corpus change and goes through the change rule below.
  3. Negative tests. Every capability the corpus proves must also carry cases that prove the rejection path (wrong URI, changed content root, wrong anchor root, missing proof in a strict mode). A green run over accept-only cases is not conformance.
  4. Reproducible verification. python conformance/run_conformance.py verifies every case offline: no network, no calendar contact, frozen block material inside the case. If a case cannot be verified offline and deterministically, it does not enter the corpus.
  5. No silent changes to accepted vectors. Once a case is accepted, its vendored bytes and its expected block are frozen. Any change requires: a new case version (rule 6), the old case retained or explicitly retired in the commit message, and a PR that names the reason. Regenerating an external vector produces a new case; it never overwrites the old one in place. This is what makes a historical green run reproducible forever.
  6. Versioning. Cases carry stable caseIds. A successor case references the case it supersedes in rationale. The corpus manifest.json is the single registry; a case directory not in the manifest is not part of the corpus.
  7. A skipped check is never a passed one. The runner declares, per case, whether it ran in FULL, ran PARTIALly, or did NOT RUN, and the summary reports those counts — there is deliberately no N/N cases pass phrase to quote. An absent optional dependency reduces the executed scope, so it must reduce the reported one; --require-anchors turns that reduction into a failure for the authoritative run. The corpus-integrity precondition names which schema validator judged it, for the same reason. Measured 2026-09-15 by an external reviewer on the documented [test] install: the old summary said 122/122 cases pass while one case had not run at all and three had skipped their anchor sub-check.

Honest labelling

A case declares what it proves and what it does not (expected). The current corpus holds both confirmed end-to-end cases and deliberately scoped ones — for example an external decision vector recorded as canonicalization/root-binding-correct with its schema findings documented as an expected-fail rather than hidden. A green corpus run therefore never overclaims: it asserts exactly the recorded expectations, nothing more.

Change process

Corpus changes come as pull requests that (a) touch only the cases they name, (b) keep make conformance green, and (c) state per changed case whether it is new, a versioned successor, or a retirement. Maintainer review is required; for external vectors the attribution and the pure-data property (no local edits to vendored bytes) are review blockers.

Reported-version status (5.0.0)

A conformant implementation writes, beside every harness-reported version field, a <field>_status drawn from exactly reported, not_reported, not_bound, and a <field>_status_reason whenever that status is not reported. It rejects a provenance block that uses a fourth literal, that omits a mandatory reason, or in which status and field contradict each other in either direction.

The corpus carries one vector per status value and one per rejection class under conformance/provenance/. A rule with no rejection vector is not conformance-testable: an implementation could satisfy every positive case and still accept anything.

Receipt-envelope profile (5.1)

conformance/envelope_profile/ holds ten vectors for the profile in docs/RECEIPT_ENVELOPE_PROFILE.md: at least one counter-proof and one positive control for each of R1 to R4. R1 carries three, because two of its three divergence axes cannot arise in a format that refuses floats — for those the refusal itself is the counter-proof.

R5 carries no vector, and that is recorded rather than quietly left out. Its field form was withdrawn on 2026-08-30 after draft-hillier-coverage-attestation-00 (CAP-1, 20 Aug 2026) was measured to rule out the shape this profile had proposed. A counter-proof against a withdrawn shape would test nothing; the R5 probe follows once CAP-1 has been read. They run through the library's own emit and verify path — a vector family checked by a purpose-built mock would prove something about the mock.

The positive controls are not decoration. Without them a verifier that rejected every input would score perfectly on the counter-proofs, and the corpus would call that conformance.

R6 (every rule ships its counter-proof) deliberately has NO vector: a case asserting that the cases exist is the tautology the rule warns about. R6 is satisfied by the family existing and by its detection rate being measured — the planted defects that remain applicable after the R5 withdrawal are counted in the profile, together with the two that escaped on the first attempt and the two attempts that were ineffective rather than escaped (they changed a message, not a property). A third gap came from reading the profile against the corpus rather than from the meta-test: a meta-test measures whether a shipped check can fail, it cannot notice a check that was never shipped.