Phase 3 — structured L3 review panels - #37
Merged
Conversation
…, checkpoints - sapiens.registry: SYNTHETIC/CORE/UNTRUSTED trust tiers derived from manifest provenance facts, replacing the Phase-0 synthetic-only gate - sapiens.permissions: owner-permission/licence manifest (permissions.json, empty by default — zero third-party permissions, clean-room invariant enforced by code); UNTRUSTED adapters require an active matching entry - sapiens.isolation: subprocess execution with POSIX rlimits (CPU, address space, open files) + wall-clock timeout for UNTRUSTED adapters; every failure mode contained fail-closed - sapiens.checkpoints: HMAC-SHA256 signed ledger checkpoints (env-held key, never stored) + external anchor export/verify; ledger gains checkpoint event kind with continuity verification - kernel: UNTRUSTED-tier adapters validate only via isolation; L4 stays human-gated; no real-data adapter ships in this phase - models: AdapterManifest gains code_origin / data_sources / third_party_source with coherence checks - tests: 45 new (permissions, registry, isolation incl. CPU/memory-hog and timeout containment, checkpoints, kernel isolation); fixed stale module-level time-budgeted ExecutionContext in photometry tests - docs: README/ROADMAP/ARCHITECTURE/PROVENANCE truthful for Phase 1; version 0.2.0 Closes #12, closes #13, closes #14, closes #15, closes #16. Part of #7.
- sapiens.validation: L1 internal-consistency gate (score presence/range, determinism across identical reruns, degenerate constant-score rejection) and L2 holdout-replication gate (declared HoldoutProtocol per domain; leakage controls on dataset collision and (dataset, seed) reuse; pass-fraction threshold); ValidationGates wiring config - sapiens.fixtures: labelled seeded-bias suite (known-good / overfit / leakage / degenerate) with expected gate outcomes - sapiens.calibration: CalibrationReport over the fixture suite — catch rate, false-reject rate, sample counts, content-hash report id - sapiens.confidence: aggregate_confidence raises UncalibratedError without sufficient calibration; emits documented heuristic (raw pass fraction x demonstrated catch rate) with provenance — no invented precision - kernel: opt-in validation wiring (fail-closed when a configured domain lacks a declared protocol); gate verdicts logged, never fabricated as ledger evidence - 33 new tests; docs (README/ROADMAP/ARCHITECTURE/VALIDATION) truthful; version 0.3.0 Closes #17, closes #18, closes #19, closes #20, closes #21, closes #22. Part of #8.
- sapiens.review: role-specialized schemas (statistician, domain theorist, methodologist, devil's advocate), typed verdicts with severity-graded objections; bounded deterministic multi-round protocol with objection lifecycle (raised/sustained/withdrawn); disagreement gate — sustained MAJOR/BLOCKING rejects, MINOR caveats recorded but non-fatal - sapiens.reviewers: four deterministic reference reviewers re-running Phase-2 gates independently + hunting seeded-bias signatures (degenerate scores, train/holdout leakage, implausible perfection, thin stages); MAJOR escalates to BLOCKING on re-affirmation - sapiens.catchrate: per-role and panel catch-rate scoring over the seeded fixture suite with explicit small-sample caveat - kernel: opt-in L3 panel gate — panel verdict recorded in the ledger as review evidence (no side channel); approval required for L3 promotion; behaviour without a panel unchanged - 44 new tests; docs truthful; version 0.4.0 Closes #23, closes #24, closes #25, closes #26, closes #27, closes #28. Part of #9.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 3 — structured L3 review panels
Implements epic #9 (tasks #23–#28). Stacked on #36; diff shrinks as earlier phases merge.
What lands
sapiens/review.pysapiens/review.pysapiens/reviewers.pysapiens/catchrate.pysapiens/kernel.pyDesign notes
Two bugs were caught by the test suite during development and fixed: leakage checks initially over-flagged legitimate review-stage holdout reuse (now train-only crossing), and sustained-objection semantics are now explicitly exercised through full round budgets.
Invariants
L4 human-gated · no discovery claims · clean-room · no secrets · $0 spend.
Tests: 44 new → 189 total green locally; ruff clean. Version 0.4.0.
Closes #23, closes #24, closes #25, closes #26, closes #27, closes #28. Part of #9.