feat: structured L3 review panels (Phase 3 v1) [stacked on #40] - #41
Open
Tilanthi wants to merge 2 commits into
Open
feat: structured L3 review panels (Phase 3 v1) [stacked on #40]#41Tilanthi wants to merge 2 commits into
Tilanthi wants to merge 2 commits into
Conversation
…idence gate Replaces rigged adapter scores with genuine, reproducible statistics so evidence reflects held-out reality, not a planted answer. Pure standard library (no numpy/scipy) — stays inside the Phase-0 runtime constraint. - src/sapiens/validation.py: regularized incomplete beta (Lentz continued fraction) as the backbone; exact Clopper-Pearson binomial CI; Pearson correlation with two-sided p-value; Benjamini-Hochberg FDR (essential when a discovery loop tests many candidates); leakage-safe holdout_split; PassPolicy + evaluate() gate that decides pass/fail from real statistics. - src/sapiens/adapters/regression.py: SyntheticRegressionAdapter that evaluates candidates via genuine held-out Pearson r and p-value through the gate — not a planted score. - tests: numerical correctness vs analytic Beta CDFs and known CI/p-value/FDR values, plus regression-adapter climb and wrong-direction-rejection tests. Notable behaviour: a candidate whose correlation is significant but in the WRONG direction fails on effect size (point below min_effect) even though its p-value is tiny — the gate rejects p-value-hunting. Phase 0 invariants hold: synthetic data, real methodology, scientific_discoveries_claimed stays 0. ruff + pytest green (28 tests). Co-Authored-By: Claude <noreply@anthropic.com>
Replaces the single-shot L3 'review' self-check with a structured, multi-reviewer panel that scrutinises a candidate's accumulated evidence before any L3 promotion. Unanimous endorsement (a disagreement gate) is required over one or more rounds, with objections carried forward so reviewers can reconsider. - src/sapiens/review.py: Objection/ReviewOpinion/ReviewReport records; Reviewer protocol; StatisticianReviewer (re-examines effect size / p-value / sample size), MethodologistReviewer (flags review-on-training-data as leakage), DevilsAdvocateReviewer (demands overwhelming evidence); ReviewPanel with a unanimous-endorsement gate and multi-round objection carry-forward; review_evidence() producing ledger-valid L3 Evidence; catch_rate() scoring the panel on known-good/known-bad candidates. - tests/test_review.py: per-reviewer endorse/object behaviour, disagreement gate, multi-round, catch-rate, and panel-driven L3 promotion through the kernel via a PanelReviewedRegression adapter (real evidence -> panel -> L3). Consumes the Phase 2 validation framework's real statistics (stacked on PR thebeastagi#40). Phase 0 invariants hold: deterministic reviewers, synthetic candidates, no discovery claim. A domain-theorist reviewer is deliberately deferred to Phase 4 (needs real domain knowledge). ruff + pytest green (36 tests). Co-Authored-By: Claude <noreply@anthropic.com>
Tilanthi
added a commit
to Tilanthi/SAPIENS
that referenced
this pull request
Jul 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The first piece of Phase 3: a structured, multi-reviewer panel that replaces the single-shot L3 "review" self-check. Before any L3 promotion, a panel of role-specialised reviewers scrutinises the candidate's accumulated evidence; unanimous endorsement is required (a disagreement gate), over one or more rounds with objections carried forward so reviewers can reconsider.
Why
L3 is the last autonomous gate before a candidate is "external-ready" (L4, human-only). A single adapter self-check is too weak to trust at that boundary. Phase 3 makes L3 a genuine structured peer review — and it consumes the real statistical evidence produced by the Phase 2 validation framework (#40), so reviewers are checking real numbers, not planted scores.
Changes
src/sapiens/review.py— the framework:Objection/ReviewOpinion/ReviewReportrecords;Reviewerprotocol + three concrete reviewers:StatisticianReviewer— re-examines effect size, p-value, sample size;MethodologistReviewer— flags review evidence computed on training data as a leakage risk;DevilsAdvocateReviewer— demands overwhelming evidence, probing for alternative explanations;ReviewPanel— unanimous-endorsement disagreement gate, multi-round with objection carry-forward;review_evidence()— produces ledger-valid L3 Evidence, so promotion still flows through the kernel/ledger transition guard (traceability preserved);catch_rate()— scores the panel itself on known-good / known-bad candidates.tests/test_review.py— per-reviewer endorse/object behaviour, the disagreement gate, multi-round, catch-rate, and an integration test where aPanelReviewedRegressionadapter drives L3 promotion through the real kernel/ledger via the panel.Notable behaviour
A candidate whose evidence is "significant but not overwhelming" (e.g. r=0.8, p=1e-10) is rejected at L3 by the devil's advocate even though it would pass a naive p-value check — the panel enforces epistemic conservatism at the boundary.
Phase 0 invariants
Deterministic reviewers, synthetic candidates, no discovery claim. A
domain-theoristreviewer is deliberately deferred to Phase 4 (it needs real domain knowledge).review.pyis a core module.ruffclean,pytest→ 36 passed.🤖 Generated with Claude Code