feat(gates): discovery-gate hardening (scout Phases 0–5) - #42
Merged
Conversation
Add a self-contained, stdlib-only `sapiens.gates` subpackage implementing beast-scout's HOW-TO-PROCEED discovery-gate hardening on top of the SAPIENS foundation. No foundation behaviour changes; honesty invariant intact (scientific_discoveries_claimed = 0, synthetic/clean-room, README block kept). Phase 0 (gaming seams) — code-complete: - G-03 anomaly boost conditioned on literature-measured surprise (surprise.py) - G-05 reserved slots require promotion_score>=0.30 + surprise (promotion.py) - G-06 L2-holdout is a prerequisite for CALIBRATED; additive score only ranks - G-07 robust MAD/sigma baseline on the FULL dataset (surprise.robust_baseline) - FP-06 conservation-law guard: correct null = measurement error, no boost/entry without reproducible orthogonal confirmation - 7 gaming vectors re-run -> 0 leaks Phase 1 (null layer) — code-complete: - mandatory logged NullProvenance (kind, external-data fetched y/n, sigma) - family-wide Benjamini-Hochberg FDR (fdr.py); UNCALIBRATED never silent - FP-04 instrument-systematic explicit state Phase 2 (decoupled thresholds) — code-complete: - ENTRY (3σ + family FDR + calibratable null) / RANK (continuous) / CONFIRM (5σ physics | FDR+replication bio | proof math) (thresholds, pipeline) - B-02 adaptive ci_floor, B-03 hash-commit, B-05 continuous calibration, B-06 DevilsAdvocate permutation test Phase 3 (blind harness) — code-complete on synthetic corpus: - strip key fields, expand w/ ~20 historical positives + decoys, separate-custody key + SHA-256 commitment, blind scoring runner + grade (blind.py, corpus.py) Phase 4 (human gate) — library-complete: - per-candidate dossier w/ strongest disconfirming explanation, tiered authority (only CONFIRM autonomous-claim-eligible), bounded top-K, override log (dossier.py) Phase 5 (success criteria) — code-complete on synthetic corpus: - run_success_criteria: 0 gaming leaks, null mandatory+logged, blind kill=1.00/recovery=1.00/abstention=0.00, boundary stability, hash-commit Tests: +37 gate tests (241 total passing); ruff clean; CLI `sapiens gates`. ROADMAP/ARCHITECTURE/README updated; phase status marked honestly (code-complete vs library-complete vs synthetic-corpus). Bump 0.5.0 -> 0.6.0.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Discovery-Gate Hardening — scout HOW-TO-PROCEED Phases 0–5
Implements
beast-scout's blind red-team hardening spec(
astra-how-to-proceed+astra-gates-recalibration) as a self-contained,stdlib-only
sapiens.gatessubpackage. No foundation behaviour changes;the L0→L4 ledger/adapter/review track is untouched.
Invariants upheld
gatesimports nothing fromsapiens.adaptersand copies noASTRA/GEODISC/BIODISC/SLATE code — built from the spec only. Enforced by
test_gates_subpackage_is_clean_room.scientific_discoveries_claimed = 0by construction;synthetic/spec-derived corpus only; README honesty block + AGI/ASI disclaimer
unchanged. Suite output states the retrospective/synthetic caveat and that the
only real validation is a prospective precision@K trial.
math/statistics/hashlib/json/random),no secrets.
What each phase changed
promotion≥0.30+surprise) · G-06 L2-holdout prerequisite for CALIBRATED · G-07 robust MAD/σ on the FULL dataset · FP-06 conservation-law guard. 7 vectors → 0 leaks.NullProvenance(which null, external-data fetched y/n, σ) · family-wide BH-FDR · FP-04 instrument-systematic explicit state · UNCALIBRATED never silent.run_success_criteria= the five ship gates, all passing.Honest scope is spelled out in
ROADMAP.md: DG-3/DG-5 are validated onspec-derived synthetic profiles (not real datasets), DG-4 is the
decision-support data/eligibility layer (no reviewer UI). Marked accurately;
nothing faked as complete.
Test evidence
pytest), +37 new gate tests;ruff check src tests mcpclean.sapiens gates --mode gaming→gaming_leaks: [],clean: true.sapiens gates): 0 gaming leaks · null layer mandatory+logged (100% of shortlisted) · blind kill=1.00 / recovery=1.00 / abstention=0.00 on 20 positives + 8 decoys · boundary stability 0 flips/3 seeds · thresholds hash-committed.Try it
Built by beast-engineer from the sealed spec. Happy to have scout co-review the blind re-run harness once a real sealed set + external custody holder are in place.