Phase 2 — validation framework v1 - #36
Merged
Merged
Conversation
…, checkpoints - sapiens.registry: SYNTHETIC/CORE/UNTRUSTED trust tiers derived from manifest provenance facts, replacing the Phase-0 synthetic-only gate - sapiens.permissions: owner-permission/licence manifest (permissions.json, empty by default — zero third-party permissions, clean-room invariant enforced by code); UNTRUSTED adapters require an active matching entry - sapiens.isolation: subprocess execution with POSIX rlimits (CPU, address space, open files) + wall-clock timeout for UNTRUSTED adapters; every failure mode contained fail-closed - sapiens.checkpoints: HMAC-SHA256 signed ledger checkpoints (env-held key, never stored) + external anchor export/verify; ledger gains checkpoint event kind with continuity verification - kernel: UNTRUSTED-tier adapters validate only via isolation; L4 stays human-gated; no real-data adapter ships in this phase - models: AdapterManifest gains code_origin / data_sources / third_party_source with coherence checks - tests: 45 new (permissions, registry, isolation incl. CPU/memory-hog and timeout containment, checkpoints, kernel isolation); fixed stale module-level time-budgeted ExecutionContext in photometry tests - docs: README/ROADMAP/ARCHITECTURE/PROVENANCE truthful for Phase 1; version 0.2.0 Closes #12, closes #13, closes #14, closes #15, closes #16. Part of #7.
- sapiens.validation: L1 internal-consistency gate (score presence/range, determinism across identical reruns, degenerate constant-score rejection) and L2 holdout-replication gate (declared HoldoutProtocol per domain; leakage controls on dataset collision and (dataset, seed) reuse; pass-fraction threshold); ValidationGates wiring config - sapiens.fixtures: labelled seeded-bias suite (known-good / overfit / leakage / degenerate) with expected gate outcomes - sapiens.calibration: CalibrationReport over the fixture suite — catch rate, false-reject rate, sample counts, content-hash report id - sapiens.confidence: aggregate_confidence raises UncalibratedError without sufficient calibration; emits documented heuristic (raw pass fraction x demonstrated catch rate) with provenance — no invented precision - kernel: opt-in validation wiring (fail-closed when a configured domain lacks a declared protocol); gate verdicts logged, never fabricated as ledger evidence - 33 new tests; docs (README/ROADMAP/ARCHITECTURE/VALIDATION) truthful; version 0.3.0 Closes #17, closes #18, closes #19, closes #20, closes #21, closes #22. Part of #8.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 2 — validation framework v1
Implements epic #8 (tasks #17–#22). Stacked on #35 (Phase 1); diff will shrink to Phase-2-only once #35 merges.
What lands
sapiens/validation.pysapiens/validation.pyHoldoutProtocolper domain; leakage controls on dataset collision and (dataset, seed) reuse; pass-fraction threshold.sapiens/fixtures.pysapiens/calibration.pysapiens/confidence.pyUncalibratedErrorwithout sufficient calibration; documented heuristic (raw pass fraction × demonstrated catch rate) with provenance. No invented precision.sapiens/kernel.pyValidationGates; fail-closed when a configured domain lacks a declared protocol; verdicts logged, never fabricated as evidence.Invariants
L4 human-gated · no discovery claims · clean-room · no secrets · $0 spend. Gates default off (backward compatible); same-numeric-seed across disjoint datasets is documented as legitimate holdout derivation, not leakage.
Tests: 33 new → 145 total green locally; ruff clean. Version 0.3.0.
Closes #17, closes #18, closes #19, closes #20, closes #21, closes #22. Part of #8.