Phase 2 — validation framework v1

[thebeastagi]

Scope (ROADMAP.md, Phase 2)

• Expand L0→L2 automated gates with statistical sanity checks, holdout protocols, and explicit leakage controls.
• Add seeded-bias fixtures and calibration reports.
• Add confidence aggregation only after calibration data exists; do not invent precision.

Acceptance checklist

• L1 (internal) gate: statistical sanity checks on adapter evidence (determinism, score-range, degenerate-distribution rejection).
• L2 (replication) gate: declared holdout protocol — replication evidence must come from a different dataset identifier and independent seeds than internal evidence.
• Explicit leakage controls: promotion rejected when train/holdout datasets collide or seeds are reused across gates.
• Seeded-bias fixture suite: known-good and known-bad candidates with planted failure modes (overfit, leakage, degenerate).
• Calibration reports: fixture-suite run producing per-gate catch/false-pass rates.
• Confidence aggregation API that refuses to emit a number without calibration data — no invented precision.
• Tests green (ruff + pytest 3.10/3.11/3.12). Docs updated truthfully.
• Invariants: L4 human-gated; no discovery claims; no secrets; $0 spend.

Dependencies

Blocked by #7 — Phase 1 epic (trust registry is the entry point adapters pass through).

Tasks

• P2-T1: L1 statistical sanity gate #17 L1 statistical sanity gate
• P2-T2: L2 holdout protocol + leakage controls #18 L2 holdout protocol + leakage controls
• P2-T3: seeded-bias fixture suite #19 seeded-bias fixture suite
• P2-T4: calibration reports #20 calibration reports
• P2-T5: calibration-gated confidence aggregation #21 calibration-gated confidence aggregation
• P2-T6: Phase 2 docs + version 0.3.0 #22 Phase 2 docs + version 0.3.0