Skip to content

feat: diagnostics-first API — ErrorReport, compare, pareto, per-sample times - #50

Merged
rxavier merged 1 commit into
mainfrom
roadmap/diagnostics-compare-timequality
Aug 1, 2026
Merged

feat: diagnostics-first API — ErrorReport, compare, pareto, per-sample times#50
rxavier merged 1 commit into
mainfrom
roadmap/diagnostics-compare-timequality

Conversation

@rxavier

@rxavier rxavier commented Aug 1, 2026

Copy link
Copy Markdown
Owner

Summary

Roadmap phases P0, P1, P3, P4 — diagnostics-first API overhaul.

P1 — ErrorAnalyzer report type

  • ErrorReport dataclass replaces loose dict from analyze()
    • universal_failures — samples every model got wrong
    • disagreement_set — samples where models split (ensembling candidates)
    • lift_by_target / lift_by_feature — error rate vs global rate
  • from_poniard defaults to all non-dummy estimators when estimator_names=None

P3 — Statistical comparison

  • compare() — paired fold t-tests with mean_diff, wins_a, wins_b, ties, p_value
  • Excludes dummies by default, supports metric/estimator subsets

P4 — Time-quality surface

  • pareto() — Pareto-optimal estimators (metric vs time)
  • best_under(seconds=...) — best metric within time budget
  • Both accept time_col with options: fit_time, score_time, fit_time_per_sample, score_time_per_sample

Per-sample times

  • fit() now stores fold sizes and computes fit_time_per_sample / score_time_per_sample
  • Both columns appear in get_results() automatically

P0 — README rewrite

  • Lead with diagnostics: multi-model diagnostics positioning
  • Error analysis is the hero section
  • Statistical comparison and time-quality promoted to top-level sections

Tests

  • 196 tests pass, ruff clean
  • 22 new tests for compare, pareto, best_under, and new ErrorReport fields

…ample times

Roadmap phases P0, P1, P3, P4:

- ErrorReport dataclass replaces loose dict from analyze()
  - universal_failures: samples every model got wrong
  - disagreement_set: samples where models split
  - lift_by_target / lift_by_feature: error rate vs global rate
- from_poniard defaults to all non-dummy estimators
- compare(): paired fold t-tests (mean_diff, wins, p_value)
- pareto(): Pareto-optimal estimators (metric vs time)
- best_under(): best metric within time budget
- fit_time_per_sample / score_time_per_sample stored during fit()
- README rewrite: leads with diagnostics, not model comparison
- 196 tests pass, ruff clean
@rxavier
rxavier merged commit 28d2cba into main Aug 1, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant