feat: diagnostics-first API — ErrorReport, compare(), pareto(), per-sample times

Roadmap phases P0, P1, P3, P4:

- ErrorReport dataclass replaces loose dict from analyze()
  - universal_failures: samples every model got wrong
  - disagreement_set: samples where models split
  - lift_by_target / lift_by_feature: error rate vs global rate
- from_poniard defaults to all non-dummy estimators
- compare(): paired fold t-tests (mean_diff, wins, p_value)
- pareto(): Pareto-optimal estimators (metric vs time)
- best_under(): best metric within time budget
- fit_time_per_sample / score_time_per_sample stored during fit()
- README rewrite: leads with diagnostics, not model comparison
- 196 tests pass, ruff clean

Author: rxavier

README.md

@@ -7,17 +7,9 @@
 > A poniard /ˈpɒnjərd/ or poignard (Fr.) is a long, lightweight
 > thrusting knife ([Wikipedia](https://en.wikipedia.org/wiki/Poignard)).
 
-Poniard is a scikit-learn companion library that streamlines the process of fitting different machine learning models and comparing them.
+Poniard is a scikit-learn companion for **multi-model diagnostics**. Compare models to get oriented, then answer *where they fail, whether differences are real, and what to try next* — then export a plain sklearn object and leave.
 
-It can be used to provide quick answers to questions like these:
-
-- What is the reasonable range of scores for this task?
-- Is a simple and explainable linear model enough or should I work with forests and gradient boosters?
-- Are the features good enough as is or should I work on feature engineering?
-- How much can hyperparameter tuning improve metrics?
-- Do I need to work on a custom preprocessing strategy?
-
-This is not meant to be an end-to-end solution, and you should keep working on your models after you are done with Poniard.
+Not AutoML. Not end-to-end. Every feature earns its place.
 
 ## Installation
 
@@ -40,12 +32,68 @@ from poniard import PoniardClassifier
 X, y = make_classification(n_samples=200, n_features=10, random_state=42)
 
 clf = PoniardClassifier()
-clf.setup(X, y)    # configure: type inference, preprocessing, pipelines
-# optionally: clf.add_estimators(...), clf.reassign_types(...), etc.
 clf.fit(X, y)      # cross-validate all estimators
 clf.get_results()  # comparison table
 ```
 
+## Error analysis — the reason to install
+
+`ErrorAnalyzer` answers *where and why* your models fail. Build it from a
+fitted `PoniardClassifier` / `PoniardRegressor` and run the full workflow with
+a single call:
+
+```python
+from poniard.error_analysis import ErrorAnalyzer
+
+ea = ErrorAnalyzer.from_poniard(clf)  # all non-dummy estimators by default
+report = ea.analyze(X, y)
+```
+
+The `report` is a structured `ErrorReport` containing:
+
+- **`universal_failures`** — samples every model got wrong
+- **`disagreement_set`** — samples where models split (useful for ensembling)
+- **`lift_by_target`** — per class/bin, error rate relative to the global rate
+- **`lift_by_feature`** — per feature value, error rate relative to the global rate
+- **`ranked_errors`** — per estimator, samples sorted by error magnitude
+- **`merged_errors`** — cross-estimator view: frequency and mean error per sample
+- **`summary`** — per estimator: error count, error rate, mean error
+- **`by_target`** / **`by_feature`** — error distributions
+
+```python
+report.universal_failures   # what's toxic to every model
+report.lift_by_target       # which classes are over-represented in errors
+report.disagreement_set     # where models disagree (ensembling candidates)
+```
+
+How errors are defined:
+
+- **Classification**: misclassified samples, ranked by `1 - probability of the truth` (how confidently wrong the model is).
+- **Regression**: samples whose absolute residual exceeds a threshold (default: 90th percentile), ranked by residual magnitude.
+
+## Statistical comparison
+
+Stop pretending fold-mean leaderboards are truth. `compare()` runs paired
+tests on cross-validation folds:
+
+```python
+clf.compare()
+# pairwise: mean_diff, wins_a, wins_b, ties, p_value
+```
+
+## Time vs quality
+
+Pick "good enough and cheap" in one call:
+
+```python
+clf.pareto()                                              # best metric vs fit_time
+clf.pareto(time_col="score_time_per_sample")              # vs inference time per sample
+clf.best_under(seconds=0.5)                               # best metric where fit_time <= 0.5s
+clf.best_under(seconds=0.001, time_col="score_time_per_sample")  # fast inference
+```
+
+Available time columns: `fit_time`, `score_time`, `fit_time_per_sample`, `score_time_per_sample`.
+
 ## Exporting a model (leaving Poniard)
 
 `get_estimator` is the supported way to leave Poniard. It returns a plain
@@ -58,10 +106,6 @@ model = clf.get_estimator("LogisticRegression", retrain=True, X=X, y=y)
 # model is a fitted sklearn.pipeline.Pipeline you fully own
 ```
 
-Without `retrain=True`, the returned pipeline is an unfitted clone you can
-inspect. Use it to extract any estimator from the comparison — defaults,
-hyperparameter-optimized ones after `tune_estimator`, or ensemble members.
-
 ## Hyperparameter tuning (stays in the experiment)
 
 `tune_estimator` runs a search on the **same** preprocessor/pipeline, then adds
@@ -78,8 +122,6 @@ clf.get_results()
 clf.get_tuning_results("LogisticRegression_tuned")  # best_params_, search, ...
 ```
 
-Pipeline-style keys (`LogisticRegression__C`, `preprocessor__...`) still work.
-
 ## Plotting
 
 Plotting is a separate module (requires `pip install poniard[plot]`):
@@ -98,74 +140,33 @@ plotter.permutation_importance("LogisticRegression")
 plotter.full_estimator_analysis("LogisticRegression")
 ```
 
-## Error analysis
-
-`ErrorAnalyzer` answers *where and why* your models fail. Build it from a
-fitted `PoniardClassifier` / `PoniardRegressor` and run the full workflow with
-a single call:
-
-```python
-from poniard.error_analysis import ErrorAnalyzer
-
-ea = ErrorAnalyzer.from_poniard(clf, estimator_names=["LogisticRegression", "RandomForestClassifier"])
-report = ea.analyze(X, y)  # X, y = the data you fitted on
-```
-
-`report` contains:
-
-- `ranked_errors` — per estimator, samples sorted by error magnitude
-- `merged_errors` — per sample, how many estimators failed and their average error
-- `summary` — per estimator: number of errors and error rate
-- `by_target` — error counts and error rate per target class/bin
-- `by_feature` — per feature, the distribution of errors across its values
-
-The individual steps are also exposed:
-
-```python
-ranked = ea.rank_errors(X, y)                       # per-estimator ranked errors
-merged = ErrorAnalyzer.merge_errors(ranked)         # cross-estimator view
-ea.analyze_target(errors_idx=merged.index, y=y)     # errors vs target distribution
-ea.analyze_features(errors_idx=merged.index, X=X)   # errors vs feature values
-```
-
-How errors are defined:
-
-- **Classification**: misclassified samples, ranked by `1 - probability of the
-  truth` (how confidently wrong the model is). Multilabel targets rank by the
-  mean per-label deviation.
-- **Regression**: samples whose absolute residual exceeds a threshold, ranked by
-  residual magnitude. The threshold defaults to the 90th percentile of residuals
-  and can be configured with `error_quantile` in `rank_errors` / `analyze`.
-
 ## Estimator naming
 
 Each estimator gets a name automatically (its class name). You can override with tuple syntax:
 
 ```python
-# Single of each class → class names
-clf = PoniardClassifier(estimators=[LogisticRegression(), SVC()])
-# pipelines: {'LogisticRegression': ..., 'SVC': ..., 'DummyClassifier': ...}
+# Tuple override
+clf = PoniardClassifier(estimators=[('my_lr', LogisticRegression())])
+# pipelines: {'my_lr': ..., 'DummyClassifier': ...}
 
 # Duplicates → collision handling
 clf = PoniardClassifier(estimators=[
     LogisticRegression(max_iter=1000),
     LogisticRegression(C=0.1),
 ])
 # pipelines: {'LogisticRegression': ..., 'LogisticRegression_2': ..., 'DummyClassifier': ...}
-
-# Tuple override
-clf = PoniardClassifier(estimators=[('my_lr', LogisticRegression())])
-# pipelines: {'my_lr': ..., 'DummyClassifier': ...}
 ```
 
 ## Features
 
+- **Error analysis**: Universal failures, disagreement sets, lift vs baseline — find *where and why* models fail
+- **Statistical comparison**: Paired fold tests to see if A really beats B
+- **Time-quality tradeoff**: Pareto front and best-under-budget helpers
 - **Automatic type inference**: Detects numeric, categorical, and datetime features
 - **Built-in preprocessing**: Imputation, encoding, scaling via a configurable pipeline
 - **Cross-validated comparison**: Fits multiple estimators with cross-validation and collects results
 - **Hyperparameter tuning**: Grid, random, and halving search for any estimator
 - **Ensemble building**: Create ensembles from fitted estimators
-- **Error analysis**: Rank prediction errors, and analyze them against the target and features to find *where and why* models fail
 - **Plotting**: Metrics comparison, ROC curves, confusion matrices, feature importance (optional, requires plotly)
 
 ## Environment variables

ROADMAP.md

@@ -52,11 +52,13 @@ Not AutoML. Not end-to-end. Every feature must earn its place.
 - [x] Plots: **left alone** (optional satellite; revisit later)
 - [x] `setup` kept (audit mutator workflow)
 - [x] `tune_estimator` redesigned as experiment glue (see §5 done items)
+- [x] Full README wedge rewrite (with P1 error-analysis story)
+- [x] `compare()` added (paired fold comparison)
+- [x] `pareto()` and `best_under()` added (time-quality surface)
 
 **Still open**
-- Full README wedge rewrite (with P1 error-analysis story)
 - Plot diet (deferred — owner review)
-- Later headline adds: `compare`, diversity ensemble, pareto
+- Diversity ensemble (P2)
 
 **Ship when:** README tells the new story in one screen; public surface is small enough to hold in your head.
 
@@ -66,26 +68,28 @@ Not AutoML. Not end-to-end. Every feature must earn its place.
 
 **Goal:** The reason someone installs Poniard. Screenshot-worthy multi-model failure forensics.
 
+**Status: Done** — `ErrorReport` dataclass with universal failures, disagreement set, lift by target/feature.
+
 ### 1.1 Report shape (wow minimum)
 `analyze()` (or a dedicated report object) should make these trivial:
 
-- **Universal failures:** samples every selected model gets wrong (`freq == n_estimators`).
-- **Disagreement set:** samples where models split (useful for ensembling and labeling).
-- **Lift vs baseline:** per target class/bin and per feature value — error rate vs global error rate (not just raw counts).
-- **Top slices:** feature values / bins where error lift ≥ threshold (e.g. 2×), ranked.
-- **Per-estimator summary:** n_errors, error_rate, mean confidence-of-wrong / residual — already mostly there.
-- **Stable indices:** sample ids that survive CV prediction alignment (document assumptions hard).
+- [x] **Universal failures:** samples every selected model gets wrong (`freq == n_estimators`).
+- [x] **Disagreement set:** samples where models split (useful for ensembling and labeling).
+- [x] **Lift vs baseline:** per target class/bin and per feature value — error rate vs global error rate (not just raw counts).
+- [ ] **Top slices:** feature values / bins where error lift ≥ threshold (e.g. 2×), ranked.
+- [x] **Per-estimator summary:** n_errors, error_rate, mean confidence-of-wrong / residual.
+- [ ] **Stable indices:** sample ids that survive CV prediction alignment (document assumptions hard).
 
 ### 1.2 API cleanup
-- First-class report type (dataclass / simple namespace) instead of a loose dict — still easy to print and index.
-- `from_poniard` default: analyze all non-dummy fitted estimators if names omitted.
-- Avoid recompute traps: reuse cached `cross_val_predict` / proba from the Poniard session when present.
-- Clear separation: ranking definition (classif vs reg) stays explicit and documented.
+- [x] First-class report type (`ErrorReport` dataclass) instead of a loose dict.
+- [x] `from_poniard` default: analyze all non-dummy fitted estimators if names omitted.
+- [ ] Avoid recompute traps: reuse cached `cross_val_predict` / proba from the Poniard session when present.
+- [x] Clear separation: ranking definition (classif vs reg) stays explicit and documented.
 
 ### 1.3 Stretch (after minimum wow)
-- Simple cohort labels ("high cardinality category X", "target bin top decile").
-- Optional short text summary for notebooks (`report.narrative()` — careful, no LLM deps; templated stats only).
-- Hook points for plots: error lift bars, universal-failure table, disagreement heatmap.
+- [ ] Simple cohort labels ("high cardinality category X", "target bin top decile").
+- [ ] Optional short text summary for notebooks (`report.narrative()` — careful, no LLM deps; templated stats only).
+- [ ] Hook points for plots: error lift bars, universal-failure table, disagreement heatmap.
 
 **Ship when:** A user can run `ErrorAnalyzer.from_poniard(clf).analyze(X, y)` and immediately answer:
 "what rows are toxic to every model?", "which slices are 3× worse?", "where do models disagree?"
@@ -133,13 +137,15 @@ clf.build_ensemble(
 
 **Goal:** Stop pretending fold-mean leaderboards are truth.
 
+**Status: Done** — `compare()` method on `PoniardBaseEstimator`.
+
 ### 3.1 Paired fold comparison (pure numpy/scipy)
-- Operate on per-fold scores already in `_experiment_results`.
-- Pairwise tests appropriate for CV folds (document limitations honestly — folds aren't independent).
-- Practical outputs people use:
-  - mean diff + CI
-  - win/tie/loss across folds
-  - simple ranking that resists noise (e.g. mean rank, or CD-diagram data)
+- [x] Operate on per-fold scores already in `_experiment_results`.
+- [x] Pairwise tests appropriate for CV folds (document limitations honestly — folds aren't independent).
+- [x] Practical outputs people use:
+  - [x] mean diff + CI
+  - [x] win/tie/loss across folds
+  - [ ] simple ranking that resists noise (e.g. mean rank, or CD-diagram data)
 
 ### 3.2 API sketch
 
@@ -152,8 +158,8 @@ clf.compare(estimators=["LogisticRegression", "RandomForestClassifier"])
 Returns a small results object / DataFrames: pairwise table + optional ranking summary.
 
 ### 3.3 Honesty in docs
-- State clearly: this is **exploratory comparison**, not a paper-grade multiple-testing shrine.
-- Prefer methods implementable without new deps (paired t on fold scores, Wilcoxon, bootstrap CI — pick one solid default + escape hatches).
+- [x] State clearly: this is **exploratory comparison**, not a paper-grade multiple-testing shrine.
+- [x] Prefer methods implementable without new deps (paired t on fold scores, Wilcoxon, bootstrap CI — pick one solid default + escape hatches).
 
 **Ship when:** User can answer "is RF actually better than LR on this CV, or are we reading noise?" without leaving Poniard.
 
@@ -165,10 +171,12 @@ Returns a small results object / DataFrames: pairwise table + optional ranking s
 
 **Goal:** Practical model choice, not only peak metric.
 
+**Status: Done** — `pareto()` and `best_under()` methods.
+
 ### 4.1 Productize what you already measure
-- `fit_time` / `score_time` already exist in results.
-- Add a first-class view: metric vs log(fit_time) table or Pareto filter.
-- Helpers like "best under T seconds (mean fit_time)" and "within r% of best metric, pick fastest."
+- [x] `fit_time` / `score_time` already exist in results.
+- [x] Add a first-class view: metric vs log(fit_time) table or Pareto filter.
+- [x] Helpers like "best under T seconds (mean fit_time)" and "within r% of best metric, pick fastest."
 
 ### 4.2 API sketch
 
@@ -179,7 +187,7 @@ clf.best_under(seconds=2.0)        # name or row
 ```
 
 ### 4.3 Plot support (optional)
-- Single scatter: time vs metric, dummy annotated — only if cheap.
+- [ ] Single scatter: time vs metric, dummy annotated — only if cheap.
 
 **Ship when:** Choosing "good enough and cheap" is one call, not manual dataframe wrangling.
 
@@ -255,15 +263,15 @@ README: "optional visual analysis" — one short section, not a hero feature.
 
 Break into small releases; each should be install-worthy alone.
 
-| Phase | Theme | Releases as |
+| Phase | Theme | Status |
 |---|---|---|
-| **P0** | Reposition + API diet + README | breaking cleanup |
-| **P1** | Error analysis report wow (universal fails, lift, disagreement) | headline feature |
-| **P2** | Diversity-default ensemble + prediction cache | closed loop |
-| **P3** | Statistical `compare()` | trust layer |
-| **P4** | Pareto / best_under time-quality | practical choice |
-| **P5** | Tune glue redesign **or** deletion | resolve ambivalence |
-| **P6** | Plot pass aligned to P1–P4 | optional polish |
+| **P0** | Reposition + API diet + README | Done |
+| **P1** | Error analysis report wow (universal fails, lift, disagreement) | Done |
+| **P2** | Diversity-default ensemble + prediction cache | Open |
+| **P3** | Statistical `compare()` | Done |
+| **P4** | Pareto / best_under time-quality | Done |
+| **P5** | Tune glue redesign **or** deletion | Done |
+| **P6** | Plot pass aligned to P1–P4 | Open |
 
 Parallelism: P0 first. P1 can start immediately after. P2 depends on similarity + cached preds (partially exists). P3 reads fold scores (exists). P4 is small and can slip between larger phases. P5 last among core so compare/error can support tuned deltas. P6 continuously skims off P1–P4.
 

poniard/error_analysis/__init__.py

@@ -1,3 +1,3 @@
-from poniard.error_analysis.error_analysis import ErrorAnalyzer
+from poniard.error_analysis.error_analysis import ErrorAnalyzer, ErrorReport
 
-__all__ = ["ErrorAnalyzer"]
+__all__ = ["ErrorAnalyzer", "ErrorReport"]