Skip to content

feat: overhaul error analysis (roadmap 1.2, 3.1-3.3) - #41

Merged
rxavier merged 2 commits into
mainfrom
fix/error-analyzer-bugs
Jul 31, 2026
Merged

feat: overhaul error analysis (roadmap 1.2, 3.1-3.3)#41
rxavier merged 2 commits into
mainfrom
fix/error-analyzer-bugs

Conversation

@rxavier

@rxavier rxavier commented Jul 31, 2026

Copy link
Copy Markdown
Owner

Summary

Overhauls ErrorAnalyzer to make it a first-class feature (roadmap items 1.2 — fix bugs in differentiated modules, 3.1 — test coverage, 3.2 — data passing, 3.3 — documentation).

Changes

Bug fixes

  • analyze_features called _train_test_split_from_cv() with no X, y (would NameError); now passes them explicitly.
  • Positional-vs-name index mismatch in analyze_features: feature selection is now unified on positional indices; a positional int is never compared to a column name.
  • _compute_predictions/rank_errors/analyze_target reached into self._poniard.X/y which were never stored on the estimator; they now require X/y explicitly (ErrorAnalyzer stays stateless).
  • _rank_errors_continuous_multioutput broadcast bug (np.std(y, axis=0) returned a Series → pandas alignment failure).
  • Multiclass ranking: pandas upcast row['y'] to float → KeyError: 'proba_2.0'.
  • Standalone analyze_features iterated .feature_types dict without .items().

Redesign

  • Return contract: rank_errors / merge_errors return plain DataFrames (dropped the {"values", "idx"} wrapper). merge_errors accepts a dict of DataFrames or a single DataFrame.
  • Regression error definition: samples flagged by |residual| > (1 - error_quantile) quantile threshold (default worst 10%), replacing the global ±1 std band. Per-target thresholds for multioutput.
  • Fold-consistent importances: permutation importances averaged over the Poniard estimator's CV folds instead of a random 80/20 split.
  • Interpretation: analyze_target returns error_count / target_count / error_rate per class/bin (always a DataFrame; as_ratio/wrt_target removed). Categorical features in analyze_features get an error-rate-per-category table.
  • Orchestrator: new analyze(X, y, ...) returns a single report with ranked_errors, merged_errors, summary, by_target, by_feature.

Tests & docs

  • New tests/test_error_analysis.py with 33 tests (all target types, error paths, report structure, quantile threshold).
  • ErrorAnalyzer section added to README.

Breaking changes

  • rank_errors/merge_errors return DataFrames instead of {"values", "idx"} dicts.
  • analyze_target drops as_ratio / wrt_target; now always returns a DataFrame with error_rate.
  • analyze_features requires X (and y when estimator_name is used).
  • rank_errors requires X and y when built via from_poniard.

Tests

pytest — 128 passed (95 existing + 33 new).

rxavier added 2 commits July 31, 2026 01:55
Fix ErrorAnalyzer bugs and make it a first-class feature:
- rank_errors/merge_errors/analyze_target now return plain DataFrames,
  dropping the redundant {'values', 'idx'} wrapper.
- require X/y explicitly instead of reaching into the estimator (which
  never stored them); rank_errors computes out-of-fold predictions.
- regression errors use a configurable quantile threshold (error_quantile)
  instead of the global +/-1 std band; consistent for multioutput.
- permutation importances are computed over the Poniard CV folds instead
  of a random 80/20 split.
- analyze_target returns error_count/target_count/error_rate per
  class/bin; categorical features get an error-rate table.
- add analyze() orchestrator returning a single structured report.
- fix latent multiclass label upcast and standalone feature-types bugs.
- add 33 tests covering rank/merge/target/feature analysis and the
  report; document ErrorAnalyzer in the README.
Add deterministic standalone tests where errors are confined to a known
class / category / numeric range, asserting analyze_target and
analyze_features correctly surface them via error_rate and group means.
Also assert permutation-importance feature selection is deterministic.
@rxavier
rxavier merged commit 9614ea9 into main Jul 31, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant