Keep permutation importance when coefficients fail; add importance_max_samples

Permutation importance came back empty for coxph, coxnet, fssvm, and
weibull_aft whenever the pipeline's selector step exposed no
selected_features_ (a chained feature_selector=[...] wraps a nested
Pipeline, and third-party selectors do not have the attribute at all). sklearn's
permutation_importance actually succeeded every time; the ValueError came
from the coefficient step that ran right after it inside the same try, and
the handler threw the computed importance away. Only those four models were
affected because they are the only wrappers that override
get_coefficients(); every other model short-circuits to a correct-length
NaN list and never reaches the bad path.

Splitting the work into two phases fixes that: a permutation failure still
returns the empty sentinel, but a coefficient problem now degrades only the
coefficient column to NaN. _get_coefficients is total, so it cannot hand a
wrong-length list to the DataFrame builder, and _get_selected_features
resolves names through a nested selector Pipeline and through
get_feature_names_out().

The same work turned up a silent wrong answer. _align_coefficients keyed
values positionally by the selector's order, but lifelines sorts Weibull-AFT
covariates, so on the ordinary feature_selector='topk' path every
weibull_aft coefficient landed on the wrong feature, sign flips included, in
both the report and the eval_importance sheet. Coefficients are now keyed by
the model's own index, which is what evaluation/_coefficients.py already
did, so the two surfaces agree.

Failures that remain are visible. log_metric_failure was resolving to
InternalWarning, which configure_default_filters ignores by default, so the
user saw a bare "failed (2.3s)." and nothing else. It now raises a
MetricFailureWarning naming the model and the exception type, and the
progress line carries the type too.

Also adds importance_max_samples, an opt-in cap on the test rows used for
permutation importance. The C-index term is superlinear in n_test, so on
large cohorts importance can out-cost model fitting. Default None keeps
every existing run byte-for-byte identical; the draw is seeded from
random_state so models stay comparable, and an unseeded draw warns. Threaded
through evaluate(), run(), tune(), and importance().

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: afpybus

docs/feature-analysis.md

@@ -35,6 +35,23 @@ print(result.evaluation.importance)  # DataFrame: model_type, feature, coefficie
 learner.importance('rsf', method='permutation', n_importance_repeats=50)
 ```
 
+#### Cost on large cohorts
+
+Permutation importance costs `n_features x n_repeats x (predict + C-index over n_test rows)`. The C-index term is superlinear in the number of test rows, so on large cohorts importance can take longer than fitting the models. Two levers:
+
+```python
+# 1. Compute importance on a capped, seeded subsample of the test set.
+learner.setup(random_state=42)   # the cap draws from this seed
+learner.evaluate(importance_max_samples=1000)
+
+# 2. Skip it during an expensive sweep, then compute it on demand later,
+#    including from a learner reloaded with load_learner().
+learner.evaluate(n_importance_repeats=0)
+learner.importance('coxph', method='permutation')
+```
+
+`importance_max_samples` defaults to `None`, which uses every row. A value at or above the cohort size is a no-op, and the minimum is 2 (the C-index needs a pair of rows to compare). The draw is seeded from `random_state`, so with a seed set all features within a model, and all models in the call, are scored on the same rows and their importances stay comparable. Without a seed each model draws its own subsample; mlsurv warns when that happens. Permutation importance is a ranking diagnostic that tolerates subsampling well: the cost is a little precision in the importance estimate, not in the model.
+
 ### Grouped Importance
 
 For one-hot encoded categorical features, grouped importance permutes all dummy columns jointly and reports a single aggregated score per original variable<sup>[2](#references),[3](#references)</sup>:

mlsurv/evaluation/_config.py

@@ -52,6 +52,8 @@ class EvalConfig:
     call layers unchanged.
     """
     n_importance_repeats: int = 10
+    # Cap on the test rows used for permutation importance. None = all rows.
+    importance_max_samples: Optional[int] = None
     eval_times: Optional[np.ndarray] = None
     random_state: Optional[int] = None
     include_shap: bool = False
@@ -81,6 +83,20 @@ class EvalConfig:
     gnd_n_bins: int = 10
 
     def __post_init__(self):
+        if self.importance_max_samples is not None:
+            if (isinstance(self.importance_max_samples, bool)
+                    or not isinstance(self.importance_max_samples,
+                                      (int, np.integer))):
+                raise ValueError(
+                    f"importance_max_samples must be an int or None, got "
+                    f"{type(self.importance_max_samples).__name__}."
+                )
+            # Lower bound of 2 explained in _subsample_for_importance.
+            if self.importance_max_samples < 2:
+                raise ValueError(
+                    f"importance_max_samples must be >= 2 or None, got "
+                    f"{self.importance_max_samples}."
+                )
         if self.include_shap:
             if self.shap_max_background < 1:
                 raise ValueError(

mlsurv/evaluation/evaluate.py

@@ -243,6 +243,7 @@ def evaluate(
     eval_times=None,
     extra_eval_times=None,
     n_importance_repeats: Union[int, str] = "auto",
+    importance_max_samples: Optional[int] = None,
     include_shap=False,
     shap_max_background=50,
     shap_max_explain=50,
@@ -318,6 +319,12 @@ def evaluate(
         ``"auto"`` (the default), uses 10 repeats for all models.
         Pass a dict of {model_name: int} for per-model repeat counts;
         models not in the dict fall back to ``"auto"``.
+    importance_max_samples : int, optional
+        Cap on the test rows used for permutation importance. ``None``
+        (default) uses every row. A value at or above the cohort size is a
+        no-op; the minimum is 2. The draw is seeded from *random_state*, so
+        with a seed set every model sees the same rows; without one, each
+        model draws its own subsample and mlsurv warns.
     include_shap : bool, default=False
         If True, compute SHAP values (requires shap package).
     shap_max_background : int, default=50
@@ -456,6 +463,7 @@ def evaluate(
             eval_times=eval_times,
             extra_eval_times=extra_eval_times,
             n_importance_repeats=resolved_repeats,
+            importance_max_samples=importance_max_samples,
             include_shap=include_shap,
             shap_max_background=shap_max_background,
             shap_max_explain=shap_max_explain,
@@ -514,6 +522,7 @@ def evaluate(
         eval_times=eval_times,
         extra_eval_times=extra_eval_times,
         n_importance_repeats=single_repeats,
+        importance_max_samples=importance_max_samples,
         include_shap=include_shap,
         shap_max_background=shap_max_background,
         shap_max_explain=shap_max_explain,
@@ -551,6 +560,7 @@ def _evaluate_single(
     eval_times=None,
     extra_eval_times=None,
     n_importance_repeats=DEFAULT_N_IMPORTANCE_REPEATS,
+    importance_max_samples=None,
     include_shap=False,
     shap_max_background=50,
     shap_max_explain=50,
@@ -662,6 +672,7 @@ def _evaluate_single(
     # Bundle eval config for helper calls
     config = EvalConfig(
         n_importance_repeats=n_importance_repeats,
+        importance_max_samples=importance_max_samples,
         eval_times=eval_times,
         random_state=random_state,
         include_shap=include_shap,
@@ -861,6 +872,7 @@ def _run_importance_phase(model, X_test, y_test, config, model_name,
             show_progress=config.show_progress, model_name=model_name,
             n_jobs=importance_n_jobs,
             categorical_groups=config.categorical_groups,
+            max_samples=config.importance_max_samples,
         )
     else:
         importance_df, iterations_df = _empty_importance_results()
@@ -1695,6 +1707,7 @@ def _evaluate_multiple(
     eval_times=None,
     extra_eval_times=None,
     n_importance_repeats=DEFAULT_N_IMPORTANCE_REPEATS,
+    importance_max_samples=None,
     include_shap=False,
     shap_max_background=50,
     shap_max_explain=50,
@@ -1821,6 +1834,7 @@ def _evaluate_multiple(
                 eval_times=eval_times,
                 extra_eval_times=extra_eval_times,
                 n_importance_repeats=model_repeats,
+                importance_max_samples=importance_max_samples,
                 include_shap=include_shap,
                 shap_max_background=shap_max_background,
                 shap_max_explain=shap_max_explain,