Skip to content
Quantoros

A feature has high permutation importance on training data but none on held-out data. What should this suggest?

Classic MLLevel: IntermediateGeneralizing feature reliance

A feature has high permutation importance on training data but none on held-out data. What should this suggest?

Hints
  • Compare reliance with out-of-sample utility.
  • A memorized pattern can be training-important.

Answer

The model may rely on a pattern that does not generalize. Inspect leakage, spurious fit and data shift; importance on training data alone does not establish predictive value in deployment.

Source and licence: scikit-learn 1.7.2 — permutation importance · BSD-3-Clause · answer: Quantoros editorial

Related questions