A feature has high permutation importance on training data but none on held-out data. What should this suggest?
Classic MLLevel: IntermediateGeneralizing feature reliance
A feature has high permutation importance on training data but none on held-out data. What should this suggest?
Hints
- Compare reliance with out-of-sample utility.
- A memorized pattern can be training-important.
Answer
The model may rely on a pattern that does not generalize. Inspect leakage, spurious fit and data shift; importance on training data alone does not establish predictive value in deployment.
Source and licence: scikit-learn 1.7.2 — permutation importance · BSD-3-Clause · answer: Quantoros editorial