A random continuous identifier gets high impurity importance but no validation permutation importance. How can this…
Classic MLLevel: IntermediateCardinality bias in tree importance
A random continuous identifier gets high impurity importance but no validation permutation importance. How can this happen?
Hints
- Count available split candidates.
- Training improvement can be accidental.
Answer
Many possible split points let high-cardinality noise improve training impurity by chance. Held-out permutation checks whether the fitted model's reliance actually helps generalization.
Source and licence: scikit-learn 1.7.2 — permutation importance · BSD-3-Clause · answer: Quantoros editorial