Skip to content
Quantoros

A random continuous identifier gets high impurity importance but no validation permutation importance. How can this…

Classic MLLevel: IntermediateCardinality bias in tree importance

A random continuous identifier gets high impurity importance but no validation permutation importance. How can this happen?

Hints
  • Count available split candidates.
  • Training improvement can be accidental.

Answer

Many possible split points let high-cardinality noise improve training impurity by chance. Held-out permutation checks whether the fitted model's reliance actually helps generalization.

Source and licence: scikit-learn 1.7.2 — permutation importance · BSD-3-Clause · answer: Quantoros editorial

Related questions