What variance threshold removes binary features constant in more than 80% of rows?
Classic MLLevel: IntermediateBinary variance filtering
What variance threshold removes binary features constant in more than 80% of rows?
Hints
- Binary variance is symmetric in p and 1-p.
- Low prevalence does not imply no predictive value.
Answer
A Bernoulli feature has variance p(1-p), so the boundary is 0.8*0.2=0.16. The filter uses no labels; a rare but useful indicator may still be discarded.
Source and licence: scikit-learn 1.7.2 — feature selection · BSD-3-Clause · answer: Quantoros editorial