What is a multiple comparisons problem, and how do you correct for it when testing many metrics/variants at once?
StatisticsLevel: FoundationA/B testing and experimentation
What is a multiple comparisons problem, and how do you correct for it when testing many metrics/variants at once?
Hints
- Identify the random variable, sample and estimated parameter.
- State the assumptions; distinguish an estimate from its uncertainty.
Answer
If you test many metrics (or many variants) simultaneously, each at α = 0.05, the probability of at least one false positive across all tests rises well above 5% — e.g., testing 20 independent metrics gives roughly a 64% chance of at least one spurious "significant" result. Corrections include the conservative Bonferroni correction (divide α by the number of tests) or the less conservative, more powerful Benjamini-Hochberg (False Discovery Rate) procedure, which controls the expected proportion of false positives among significant results rather than the probability of any false positive at all.
Source and licence: Ishan Dutta — Awesome Data Science Interview Q&A · MIT