Skip to content
Quantoros

What is a multiple comparisons problem, and how do you correct for it when testing many metrics/variants at once?

StatisticsLevel: FoundationA/B testing and experimentation

What is a multiple comparisons problem, and how do you correct for it when testing many metrics/variants at once?

Hints
  • Identify the random variable, sample and estimated parameter.
  • State the assumptions; distinguish an estimate from its uncertainty.

Answer

If you test many metrics (or many variants) simultaneously, each at α = 0.05, the probability of at least one false positive across all tests rises well above 5% — e.g., testing 20 independent metrics gives roughly a 64% chance of at least one spurious "significant" result. Corrections include the conservative Bonferroni correction (divide α by the number of tests) or the less conservative, more powerful Benjamini-Hochberg (False Discovery Rate) procedure, which controls the expected proportion of false positives among significant results rather than the probability of any false positive at all.

Source and licence: Ishan Dutta — Awesome Data Science Interview Q&A · MIT

Related questions