What is "peeking" in A/B testing, and why is it a statistical problem?
StatisticsLevel: FoundationA/B testing and experimentation
What is "peeking" in A/B testing, and why is it a statistical problem?
Hints
- Specify the null hypothesis, observation unit and test statistic.
- Separate effect size, uncertainty and the probability of data under the null.
Answer
Peeking is repeatedly checking test results before the pre-determined sample size/duration is reached and stopping as soon as the result looks significant. This inflates the false positive rate far beyond the nominal α, because each additional look is another chance for random noise to cross the significance threshold — with enough peeks, you're almost guaranteed to see a "significant" result eventually even with no true effect. Fixes: commit to a fixed sample size/duration in advance, or use sequential testing methods (e.g., mSPRT, Bayesian sequential testing) explicitly designed to allow valid continuous monitoring.
Source and licence: Ishan Dutta — Awesome Data Science Interview Q&A · MIT