How do you calculate the required sample size for an A/B test?
StatisticsLevel: IntermediateA/B testing and experimentation
How do you calculate the required sample size for an A/B test?
Hints
- Specify the null hypothesis, observation unit and test statistic.
- Separate effect size, uncertainty and the probability of data under the null.
Answer
Sample size depends on: baseline conversion rate (p), minimum detectable effect (MDE, the smallest lift worth detecting), significance level (α, usually 0.05), and desired power (1−β, usually 0.8). For a two-proportion test, the formula is approximately:
n ≈ 2 × (z_{α/2} + z_β)² × p(1−p) / MDE²
Smaller MDEs and higher power requirements both increase required sample size substantially — halving the MDE roughly quadruples the required sample size, which is a key intuition to communicate to stakeholders who want to detect tiny effects quickly.
Source and licence: Ishan Dutta — Awesome Data Science Interview Q&A · MIT