Skip to content
Quantoros

How do you calculate the required sample size for an A/B test?

StatisticsLevel: IntermediateA/B testing and experimentation

How do you calculate the required sample size for an A/B test?

Hints
  • Specify the null hypothesis, observation unit and test statistic.
  • Separate effect size, uncertainty and the probability of data under the null.

Answer

Sample size depends on: baseline conversion rate (p), minimum detectable effect (MDE, the smallest lift worth detecting), significance level (α, usually 0.05), and desired power (1−β, usually 0.8). For a two-proportion test, the formula is approximately:

n ≈ 2 × (z_{α/2} + z_β)² × p(1−p) / MDE²

Smaller MDEs and higher power requirements both increase required sample size substantially — halving the MDE roughly quadruples the required sample size, which is a key intuition to communicate to stakeholders who want to detect tiny effects quickly.

Source and licence: Ishan Dutta — Awesome Data Science Interview Q&A · MIT

Related questions