How would you run an A/B test when you can't randomize at the individual user level (e.g., a pricing change visible to…
StatisticsLevel: IntermediateA/B testing and experimentation
How would you run an A/B test when you can't randomize at the individual user level (e.g., a pricing change visible to all users in a region)?
Hints
- Specify the null hypothesis, observation unit and test statistic.
- Separate effect size, uncertainty and the probability of data under the null.
Answer
Use cluster randomization at whatever level prevents contamination — e.g., randomize by geographic region, city, or store, rather than by individual user. Analysis must then account for the reduced effective sample size (clusters, not individual users, are the true unit of randomization) using cluster-robust standard errors or mixed-effects models, otherwise you'll understate variance and overstate significance. When true randomization isn't feasible at all, consider quasi-experimental designs like difference-in-differences with a comparable control region.
Source and licence: Ishan Dutta — Awesome Data Science Interview Q&A · MIT