What is A/B testing a model in production called, and what's the difference between that and shadow deployment /…
ML System DesignLevel: FoundationML system design and MLOps
What is A/B testing a model in production called, and what's the difference between that and shadow deployment / canary deployment?
Hints
- Specify the null hypothesis, observation unit and test statistic.
- Separate effect size, uncertainty and the probability of data under the null.
Answer
A/B testing (champion/challenger) routes a portion of live traffic to the new model and compares business/model metrics against the current production model directly. Shadow deployment runs the new model alongside the production model on live traffic, but its predictions are only logged, not acted on — useful for validating a new model's behavior/latency against real traffic without any user-facing risk, before trusting it enough to run a real A/B test. Canary deployment gradually rolls out the new model to a small percentage of traffic (e.g., 1% → 5% → 25% → 100%), monitoring closely at each stage, primarily to catch operational failures (crashes, latency spikes, errors) early and limit blast radius, rather than to compare business metrics per se.
Source and licence: Ishan Dutta — Awesome Data Science Interview Q&A · MIT