What is a guardrail metric, and why do you need them alongside your primary success metric?
StatisticsLevel: FoundationA/B testing and experimentation
What is a guardrail metric, and why do you need them alongside your primary success metric?
Hints
- Identify the random variable, sample and estimated parameter.
- State the assumptions; distinguish an estimate from its uncertainty.
Answer
A guardrail metric monitors for unintended negative side effects of a change, even if the primary metric improves — e.g., a UI change might boost click-through rate (primary metric) while degrading page load time or increasing customer support tickets (guardrails). Without guardrails, you risk shipping a change that "wins" on the metric you optimized for while quietly harming the broader user experience or business health.
Source and licence: Ishan Dutta — Awesome Data Science Interview Q&A · MIT