How do you do A/B testing for LLM applications?
ML System DesignLevel: IntermediateDeployment & Serving
How do you do A/B testing for LLM applications?
Hints
- Specify the null hypothesis, observation unit and test statistic.
- Separate effect size, uncertainty and the probability of data under the null.
Answer
LLM A/B testing is harder than standard A/B testing because responses are non-deterministic and quality evaluation requires subjective judgment.
Approach:
- Define metrics upfront: CSAT, task completion rate, thumbs up/down, follow-up question rate (indicator of unhelpful first response)
- Traffic splitting: route X% of users to variant B (new prompt/model/RAG). Ensure random assignment, not self-selection.
- Logging: log experiment ID, user ID, query, response, and all quality metrics per request
- Statistical significance: need enough samples for low-frequency quality signals. Plan for minimum 2 weeks, 10K+ queries per variant.
- Automatic guardrails: monitor error rate and P99 latency in real-time; auto-rollback if variant B degrades significantly.
Tools: LaunchDarkly / GrowthBook (feature flags for routing), Langfuse (logging), custom dashboards for LLM-specific metrics.
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0