Skip to content
Quantoros

How do you do A/B testing for LLM applications?

ML System DesignLevel: IntermediateDeployment & Serving

How do you do A/B testing for LLM applications?

Hints
  • Specify the null hypothesis, observation unit and test statistic.
  • Separate effect size, uncertainty and the probability of data under the null.

Answer

LLM A/B testing is harder than standard A/B testing because responses are non-deterministic and quality evaluation requires subjective judgment.

Approach:

  1. Define metrics upfront: CSAT, task completion rate, thumbs up/down, follow-up question rate (indicator of unhelpful first response)
  2. Traffic splitting: route X% of users to variant B (new prompt/model/RAG). Ensure random assignment, not self-selection.
  3. Logging: log experiment ID, user ID, query, response, and all quality metrics per request
  4. Statistical significance: need enough samples for low-frequency quality signals. Plan for minimum 2 weeks, 10K+ queries per variant.
  5. Automatic guardrails: monitor error rate and P99 latency in real-time; auto-rollback if variant B degrades significantly.

Tools: LaunchDarkly / GrowthBook (feature flags for routing), Langfuse (logging), custom dashboards for LLM-specific metrics.

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions