Skip to content
Quantoros

Your eval suite shows great offline scores but users are unhappy in production. What's wrong?

LLM & AILevel: IntermediateTroubleshooting Scenarios

Your eval suite shows great offline scores but users are unhappy in production. What's wrong?

Hints
  • Identify the stage: pretraining, adaptation, generation or evaluation.
  • Name the inputs, constraints and a case where the approach fails.

Answer

This is a classic evaluation-production distribution mismatch:

  1. Eval dataset is not representative: collected from cherry-picked examples, not real user queries. Real queries are messier, more ambiguous.
  2. Metric doesn't capture what users care about: LLM-judge scoring accuracy; users care about helpfulness and brevity. Different things.
  3. Context mismatch: eval runs queries in isolation; in production, users have multi-turn conversations, and context matters.
  4. Recency: eval dataset is stale; user queries have evolved (new product features, new terminology).

Fix:

  • Continuously sample real production queries to refresh the eval dataset
  • Add user-feedback-correlated metrics (thumbs-up rate, follow-up question rate) to eval suite
  • Add multi-turn conversation eval scenarios

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions