Your eval suite shows great offline scores but users are unhappy in production. What's wrong?
LLM & AILevel: IntermediateTroubleshooting Scenarios
Your eval suite shows great offline scores but users are unhappy in production. What's wrong?
Hints
- Identify the stage: pretraining, adaptation, generation or evaluation.
- Name the inputs, constraints and a case where the approach fails.
Answer
This is a classic evaluation-production distribution mismatch:
- Eval dataset is not representative: collected from cherry-picked examples, not real user queries. Real queries are messier, more ambiguous.
- Metric doesn't capture what users care about: LLM-judge scoring accuracy; users care about helpfulness and brevity. Different things.
- Context mismatch: eval runs queries in isolation; in production, users have multi-turn conversations, and context matters.
- Recency: eval dataset is stale; user queries have evolved (new product features, new terminology).
Fix:
- Continuously sample real production queries to refresh the eval dataset
- Add user-feedback-correlated metrics (thumbs-up rate, follow-up question rate) to eval suite
- Add multi-turn conversation eval scenarios
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0