How do you evaluate and iterate on prompt quality?
LLM & AILevel: IntermediateProduction Prompt Design
How do you evaluate and iterate on prompt quality?
Hints
- Identify the stage: pretraining, adaptation, generation or evaluation.
- Name the inputs, constraints and a case where the approach fails.
Answer
- Golden dataset — curate 50–200 representative inputs with expected outputs
- Evaluation rubric — define metrics: correctness, format compliance, tone, length
- LLM-as-judge — automate evaluation with a separate model (GPT-4 as judge)
- Change control — always evaluate on golden dataset before deploying a prompt change
- Production sampling — log 1–5% of live requests; periodically human-evaluate a sample
- Regression alerts — if automated eval score drops >5% in production, trigger alert
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0