Your LLM-as-judge scores are inconsistent across runs. How do you stabilize them?
LLM & AILevel: IntermediateTroubleshooting Scenarios
Your LLM-as-judge scores are inconsistent across runs. How do you stabilize them?
Hints
- Identify the stage: pretraining, adaptation, generation or evaluation.
- Name the inputs, constraints and a case where the approach fails.
Answer
- Set temperature=0 for the judge model — eliminates randomness in scoring
- Require chain-of-thought reasoning before the score — anchors the judge's evaluation process
- Structured output: enforce JSON output schema
{"reasoning": "...", "score": 1-5}— prevents parsing failures - Calibration examples: add 3–5 scored examples to the judge prompt showing the expected scoring behavior
- Ensemble judges: run each example through 3 different judge calls (or 3 different models); take the median score; flag if standard deviation > 1
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0