Skip to content
Quantoros

Your LLM-as-judge scores are inconsistent across runs. How do you stabilize them?

LLM & AILevel: IntermediateTroubleshooting Scenarios

Your LLM-as-judge scores are inconsistent across runs. How do you stabilize them?

Hints
  • Identify the stage: pretraining, adaptation, generation or evaluation.
  • Name the inputs, constraints and a case where the approach fails.

Answer

  1. Set temperature=0 for the judge model — eliminates randomness in scoring
  2. Require chain-of-thought reasoning before the score — anchors the judge's evaluation process
  3. Structured output: enforce JSON output schema {"reasoning": "...", "score": 1-5} — prevents parsing failures
  4. Calibration examples: add 3–5 scored examples to the judge prompt showing the expected scoring behavior
  5. Ensemble judges: run each example through 3 different judge calls (or 3 different models); take the median score; flag if standard deviation > 1

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions