Skip to content
Quantoros

Your LLM doesn't admit when it doesn't know the answer. How do you fix it?

LLM & AILevel: IntermediateTroubleshooting Scenarios

Your LLM doesn't admit when it doesn't know the answer. How do you fix it?

Hints
  • Identify the stage: pretraining, adaptation, generation or evaluation.
  • Name the inputs, constraints and a case where the approach fails.

Answer

  1. System prompt calibration: explicitly instruct "If you are not certain, say 'I don't know' instead of guessing"
  2. Include confidence instruction: "Rate your confidence 1–5. If below 3, decline to answer"
  3. RAG with retrieval threshold: if no context was retrieved above a similarity threshold, route to a fallback response
  4. Fine-tuning with abstention examples: train on examples where the correct output is "I don't know"
  5. Evaluator chain: a second LLM call checks whether the answer is grounded in available context

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions