Skip to content
Quantoros

What is LLM tracing and why is it important?

ML System DesignLevel: IntermediateObservability & Monitoring

What is LLM tracing and why is it important?

Hints
  • Start with quality, load and latency requirements.
  • Trace data to prediction; identify a failure point and a validation method.

Answer

LLM tracing captures the full execution tree of an LLM request — every prompt sent, tool called, retrieval done, and response generated — as a structured trace.

Trace: user_query="How do I reset my password?"
  ├─ span: intent_classification (12ms)
  │   └─ prompt: "Classify: 'How do I reset...'" → "account_help"
  ├─ span: rag_retrieval (145ms)
  │   ├─ embedding: "password reset" → [...]
  │   └─ vector_search: top_5_chunks retrieved
  └─ span: llm_generation (2340ms)
      ├─ prompt_tokens: 1247
      ├─ completion_tokens: 312
      └─ response: "To reset your password..."

Why tracing matters:

  • Debug failures: which step caused the wrong answer?
  • Optimize latency: which span is the bottleneck?
  • Cost attribution: which features consume most tokens?
  • Detect drift: has retrieval quality changed since last deployment?

Tools: Langfuse (open source), Arize AI, LangSmith, Weights & Biases (Weave).

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions