What is LLM tracing and why is it important?
ML System DesignLevel: IntermediateObservability & Monitoring
What is LLM tracing and why is it important?
Hints
- Start with quality, load and latency requirements.
- Trace data to prediction; identify a failure point and a validation method.
Answer
LLM tracing captures the full execution tree of an LLM request — every prompt sent, tool called, retrieval done, and response generated — as a structured trace.
Trace: user_query="How do I reset my password?"
├─ span: intent_classification (12ms)
│ └─ prompt: "Classify: 'How do I reset...'" → "account_help"
├─ span: rag_retrieval (145ms)
│ ├─ embedding: "password reset" → [...]
│ └─ vector_search: top_5_chunks retrieved
└─ span: llm_generation (2340ms)
├─ prompt_tokens: 1247
├─ completion_tokens: 312
└─ response: "To reset your password..."
Why tracing matters:
- Debug failures: which step caused the wrong answer?
- Optimize latency: which span is the bottleneck?
- Cost attribution: which features consume most tokens?
- Detect drift: has retrieval quality changed since last deployment?
Tools: Langfuse (open source), Arize AI, LangSmith, Weights & Biases (Weave).
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0