How do you implement CI/CD for an LLM application?
ML System DesignLevel: IntermediateCI/CD for AI Systems
How do you implement CI/CD for an LLM application?
Hints
- Start with quality, load and latency requirements.
- Trace data to prediction; identify a failure point and a validation method.
Answer
LLM CI/CD requires testing non-deterministic, subjective-quality outputs — very different from unit tests.
Pipeline stages:
1. Linting & unit tests
└─ Format, syntax, tool schemas, guardrail logic
2. Offline evaluation
└─ Run eval suite on golden dataset (100–500 Q&A pairs)
└─ LLM-as-judge scores each response; compare to baseline
└─ FAIL if score drops >2% on any critical category
3. Integration tests
└─ End-to-end flow with real retrieval and LLM calls (dev environment)
└─ Validate response structure, citation format, latency
4. Canary deploy
└─ Route 5% of prod traffic to new version
└─ Monitor for 24h: error rate, latency, user feedback
└─ Auto-rollback if metrics degrade
5. Full rollout
└─ Gradually increase from 5% → 25% → 100%
Regression suite: store pairs of (query, expected_behavior_rubric). On each code change, run all pairs and get LLM-as-judge scores. Track scores over time on a dashboard.
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0