Skip to content
Quantoros

How do you implement CI/CD for an LLM application?

ML System DesignLevel: IntermediateCI/CD for AI Systems

How do you implement CI/CD for an LLM application?

Hints
  • Start with quality, load and latency requirements.
  • Trace data to prediction; identify a failure point and a validation method.

Answer

LLM CI/CD requires testing non-deterministic, subjective-quality outputs — very different from unit tests.

Pipeline stages:

1. Linting & unit tests
   └─ Format, syntax, tool schemas, guardrail logic

2. Offline evaluation
   └─ Run eval suite on golden dataset (100–500 Q&A pairs)
   └─ LLM-as-judge scores each response; compare to baseline
   └─ FAIL if score drops >2% on any critical category

3. Integration tests
   └─ End-to-end flow with real retrieval and LLM calls (dev environment)
   └─ Validate response structure, citation format, latency

4. Canary deploy
   └─ Route 5% of prod traffic to new version
   └─ Monitor for 24h: error rate, latency, user feedback
   └─ Auto-rollback if metrics degrade

5. Full rollout
   └─ Gradually increase from 5% → 25% → 100%

Regression suite: store pairs of (query, expected_behavior_rubric). On each code change, run all pairs and get LLM-as-judge scores. Track scores over time on a dashboard.

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions