Design an AI customer support chatbot for an e-commerce company handling 100K queries/day.
ML System DesignLevel: IntermediateDesign Customer Support Chatbot
Design an AI customer support chatbot for an e-commerce company handling 100K queries/day.
Hints
- Start with quality, load and latency requirements.
- Trace data to prediction; identify a failure point and a validation method.
Answer
Requirements clarification:
- Real-time chat (streaming responses), P95 < 3s
- Must access order status, shipping, return policies
- Escalation to human agent when needed
- Multi-language support
- Compliance: no PII in LLM prompts; SOC 2
Architecture:
User Browser
↓ WebSocket
API Gateway (rate limiting, auth)
↓
Session Service (Redis — conversation history)
↓
Intent Router ──→ [Order Status] → Order DB API
↓ [Policy Q&A] → RAG Pipeline
↓ [Complaints] → Human Queue (Zendesk API)
↓ [General] → LLM with RAG
↓
LLM Service (streaming)
↓
Response Filter (PII masking, guardrails)
↓
User
RAG pipeline:
- Knowledge base: product docs, FAQs, return policies (Markdown → chunked → embedded)
- Vector DB: Pinecone with namespace per region/language
- Hybrid search: dense (semantic) + sparse (keyword) for product SKUs and order IDs
PII protection:
- Order ID / user email extracted from session auth token — never passed raw to LLM
- LLM only sees: "User with order status: SHIPPED, ETA: Dec 5". Never raw PII.
Escalation logic:
if sentiment_score < threshold or confidence < 0.7 or "speak to agent" in query:
route_to_human(session_id, conversation_summary)
Monitoring: CSAT score, deflection rate (% resolved without human), mean handle time, escalation rate, P95 latency.
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0