Skip to content
Quantoros

Design an AI customer support chatbot for an e-commerce company handling 100K queries/day.

ML System DesignLevel: IntermediateDesign Customer Support Chatbot

Design an AI customer support chatbot for an e-commerce company handling 100K queries/day.

Hints
  • Start with quality, load and latency requirements.
  • Trace data to prediction; identify a failure point and a validation method.

Answer

Requirements clarification:

  • Real-time chat (streaming responses), P95 < 3s
  • Must access order status, shipping, return policies
  • Escalation to human agent when needed
  • Multi-language support
  • Compliance: no PII in LLM prompts; SOC 2

Architecture:

User Browser
    ↓ WebSocket
API Gateway (rate limiting, auth)
    ↓
Session Service (Redis — conversation history)
    ↓
Intent Router ──→ [Order Status] → Order DB API
                ↓ [Policy Q&A] → RAG Pipeline
                ↓ [Complaints] → Human Queue (Zendesk API)
                ↓ [General] → LLM with RAG
    ↓
LLM Service (streaming)
    ↓
Response Filter (PII masking, guardrails)
    ↓
User

RAG pipeline:

  • Knowledge base: product docs, FAQs, return policies (Markdown → chunked → embedded)
  • Vector DB: Pinecone with namespace per region/language
  • Hybrid search: dense (semantic) + sparse (keyword) for product SKUs and order IDs

PII protection:

  • Order ID / user email extracted from session auth token — never passed raw to LLM
  • LLM only sees: "User with order status: SHIPPED, ETA: Dec 5". Never raw PII.

Escalation logic:

if sentiment_score < threshold or confidence < 0.7 or "speak to agent" in query:
    route_to_human(session_id, conversation_summary)

Monitoring: CSAT score, deflection rate (% resolved without human), mean handle time, escalation rate, P95 latency.

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions