What is semantic caching and how does it work?
ML System DesignLevel: IntermediateDeployment & Serving
What is semantic caching and how does it work?
Hints
- Start with quality, load and latency requirements.
- Trace data to prediction; identify a failure point and a validation method.
Answer
Semantic caching stores LLM responses and retrieves them when a new query is semantically equivalent to a cached query — not just exact-string match.
def get_response(query: str) -> str:
# 1. Embed the query
query_embedding = embed(query)
# 2. Search cache for similar queries
cached = cache.search(query_embedding, threshold=0.92)
if cached:
return cached.response # instant, free
# 3. Cache miss — call LLM
response = llm.call(query)
cache.store(query_embedding, query, response)
return response
Cache hit rates: 30–60% typical for customer support bots (users ask similar questions); lower for creative/analytical workloads.
Implementations: GPTCache, Langfuse with Redis, Momento Semantic Cache.
Pitfalls: stale cached responses when the underlying knowledge changes; threshold too low → incorrect cache hits; threshold too high → low hit rate.
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0