Skip to content
Quantoros

What is semantic caching and how does it work?

ML System DesignLevel: IntermediateDeployment & Serving

What is semantic caching and how does it work?

Hints
  • Start with quality, load and latency requirements.
  • Trace data to prediction; identify a failure point and a validation method.

Answer

Semantic caching stores LLM responses and retrieves them when a new query is semantically equivalent to a cached query — not just exact-string match.

def get_response(query: str) -> str:
    # 1. Embed the query
    query_embedding = embed(query)
    
    # 2. Search cache for similar queries
    cached = cache.search(query_embedding, threshold=0.92)
    if cached:
        return cached.response  # instant, free
    
    # 3. Cache miss — call LLM
    response = llm.call(query)
    cache.store(query_embedding, query, response)
    return response

Cache hit rates: 30–60% typical for customer support bots (users ask similar questions); lower for creative/analytical workloads.

Implementations: GPTCache, Langfuse with Redis, Momento Semantic Cache.

Pitfalls: stale cached responses when the underlying knowledge changes; threshold too low → incorrect cache hits; threshold too high → low hit rate.

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions