Skip to content
Quantoros

What is prompt compression and how does it work?

ML System DesignLevel: IntermediateCost Optimization

What is prompt compression and how does it work?

Hints
  • Start with quality, load and latency requirements.
  • Trace data to prediction; identify a failure point and a validation method.

Answer

Prompt compression reduces the length of prompts before sending to the LLM, saving tokens without losing critical information.

Techniques:

  1. LLMLingua / LLMLingua-2: small LM assigns perplexity scores to each token; tokens above a surprise threshold are dropped. Achieves 2–5× compression with <5% quality loss.

  2. Selective summarization: compress conversation history older than the last K turns. Keep recent turns verbatim; summarize everything older.

  3. RAG chunk reranking: out of 20 retrieved chunks, only include the top 5 most relevant. Token savings proportional to excluded chunks.

  4. Prompt distillation: experiment with shorter system prompts that achieve the same behavior; often 50% of system prompt is redundant.

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions