What is prompt compression and how does it work?
ML System DesignLevel: IntermediateCost Optimization
What is prompt compression and how does it work?
Hints
- Start with quality, load and latency requirements.
- Trace data to prediction; identify a failure point and a validation method.
Answer
Prompt compression reduces the length of prompts before sending to the LLM, saving tokens without losing critical information.
Techniques:
-
LLMLingua / LLMLingua-2: small LM assigns perplexity scores to each token; tokens above a surprise threshold are dropped. Achieves 2–5× compression with <5% quality loss.
-
Selective summarization: compress conversation history older than the last K turns. Keep recent turns verbatim; summarize everything older.
-
RAG chunk reranking: out of 20 retrieved chunks, only include the top 5 most relevant. Token savings proportional to excluded chunks.
-
Prompt distillation: experiment with shorter system prompts that achieve the same behavior; often 50% of system prompt is redundant.
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0