What is Prefix Tuning and Prompt Tuning?
LLM & AILevel: IntermediatePEFT Methods
What is Prefix Tuning and Prompt Tuning?
Hints
- Identify the stage: pretraining, adaptation, generation or evaluation.
- Name the inputs, constraints and a case where the approach fails.
Answer
Prompt Tuning: prepend trainable "soft" token embeddings to the input. Only these embeddings are updated, not any model weights. Extremely parameter-efficient (~10K–100K params). Quality scales with model size — works well for models >10B.
Prefix Tuning: extends prompt tuning by prepending trainable embeddings to each Transformer layer's keys and values, not just the input. More expressive than prompt tuning. Used for encoder-decoder models.
Compared to LoRA: LoRA generally achieves better quality at comparable parameter counts and has become the dominant PEFT method. Prefix/prompt tuning is interesting for academic research but rarely used in production.
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0