Skip to content
Quantoros

What is Prefix Tuning and Prompt Tuning?

LLM & AILevel: IntermediatePEFT Methods

What is Prefix Tuning and Prompt Tuning?

Hints
  • Identify the stage: pretraining, adaptation, generation or evaluation.
  • Name the inputs, constraints and a case where the approach fails.

Answer

Prompt Tuning: prepend trainable "soft" token embeddings to the input. Only these embeddings are updated, not any model weights. Extremely parameter-efficient (~10K–100K params). Quality scales with model size — works well for models >10B.

Prefix Tuning: extends prompt tuning by prepending trainable embeddings to each Transformer layer's keys and values, not just the input. More expressive than prompt tuning. Used for encoder-decoder models.

Compared to LoRA: LoRA generally achieves better quality at comparable parameter counts and has become the dominant PEFT method. Prefix/prompt tuning is interesting for academic research but rarely used in production.

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions