What is the difference between a discriminative fine-tuning approach and prompting/in-context learning for large…
Deep LearningLevel: FoundationNeural networks
What is the difference between a discriminative fine-tuning approach and prompting/in-context learning for large language models?
Hints
- Which parameters change and which stay frozen?
- Separate memory for weights, gradients, optimizer states and activations.
Answer
Fine-tuning updates the model's weights on task-specific labeled data (fully or via parameter-efficient methods like LoRA), permanently specializing the model for that task — requires labeled data and compute but tends to yield strong, consistent task performance. In-context learning/prompting provides task instructions and/or a few examples directly in the input at inference time without updating any weights — cheap and flexible (no training needed), leveraging patterns the model already learned during pretraining, but performance can be less consistent and is sensitive to prompt phrasing and example selection.
Source and licence: Ishan Dutta — Awesome Data Science Interview Q&A · MIT