What does in-context learning mean, and what makes it different from fine-tuning?
LLM & AILevel: FoundationPrompting & In-Context Learning
What does in-context learning mean, and what makes it different from fine-tuning?
Hints
- Which parameters change and which stay frozen?
- Separate memory for weights, gradients, optimizer states and activations.
Answer
In-context learning is a model's ability to do a new task purely from the instructions or examples in the prompt, with no weight updates at all. Fine-tuning permanently changes the weights on a labelled dataset.
Source and licence: Skillumen — Awesome AI Engineer Interview Questions · CC0-1.0