What is transfer learning, and when would you fine-tune vs. use a model as a fixed feature extractor?
Deep LearningLevel: FoundationNeural networks
What is transfer learning, and when would you fine-tune vs. use a model as a fixed feature extractor?
Hints
- Which parameters change and which stay frozen?
- Separate memory for weights, gradients, optimizer states and activations.
Answer
Transfer learning reuses a model pretrained on a large dataset (e.g., ImageNet, or a large text corpus) as a starting point for a new, often smaller-data task, since early/general layers tend to learn transferable features (edges, textures, or general language patterns). Use it as a fixed feature extractor (freeze pretrained layers, train only a new head) when your target dataset is small and similar to the source domain. Fine-tune (unfreeze some/all layers with a low learning rate) when you have more target data or the target domain differs meaningfully from the source, allowing the model to adapt its learned representations.
Source and licence: Ishan Dutta — Awesome Data Science Interview Q&A · MIT