Explain what Dropout does and why it helps prevent overfitting.
Deep LearningLevel: IntermediateNeural networks
Explain what Dropout does and why it helps prevent overfitting.
Hints
- Which activations change randomly during training?
- Compare training with inference; how can their signal scales remain comparable?
Answer
Dropout randomly "drops" (zeroes out) a fraction of neurons during each training forward pass, forcing the network to not rely too heavily on any single neuron or co-adapted group of neurons. This acts like training an ensemble of many thinned sub-networks that share weights, and at test time the full network is used (with weights scaled, or using "inverted dropout" during training) to approximate averaging over that ensemble — reducing overfitting, especially in large fully-connected layers.
Source and licence: Ishan Dutta — Awesome Data Science Interview Q&A · MIT