What is the difference between batch, mini-batch, and stochastic gradient descent?
Deep LearningLevel: FoundationNeural networks
What is the difference between batch, mini-batch, and stochastic gradient descent?
Hints
- What changes in one update and which information is used?
- Check the sign, step size and behavior on an ill-conditioned surface.
Answer
Batch GD computes the gradient using the entire training set per update — accurate but slow and memory-heavy. Stochastic GD (SGD) updates using a single example at a time — fast and noisy, which can help escape local minima but makes convergence erratic. Mini-batch GD (the standard in practice) uses small batches (e.g., 32–256 examples) — balances computational efficiency (via vectorization/GPU parallelism) with a reasonable gradient estimate, and the noise acts as a mild regularizer.
Source and licence: Ishan Dutta — Awesome Data Science Interview Q&A · MIT