Skip to content
Quantoros

What is the difference between batch, mini-batch, and stochastic gradient descent?

Deep LearningLevel: FoundationNeural networks

What is the difference between batch, mini-batch, and stochastic gradient descent?

Hints
  • What changes in one update and which information is used?
  • Check the sign, step size and behavior on an ill-conditioned surface.

Answer

Batch GD computes the gradient using the entire training set per update — accurate but slow and memory-heavy. Stochastic GD (SGD) updates using a single example at a time — fast and noisy, which can help escape local minima but makes convergence erratic. Mini-batch GD (the standard in practice) uses small batches (e.g., 32–256 examples) — balances computational efficiency (via vectorization/GPU parallelism) with a reasonable gradient estimate, and the noise acts as a mild regularizer.

Source and licence: Ishan Dutta — Awesome Data Science Interview Q&A · MIT

Related questions