Skip to content
Quantoros

What is SGD — stochastic gradient descent? What’s the difference with the usual gradient descent?

Classic MLLevel: IntermediateLinear regression

What is SGD — stochastic gradient descent? What’s the difference with the usual gradient descent?

Hints
  • What changes in one update and which information is used?
  • Check the sign, step size and behavior on an ill-conditioned surface.

Answer

In both gradient descent (GD) and stochastic gradient descent (SGD), you update a set of parameters in an iterative manner to minimize an error function.

The difference lies in how the gradient of the loss function is estimated. In the usual GD, you have to run through ALL the samples in your training set in order to estimate the gradient and do a single update for a parameter in a particular iteration. In SGD, on the other hand, you use ONLY ONE or SUBSET of training sample from your training set to estimate the gradient and do the update for a parameter in a particular iteration. If you use SUBSET, it is called Minibatch Stochastic gradient Descent.

Source and licence: Alexey Grigorev & contributors — Data Science Interviews · CC-BY-4.0

Related questions