Skip to content
Quantoros

How do we use SGD (stochastic gradient descent) for training a neural net?

Deep LearningLevel: IntermediateOptimization in neural networks

How do we use SGD (stochastic gradient descent) for training a neural net?

Hints
  • What changes in one update and which information is used?
  • Check the sign, step size and behavior on an ill-conditioned surface.

Answer

SGD approximates the expectation with few randomly selected samples (instead of the full data). In comparison to batch gradient descent, we can efficiently approximate the expectation in large data sets using SGD. For neural networks this reduces the training time a lot even considering that it will converge later as the random sampling adds noise to the gradient descent.

Source and licence: Alexey Grigorev & contributors — Data Science Interviews · CC-BY-4.0

Related questions