What is gradient descent? How does it work?
Classic MLLevel: IntermediateLinear regression
What is gradient descent? How does it work?
Hints
- What changes in one update and which information is used?
- Check the sign, step size and behavior on an ill-conditioned surface.
Answer
Gradient descent is an algorithm that uses calculus concept of gradient to try and reach local or global minima. It works by taking the negative of the gradient in a point of a given function, and updating that point repeatedly using the calculated negative gradient, until the algorithm reaches a local or global minimum, which will cause future iterations of the algorithm to return values that are equal or too close to the current point. It is widely used in machine learning applications.
Source and licence: Alexey Grigorev & contributors — Data Science Interviews · CC-BY-4.0