An analytic gradient disagrees with finite differences. What should be checked before blaming the optimizer?
MathematicsLevel: IntermediateMultivariate optimizer reasoning
An analytic gradient disagrees with finite differences. What should be checked before blaming the optimizer?
Hints
- Both methods must differentiate the same objective.
- Vary step size.
Answer
Check sign, shape, parameter ordering, objective normalization and deterministic evaluation. Then vary finite-difference step to distinguish formula mistakes from numerical noise.
Source and licence: SciPy — scalar and multivariate minimization · BSD-3-Clause · answer: Quantoros editorial