Why can stochastic objective evaluations ruin an otherwise sensible finite-difference gradient check?
MathematicsLevel: IntermediateDerivative and probability numerics
Why can stochastic objective evaluations ruin an otherwise sensible finite-difference gradient check?
Hints
- Noise is amplified by 1/h.
- Compare like-for-like evaluations.
Answer
Evaluation noise is divided by the small step and can dominate the signal. Freeze random draws or use controlled common randomness during checking.
Source and licence: SciPy — numerical differentiation · BSD-3-Clause · answer: Quantoros editorial