What's the difference between a loss function and a metric, and why can't you always optimize the metric directly?
Deep LearningLevel: IntermediateNeural networks
What's the difference between a loss function and a metric, and why can't you always optimize the metric directly?
Hints
- Trace data and gradients through the model, noting shapes.
- Separate training from inference and identify a necessary assumption.
Answer
A loss is the objective used to fit parameters; a metric measures performance for evaluation or decision-making. They may coincide, but often differ. Accuracy and F1 depend on discrete predictions and generally do not provide useful local derivatives of model parameters, so classification often trains with cross-entropy and evaluates with task metrics afterward.
A loss need not be differentiable everywhere. Hinge loss has a kink at its margin, and subgradient-based methods can optimize it; autodiff systems also define gradient conventions at certain nondifferentiable points. Other optimization methods do not require gradients at all.
A convenient surrogate is not guaranteed to optimize the business metric. Validate the actual metric and, where relevant, tune the prediction threshold separately.
Source and licence: Ishan Dutta — Awesome Data Science Interview Q&A · MIT · answer: Quantoros editorial