Skip to content
Quantoros

What's the difference between a loss function and a metric, and why can't you always optimize the metric directly?

Deep LearningLevel: IntermediateNeural networks

What's the difference between a loss function and a metric, and why can't you always optimize the metric directly?

Hints
  • Trace data and gradients through the model, noting shapes.
  • Separate training from inference and identify a necessary assumption.

Answer

A loss is the objective used to fit parameters; a metric measures performance for evaluation or decision-making. They may coincide, but often differ. Accuracy and F1 depend on discrete predictions and generally do not provide useful local derivatives of model parameters, so classification often trains with cross-entropy and evaluates with task metrics afterward.

A loss need not be differentiable everywhere. Hinge loss has a kink at its margin, and subgradient-based methods can optimize it; autodiff systems also define gradient conventions at certain nondifferentiable points. Other optimization methods do not require gradients at all.

A convenient surrogate is not guaranteed to optimize the business metric. Validate the actual metric and, where relevant, tune the prediction threshold separately.

Source and licence: Ishan Dutta — Awesome Data Science Interview Q&A · MIT · answer: Quantoros editorial

Related questions