Skip to content
Quantoros

Among cases assigned probability 0.8, only half are positive. What calibration problem does this reveal?

Classic MLLevel: FoundationProbability meaning

Among cases assigned probability 0.8, only half are positive. What calibration problem does this reveal?

Hints
  • Compare predicted probability with empirical frequency.
  • Small bins are noisy.

Answer

The model is overconfident in that score region: estimated risk exceeds observed frequency. Evaluate enough held-out cases and uncertainty before inferring a systematic defect from a small bin.

Source and licence: scikit-learn 1.7.2 — calibration · BSD-3-Clause · answer: Quantoros editorial

Related questions