Among cases assigned probability 0.8, only half are positive. What calibration problem does this reveal?
Classic MLLevel: FoundationProbability meaning
Among cases assigned probability 0.8, only half are positive. What calibration problem does this reveal?
Hints
- Compare predicted probability with empirical frequency.
- Small bins are noisy.
Answer
The model is overconfident in that score region: estimated risk exceeds observed frequency. Evaluate enough held-out cases and uncertainty before inferring a systematic defect from a small bin.
Source and licence: scikit-learn 1.7.2 — calibration · BSD-3-Clause · answer: Quantoros editorial