ORIGINAL TECHNICAL PRACTICE
Distinguish ranking from calibration
Two models have similar ranking metrics but different predicted probabilities. Why might that matter for a decision system?
Written for independent practice. The model answer presents one approach; clarify constraints and explain alternatives.
A REASONED APPROACH
Model answer.
A ranking metric assesses ordering, while calibration concerns whether predicted probabilities correspond to observed frequencies. A model can rank cases well yet systematically overstate or understate the chance of an outcome.
Work through it
- Inspect calibration on an independent sample using suitable plots and scoring rules. Check sufficiently supported subgroups rather than trusting tiny probability bins.
- If calibration is needed, fit a calibration method using data separate from the model’s training data, or use an appropriate cross-validation procedure.
- Choose decision thresholds using the calibrated outputs and operational costs, then validate under the class prevalence and conditions expected in deployment.
Tradeoffs to discuss
- A calibration method can improve probability estimates without improving ranking. Flexible calibrators need enough data and may overfit a small sample.
Common pitfalls
- Evaluating a calibrator on the same examples used to fit it is optimistic. A change in class prevalence can invalidate previously useful probability estimates.