Probability Calibration

Definition

A probabilistic classifier (or forecaster) is calibrated if its stated probabilities match observed frequencies: among all cases assigned probability $p$, a fraction $\approx p$ actually turn out positive. Calibration is a property distinct from discrimination (the ability to rank cases, measured by accuracy, AUC/lift): a model can rank perfectly yet be badly miscalibrated (scores too extreme or too timid), and vice versa. Calibration matters wherever the magnitude of a probability drives a decision — pricing default risk, expected-loss reserving, medical risk communication, cost-sensitive classification.

Key Ideas

• Calibration vs. discrimination. Discrimination asks "are positives scored higher than negatives?"; calibration asks "does score $p$ mean an event rate of $p$?". Error rate, AUC, and lift measure the former and are invariant to any monotone rescaling of the scores — so they say nothing about calibration (Yeh-Lien 2009).
• Reliability diagram. The standard visual: bin cases by predicted probability, plot observed frequency against mean predicted probability; the 45° line is perfect calibration, deviations show over/under-confidence.
• Calibration slope/intercept. Regressing the (estimated) true probability $Y$ on the predicted probability $X$, $Y=A+BX$, summarizes calibration: $A\approx0,\ B\approx1$ is well-calibrated; $B<1$ signals overconfident (too-extreme) scores. The Sorting-Smoothing Method (Yeh-Lien 2009) estimates the unobserved $Y$ by sorting cases on score and locally averaging the binary outcomes.
• Recalibration transforms (Zadrozny-Elkan 2002). A miscalibrated but well-ranking model can be post-hoc recalibrated by a monotone map from scores to probabilities. Three standard choices span a bias–flexibility spectrum: Platt scaling (fit a parametric sigmoid — biased if the true map isn't sigmoidal), histogram binning (average labels in fixed score bins — sensitive to bin count/boundaries), and isotonic regression (the non-parametric middle ground: the best non-decreasing stepwise function under MSE, computed by the pair-adjacent-violators (PAV) algorithm — pool adjacent monotonicity-violating bins into their weighted average). Isotonic (via PAV) is the ML default; it can overfit small calibration sets.
• Multiclass calibration (Zadrozny-Elkan 2002): the standard methods are binary-only, so decompose the $k$-class problem into binary sub-problems (one-vs-all / all-pairs), calibrate each, then couple the calibrated binary probabilities (normalize / solve for consistent class probabilities) into multiclass estimates — the first general multiclass calibration recipe. Fit the recalibration map on a held-out calibration split, distinct from training.
• Scoring rules connect calibration and sharpness. A proper scoring rule (log score, Brier score) is minimized in expectation by the true probabilities, and decomposes into calibration (reliability) + refinement/sharpness terms — so proper scores reward being calibrated and confident.

How It Works

Assess calibration on held-out data: build a reliability diagram or fit the calibration slope/intercept (or the Spiegelhalter/Cox tests); summarize overall accuracy with the Brier score and its reliability decomposition. If miscalibrated, fit a recalibration map (Platt/isotonic) on a separate calibration split and apply it to future scores. Beware small bins (noisy reliability estimates) and distribution shift (a recalibration learned on one population need not transfer).

Why It Matters

• Decisions use probabilities, not ranks. Expected-loss pricing, reserving, and cost-sensitive actions all plug the probability into a formula; a miscalibrated $p$ biases every downstream number even when classification "accuracy" looks fine.
• Model comparison. Two classifiers with equal AUC can differ sharply in calibration; choosing on accuracy alone can pick the worse decision tool (Yeh-Lien 2009 find the neural net uniquely well-calibrated on the credit-default data).
• Trust and communication. Calibrated risk scores (clinical, credit, weather) can be taken at face value; uncalibrated ones mislead.

Open Questions

• Multiclass and structured-output calibration is harder than binary (per-class vs. joint calibration).
• Calibration under distribution shift — recalibration maps are population-specific.
• Reconciling calibration with sharpness: the goal is the most confident forecasts that remain calibrated, not calibration alone.

Related

• Conformal Prediction
• Proper Scoring Rule
• Interval Forecast Evaluation
• Diagnostic Accuracy
• Binary Probit
• Yeh-Lien (2009) Comparisons of Data Mining Techniques for Predicting Probability of Default of Credit Card Clients
• Zadrozny-Elkan (2002) Transforming Classifier Scores into Accurate Multiclass Probability Estimates
• I-Cheng Yeh
• Che-hui Lien
• Bianca Zadrozny
• Charles Elkan
• Monte Carlo Dropout
• Probability Integral Transform