Calibration — Are the Probabilities Right, and How to Fix Them

Discrimination Is Not Calibration

A classifier can rank cases well and still emit probabilities that are wrong. AUC (the area under the receiver-operating-characteristic curve: the probability a randomly chosen positive case is ranked above a randomly chosen negative one) measures only that ranking, but a credit provision, an option price or a Kelly bet size uses the number — if the model says 0.30, that outcome had better occur about 30% of the time. That property is calibration, and it is a separate axis of quality from discrimination.

The standard summary is the Expected Calibration Error: bin the predictions, compare each bin's mean predicted probability against its observed frequency, and average the gaps weighted by bin size. It is a useful number and a slippery one, in two specific ways that have to be settled before any ECE can be read.

What ECE reports when the model is perfect

First, zero is not achievable even in principle. ECE averages absolute gaps, so sampling noise in each bin contributes a positive amount however good the model is. The fix is to simulate the floor: take a model's predicted probabilities, draw fresh outcomes from those exact probabilities so that calibration is perfect by construction, and see what comes back. On this data it is about 0.009 on average and 0.012 at the 95th percentile. Every figure below is measured against that, not against zero.

Second, ECE depends on the binning. Naive Bayes' catastrophe survives every scheme, but the random forest's ECE doubles between 5 and 50 equal-width bins — 0.012 to 0.026. A headline ECE is one defensible number among several, so the binning has to be reported or the comparison made against a floor computed the same way.

Four Models, Three Scores

With that in place, the headline holds and a second result appears beneath it. Naive Bayes ranks defaults about as well as logistic regression — AUC 0.719 against 0.715 — while its ECE is 29× its noise floor, the classic Niculescu-Mizil result. AUC alone would never reveal it.

The three scores in the table answer different questions, which is why all three are shown. AUC measures discrimination only — whether cases are ordered correctly — and is blind to the values themselves: add 0.3 to every prediction and it does not move. ECE measures calibration only — bin the predictions, compare each bin's average claim against the frequency observed in it, average the gaps — and is blind to ordering. The Brier score sits between them: the mean squared error of the predicted probabilities, $\frac{1}{n}\sum_i (p_i - y_i)^2$, where $y_i$ is 0 or 1. Because it penalises both being wrong about the ranking and being wrong about the level, it moves for either failure — which makes it a reasonable single summary and a poor diagnostic, since a middling Brier score does not say which of the two is at fault. That is the case for reading it next to the other two rather than instead of them.

The quieter finding is that logistic regression is the second-worst of the four, at over four times its own floor and worse than the SVM. And the SVM's good showing is an artefact of the setup: SVC(probability=True) fits an internal Platt calibrator by cross-validation, so that row was already recalibrated while the other three were raw. It was never a like-for-like comparison of model families.

The reliability curve shows why logistic fails, and the shape turns out to matter. It under-predicts at the bottom (0.06 predicted where 0.13 default), over-predicts around 0.2–0.3, then under-predicts badly through the middle (0.55 predicted where 0.72 default). The sign of the error changes twice.

Fixing It — and One Fix That Cannot Work

Recalibration is a monotonic map from raw scores to probabilities, fit on held-out data. Platt scaling fits a sigmoid — two parameters, works with little data. Isotonic regression fits a free-form monotonic step function — more flexible, needs more data. Applied to all three raw models, isotonic drives every one of them to roughly the noise floor while leaving AUC essentially unchanged: calibration adjusts probabilities, not ranking.

The sharpest result is the failure. Platt scaling cannot repair logistic regression at all — 0.0551 to 0.0560, no improvement whatever. This is exactly what the theory predicts and is rarely shown: a sigmoid applied to a model that is already a sigmoid of a linear index $\sigma(a\cdot z + b)$ can only rewrite that index, so it is a two-parameter rescaling that corrects over- and under-confidence and nothing else. The distortion here changes sign twice, and only the free-form isotonic map can absorb it. Isotonic improves the same model by a factor of eight.

Where this sits

Reliability curves appear as diagnostics in XGBoost, LightGBM & CatBoost and Random Forests; this example is the machinery behind them. The habit of establishing what a metric reports when there is nothing to report runs through the whole subsection — the ECE floor here is the same move as the noise control in t-SNE & UMAP and the fresh-draw replication in Model Selection. Bayesian models aim at calibration by construction through their posteriors, as in BART and Gaussian Processes. The same credit data is modelled in Interpretability.

Notebook

Downloads

References

• Niculescu-Mizil, A. & Caruana, R. (2005). Predicting good probabilities with supervised learning. ICML, 625–632. — which model families are miscalibrated, and in which direction
• Platt, J. (1999). Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in Large Margin Classifiers. — Platt scaling
• Zadrozny, B. & Elkan, C. (2002). Transforming classifier scores into accurate multiclass probability estimates. KDD, 694–699. — isotonic recalibration
• Guo, C., Pleiss, G., Sun, Y. & Weinberger, K. Q. (2017). On calibration of modern neural networks. ICML, 1321–1330. — ECE and reliability diagrams in their current form
• Vaicenavicius, J. et al. (2019). Evaluating model calibration in classification. AISTATS, 3459–3467. — ECE's bias and its dependence on binning, the reason for the simulated floor
• Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review 78(1), 1–3. — the Brier score