Model Evaluation III — Calibration: are the probabilities right, and how to fix them¶
Reliability diagrams, the Expected Calibration Error, Platt scaling & isotonic regression¶
A classifier can rank cases well yet output probabilities that are wrong. AUC only measures ranking; but a credit decision, an option price, or a Kelly bet size uses the number — if the model says 0.30, that outcome had better occur about 30% of the time. That property is calibration, and it is distinct from discrimination.
This notebook makes the distinction concrete and actionable:
- Reliability diagram — bin predictions by probability and plot observed frequency vs mean predicted probability; on the 45° line the model is calibrated.
- Expected Calibration Error (ECE) — the bin-size-weighted average gap between predicted and observed, $\text{ECE}=\sum_b \frac{n_b}{n}\,|\,\text{acc}_b-\text{conf}_b\,|$ — one number for miscalibration. It is a useful number and a slippery one: it depends on how the bins are drawn, and because it averages absolute gaps it is strictly positive even for a perfectly calibrated model. Both properties are measured below before any ECE is interpreted.
- The fixes — Platt scaling (fit a sigmoid to the scores) and isotonic regression (a nonparametric monotonic remap), each fit on held-out data.
We show that some models are badly miscalibrated despite good AUC (the classic Niculescu-Mizil finding), and that recalibration repairs the probabilities without touching the ranking. Data: Taiwan credit default. Python-lead.
1. Discrimination is not calibration¶
We fit four classifiers and score each three ways. AUC measures discrimination alone — whether cases are ranked correctly — and is blind to the values themselves: add 0.3 to every prediction and it does not move. ECE measures calibration alone — bin the predictions, compare each bin's average claim against the frequency observed in it, average the gaps — and is blind to ranking. The Brier score sits between them: the mean squared error of the predicted probabilities,
$$\text{Brier} = \frac{1}{n}\sum_{i=1}^{n}\left(p_i - y_i\right)^2,\qquad y_i \in \{0, 1\},$$
so it is penalised both for ordering cases wrongly and for stating the wrong level. That makes it a reasonable single summary and a poor diagnostic: a middling Brier score does not say which of the two went wrong, which is the entire point of this subsection. It is reported here beside the other two rather than instead of them.
The headline is immediate — Naive Bayes ranks about as well as logistic regression yet is wildly miscalibrated, because its feature-independence assumption pushes probabilities to the extremes. AUC alone would never reveal it.
Two things have to be settled before the smaller numbers in that table can be read at all.
First, an ECE of zero is not achievable even in principle. ECE averages absolute gaps, so sampling noise in each bin contributes a positive amount no matter how good the model is. The fix is to establish the noise floor by simulation: take a model's predicted probabilities, draw fresh outcomes from those exact probabilities so that calibration is perfect by construction, and see what ECE comes back. Any observed ECE at or below that floor is indistinguishable from perfectly calibrated.
Second, the SVM is not in a like-for-like comparison with the others. SVC(probability=True) fits an internal Platt calibrator by 5-fold cross-validation, so that row is already recalibrated while the other three are raw. It is marked as such below, because otherwise the table invites a conclusion about model families that it does not support.
import numpy as np, pandas as pd, matplotlib.pyplot as plt, warnings
warnings.filterwarnings("ignore")
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.svm import SVC
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import roc_auc_score, brier_score_loss
BLUE="#2b6cb0"; RED="#c53030"; GREEN="#2f855a"; ORANGE="#dd6b20"; GREY="#a0aec0"; PURP="#6b46c1"
d=pd.read_csv("credit_default.csv"); feat=[c for c in d.columns if c!="default"]
X=d[feat].values; y=d["default"].values
Xtr,Xte,ytr,yte=train_test_split(X,y,test_size=0.3,random_state=0,stratify=y)
sc=StandardScaler().fit(Xtr); Ztr=sc.transform(Xtr); Zte=sc.transform(Xte)
def ece(p,y,nb=10):
b=np.linspace(0,1,nb+1); e=0.0
for i in range(nb):
m=(p>=b[i])&(p<=b[i+1] if i==nb-1 else p<b[i+1])
if m.sum()>0: e+=m.mean()*abs(y[m].mean()-p[m].mean())
return e
sub=np.random.default_rng(0).choice(len(Ztr),6000,replace=False)
models={"logistic":(LogisticRegression(max_iter=2000).fit(Ztr,ytr),Zte),
"random forest":(RandomForestClassifier(300,min_samples_leaf=5,random_state=0,n_jobs=-1).fit(Xtr,ytr),Xte),
"SVM":(SVC(probability=True,random_state=0).fit(Ztr[sub],ytr[sub]),Zte),
"naive Bayes":(GaussianNB().fit(Ztr,ytr),Zte)}
P={nm:m.predict_proba(Xin)[:,1] for nm,(m,Xin) in models.items()}
tab=pd.DataFrame({nm:{"AUC":roc_auc_score(yte,P[nm]),"Brier":brier_score_loss(yte,P[nm]),"ECE":ece(P[nm],yte)} for nm in models}).T
# The noise floor: resample outcomes FROM each model's own probabilities, so calibration is exact by construction.
_r=np.random.default_rng(0)
floor={nm:np.quantile([ece(P[nm],_r.binomial(1,P[nm])) for _ in range(200)],0.95) for nm in models}
tab["ECE floor (95%)"]=[floor[nm] for nm in tab.index]
tab["above floor?"]=["yes" if tab.loc[nm,"ECE"]>floor[nm] else "no" for nm in tab.index]
print(tab.round(3).to_string())
print(f"\n 'ECE floor' is the 95th percentile of ECE for a PERFECTLY calibrated model with these same predicted")
print(f" probabilities and this sample size -- about {np.mean(list(floor.values())):.3f}. Nothing can score below it, so it is the")
print(" reference every ECE in this notebook is measured against. (*) SVM is pre-calibrated by SVC(probability=True).")
print(f"\nNaive Bayes: AUC {tab.loc['naive Bayes','AUC']:.3f}, comparable to logistic's {tab.loc['logistic','AUC']:.3f}, but ECE {tab.loc['naive Bayes','ECE']:.2f} -- roughly {tab.loc['naive Bayes','ECE']/floor['naive Bayes']:.0f} times the")
print("floor. A good ranker with badly wrong probabilities, which is the whole point of the section.")
print(f"\nThe quieter result is logistic regression at ECE {tab.loc['logistic','ECE']:.3f} -- second worst of the four, {tab.loc['logistic','ECE']/floor['logistic']:.0f} times its floor,")
print(f"and worse than the pre-calibrated SVM's {tab.loc['SVM','ECE']:.3f}. The random forest at {tab.loc['random forest','ECE']:.3f} sits just above its own floor of")
print(f"{floor['random forest']:.3f} -- close enough to perfect that the gap barely registers. Section 3 recalibrates all of them, and")
print("logistic regression turns out not to deserve the reputation it is usually given here.")
print("\nHow much of an ECE is a choice rather than a fact? The same predictions, binned different ways:")
def ece_q(p,y_,nb=10): # equal-FREQUENCY bins, the other convention
qs=np.quantile(p,np.linspace(0,1,nb+1)); qs[0]-=1e-9; qs[-1]+=1e-9; e=0.0
for i in range(nb):
m=(p>qs[i])&(p<=qs[i+1])
if m.sum()>0: e+=m.mean()*abs(y_[m].mean()-p[m].mean())
return e
NBINS=[5,10,15,20,50]
print(f" {'model':16s} " + " ".join(f"{'width '+str(n):>10}" for n in NBINS) + " " + " ".join(f"{'freq '+str(n):>9}" for n in NBINS))
for nm in models:
print(f" {nm:16s} " + " ".join(f"{ece(P[nm],yte,n):>10.3f}" for n in NBINS) + " " + " ".join(f"{ece_q(P[nm],yte,n):>9.3f}" for n in NBINS))
print(" Naive Bayes' catastrophe shows up under every scheme. But the random forest's ECE doubles between 5 and 50")
print(" equal-width bins, so a headline ECE of 0.015 is one defensible number among several. Report the binning, or")
print(" compare against the floor computed under the same binning -- an unqualified ECE is not a portable quantity.")
AUC Brier ECE ECE floor (95%) above floor? logistic 0.715 0.146 0.055 0.012 yes random forest 0.775 0.136 0.015 0.013 yes SVM 0.711 0.143 0.026 0.011 yes naive Bayes 0.719 0.327 0.378 0.013 yes 'ECE floor' is the 95th percentile of ECE for a PERFECTLY calibrated model with these same predicted probabilities and this sample size -- about 0.012. Nothing can score below it, so it is the reference every ECE in this notebook is measured against. (*) SVM is pre-calibrated by SVC(probability=True). Naive Bayes: AUC 0.719, comparable to logistic's 0.715, but ECE 0.38 -- roughly 29 times the floor. A good ranker with badly wrong probabilities, which is the whole point of the section. The quieter result is logistic regression at ECE 0.055 -- second worst of the four, 4 times its floor, and worse than the pre-calibrated SVM's 0.026. The random forest at 0.015 sits just above its own floor of 0.013 -- close enough to perfect that the gap barely registers. Section 3 recalibrates all of them, and logistic regression turns out not to deserve the reputation it is usually given here. How much of an ECE is a choice rather than a fact? The same predictions, binned different ways: model width 5 width 10 width 15 width 20 width 50 freq 5 freq 10 freq 15 freq 20 freq 50 logistic 0.042 0.055 0.059 0.061 0.064 0.059 0.059 0.059 0.059 0.062 random forest 0.012 0.015 0.013 0.018 0.026 0.010 0.015 0.014 0.018 0.024 SVM 0.022 0.026 0.029 0.026 0.037 0.025 0.031 0.038 0.038 0.042 naive Bayes 0.373 0.378 0.378 0.378 0.379 0.376 0.376 0.376 0.378 0.378 Naive Bayes' catastrophe shows up under every scheme. But the random forest's ECE doubles between 5 and 50 equal-width bins, so a headline ECE of 0.015 is one defensible number among several. Report the binning, or compare against the floor computed under the same binning -- an unqualified ECE is not a portable quantity.
2. The reliability diagrams¶
The reliability diagram shows how each model is miscalibrated. On the 45° line, a predicted probability equals the observed frequency. Naive Bayes hugs the axes — it says 0.99 for cases that default far less often and 0.01 for cases that default more — the signature of overconfidence. The random forest tracks the diagonal closely, and the internally-Platt-calibrated SVM is reasonable.
Logistic regression does not, and its curve is worth reading carefully because the shape explains what happens in the next section. It under-predicts at the bottom (predicting 0.06 where 0.13 default), over-predicts in the 0.2–0.3 range, and then under-predicts badly through the middle — predicting 0.55 where 0.72 default. That is not a sigmoidal distortion, and it is not one a sigmoid can undo.
def reliability(p,y,nb=10):
b=np.linspace(0,1,nb+1); xs=[];ys=[]
for i in range(nb):
m=(p>=b[i])&(p<=b[i+1] if i==nb-1 else p<b[i+1])
if m.sum()>20: xs.append(p[m].mean()); ys.append(y[m].mean())
return np.array(xs),np.array(ys)
fig,ax=plt.subplots(1,2,figsize=(13,4.6)); cols={"logistic":BLUE,"random forest":GREEN,"SVM":ORANGE,"naive Bayes":RED}
ax[0].plot([0,1],[0,1],"k--",lw=1,label="perfect calibration")
for nm in models: xs,ys=reliability(P[nm],yte); ax[0].plot(xs,ys,"o-",color=cols[nm],lw=2,ms=4,label=f"{nm} (ECE {tab.loc[nm,'ECE']:.3f})")
ax[0].set_xlabel("mean predicted probability"); ax[0].set_ylabel("observed default frequency"); ax[0].set_title("Reliability diagram"); ax[0].legend(fontsize=8)
nm_s=tab["ECE"].sort_values().index; ax[1].bar(nm_s,tab.loc[nm_s,"ECE"],color=[cols[n] for n in nm_s]); ax[1].set_ylabel("Expected Calibration Error"); ax[1].set_title("ECE by model (lower=better)")
plt.setp(ax[1].get_xticklabels(),rotation=20,ha="right",fontsize=8)
plt.tight_layout(); plt.show()
print("Naive Bayes' curve sticks to the axes -- overconfident at both ends, the classic signature. The random forest")
print("hugs the diagonal and the pre-calibrated SVM is close behind.")
print("\nLogistic regression is the interesting one, and the numbers behind its curve are worth reading directly:")
_p=P["logistic"]; _b=np.linspace(0,1,11)
print(f" {'bin':>12} {'n':>7} {'mean predicted':>15} {'observed':>10} {'gap':>8}")
for _i in range(10):
_m=(_p>=_b[_i])&(_p<=_b[_i+1] if _i==9 else _p<_b[_i+1])
if _m.sum()>20:
print(f" [{_b[_i]:.1f},{_b[_i+1]:.1f}) {_m.sum():>7} {_p[_m].mean():>15.3f} {yte[_m].mean():>10.3f} {yte[_m].mean()-_p[_m].mean():>+8.3f}")
print("It under-predicts at the bottom, over-predicts around 0.2-0.3, then under-predicts badly through the middle.")
print("The sign of the error changes twice. That shape matters for section 3: it is not a distortion a sigmoid can undo.")
Naive Bayes' curve sticks to the axes -- overconfident at both ends, the classic signature. The random forest
hugs the diagonal and the pre-calibrated SVM is close behind.
Logistic regression is the interesting one, and the numbers behind its curve are worth reading directly:
bin n mean predicted observed gap
[0.0,0.1) 1731 0.059 0.126 +0.066
[0.1,0.2) 3028 0.150 0.134 -0.016
[0.2,0.3) 2614 0.239 0.171 -0.068
[0.3,0.4) 489 0.345 0.364 +0.019
[0.4,0.5) 487 0.452 0.559 +0.106
[0.5,0.6) 366 0.547 0.719 +0.171
[0.6,0.7) 173 0.639 0.734 +0.095
[0.7,0.8) 57 0.743 0.772 +0.028
[0.8,0.9) 40 0.844 0.725 -0.119
It under-predicts at the bottom, over-predicts around 0.2-0.3, then under-predicts badly through the middle.
The sign of the error changes twice. That shape matters for section 3: it is not a distortion a sigmoid can undo.
3. Fixing it — Platt scaling and isotonic regression¶
Miscalibration is repairable by learning a monotonic map from the model's raw scores to calibrated probabilities, fit on held-out data (never the training set):
- Platt scaling — fit a logistic (sigmoid) to the scores; parametric, works with little calibration data, assumes a sigmoidal distortion.
- Isotonic regression — a free-form monotonic step function; more flexible, corrects any monotonic distortion, but needs more data and can overfit small sets.
Applied to Naive Bayes via CalibratedClassifierCV, both slash the ECE — isotonic taking it all the way to the noise floor — and, crucially, the AUC is unchanged: recalibration fixes the probabilities without altering the ranking.
Applying both to every model then settles the question section 1 raised, and produces the sharper result. Isotonic repairs all three. Platt scaling does not repair logistic regression at all — which is exactly what the theory predicts and is rarely shown: a sigmoid applied to the output of a model that is already a sigmoid of a linear index can only re-scale that index, so it cannot undo a distortion that is not sigmoidal. The reliability curve in section 2 shows the distortion is not.
from sklearn.calibration import CalibratedClassifierCV
fig,ax=plt.subplots(1,2,figsize=(13,4.6)); ax[0].plot([0,1],[0,1],"k--",lw=1,label="perfect")
xs,ys=reliability(P["naive Bayes"],yte); ax[0].plot(xs,ys,"o-",color=RED,lw=2,label=f"raw NB (ECE {tab.loc['naive Bayes','ECE']:.3f})")
rows=[("raw naive Bayes",roc_auc_score(yte,P['naive Bayes']),ece(P['naive Bayes'],yte))]
for meth,c in [("sigmoid",BLUE),("isotonic",GREEN)]:
cc=CalibratedClassifierCV(GaussianNB(),method=meth,cv=5).fit(Ztr,ytr); p=cc.predict_proba(Zte)[:,1]
xs,ys=reliability(p,yte); ax[0].plot(xs,ys,"o-",color=c,lw=2,label=f"NB + {meth} (ECE {ece(p,yte):.3f})")
rows.append((f"NB + {meth}",roc_auc_score(yte,p),ece(p,yte)))
ax[0].set_xlabel("mean predicted probability"); ax[0].set_ylabel("observed frequency"); ax[0].set_title("Naive Bayes: before vs after recalibration"); ax[0].legend(fontsize=8)
rt=pd.DataFrame(rows,columns=["model","AUC","ECE"]).set_index("model")
# Now do it for every model, not just the worst one.
specs={"logistic":(LogisticRegression(max_iter=2000),Ztr,Zte),
"random forest":(RandomForestClassifier(300,min_samples_leaf=5,random_state=0,n_jobs=-1),Xtr,Xte),
"naive Bayes":(GaussianNB(),Ztr,Zte)}
allrows=[]
for nm,(est,A,B) in specs.items():
r_={}
for meth in ["isotonic","sigmoid"]:
cc=CalibratedClassifierCV(est,method=meth,cv=5).fit(A,ytr); pp=cc.predict_proba(B)[:,1]
r_[meth]=(ece(pp,yte),roc_auc_score(yte,pp))
allrows.append((nm,ece(P[nm],yte),r_["isotonic"][0],r_["sigmoid"][0],roc_auc_score(yte,P[nm]),r_["isotonic"][1]))
at=pd.DataFrame(allrows,columns=["model","raw ECE","+isotonic","+sigmoid","raw AUC","isotonic AUC"]).set_index("model")
w=0.27; xp=np.arange(len(at))
ax[1].bar(xp-w,at["raw ECE"],w,color=RED,label="raw")
ax[1].bar(xp,at["+isotonic"],w,color=GREEN,label="+ isotonic")
ax[1].bar(xp+w,at["+sigmoid"],w,color=BLUE,label="+ sigmoid (Platt)")
ax[1].axhline(np.mean(list(floor.values())),color="k",ls=":",lw=1.4,label="ECE noise floor")
ax[1].set_yscale("log"); ax[1].set_xticks(xp); ax[1].set_xticklabels(at.index,rotation=15,ha="right",fontsize=8)
ax[1].set_ylabel("ECE (log scale)"); ax[1].set_title("Recalibration, all three models"); ax[1].legend(fontsize=7)
plt.tight_layout(); plt.show()
print(rt.round(3).to_string())
print(f"\nECE plummets while AUC is essentially unchanged -- calibration adjusts probabilities, not ranking.")
print("\nThe same treatment applied to every model:")
print(at.round(4).to_string())
print(f"\nTwo results worth separating. First, isotonic drives all three to roughly the noise floor "
f"({np.mean(list(floor.values())):.3f}): {at.loc['naive Bayes','+isotonic']:.4f}, {at.loc['logistic','+isotonic']:.4f}")
print(f"and {at.loc['random forest','+isotonic']:.4f}. Logistic regression's ECE falls by a factor of {at.loc['logistic','raw ECE']/at.loc['logistic','+isotonic']:.0f} -- it was NOT well calibrated,")
print("despite being the model most often described that way, and despite fitting the probabilities by maximum likelihood.")
print(f"\nSecond, Platt scaling leaves logistic regression exactly where it started: {at.loc['logistic','raw ECE']:.4f} to {at.loc['logistic','+sigmoid']:.4f}. That is not a")
print("failure of the implementation. A sigmoid applied to a logistic model's output can only rewrite a*z+b inside the")
print("same sigmoid -- it is a two-parameter rescaling of the linear index, so it can fix over- or under-confidence but")
print("nothing else. The distortion here is non-monotonic in shape (under, over, then under again), and only the")
print("free-form isotonic map can absorb it. Matching the corrector to the DISTORTION is the practical lesson: reach")
print("for Platt when the model is confidently the wrong scale, and for isotonic when the shape itself is wrong.")
AUC ECE
model
raw naive Bayes 0.719 0.378
NB + sigmoid 0.720 0.076
NB + isotonic 0.721 0.011
ECE plummets while AUC is essentially unchanged -- calibration adjusts probabilities, not ranking.
The same treatment applied to every model:
raw ECE +isotonic +sigmoid raw AUC isotonic AUC
model
logistic 0.0551 0.0071 0.0560 0.7145 0.7158
random forest 0.0150 0.0112 0.0182 0.7751 0.7760
naive Bayes 0.3781 0.0113 0.0755 0.7190 0.7211
Two results worth separating. First, isotonic drives all three to roughly the noise floor (0.012): 0.0113, 0.0071
and 0.0112. Logistic regression's ECE falls by a factor of 8 -- it was NOT well calibrated,
despite being the model most often described that way, and despite fitting the probabilities by maximum likelihood.
Second, Platt scaling leaves logistic regression exactly where it started: 0.0551 to 0.0560. That is not a
failure of the implementation. A sigmoid applied to a logistic model's output can only rewrite a*z+b inside the
same sigmoid -- it is a two-parameter rescaling of the linear index, so it can fix over- or under-confidence but
nothing else. The distortion here is non-monotonic in shape (under, over, then under again), and only the
free-form isotonic map can absorb it. Matching the corrector to the DISTORTION is the practical lesson: reach
for Platt when the model is confidently the wrong scale, and for isotonic when the shape itself is wrong.
4. Summary¶
Calibration is a separate axis of model quality from discrimination. Naive Bayes ranked defaults about as well as logistic regression (AUC 0.719 against 0.715) yet its probabilities were badly wrong (ECE 0.38, some 29× the noise floor) — useless for any decision that uses the number rather than the rank. Reliability diagrams show the distortion and ECE quantifies it; isotonic regression, fit on held-out data, repaired it without changing the AUC.
Two things had to be established before any of those numbers meant anything. ECE has a noise floor — it averages absolute gaps, so a perfectly calibrated model still scores about 0.009 on average at this sample size and 0.012 at the 95th percentile, and that is the reference every other figure is measured against. And ECE depends on the binning: the random forest's doubles between 5 and 50 equal-width bins, so an unqualified ECE is not a portable quantity.
The folklore about which models are calibrated did not survive contact with the data. Logistic regression was the second-worst of the four at ECE 0.055, over four times its floor, and isotonic recalibration improved it by a factor of eight — so it was not well calibrated, despite fitting probabilities by maximum likelihood and despite being the model usually cited as the safe one. The SVM only looked good because SVC(probability=True) had already Platt-calibrated it internally; that row was never a like-for-like comparison. The one part of the folklore that held up was the random forest, which sat just above its own noise floor.
The sharpest practical result is that Platt scaling could not repair logistic regression at all — 0.0551 to 0.0560, no improvement. A sigmoid on top of a sigmoid is only a two-parameter rescaling of the linear index, so it corrects over- and under-confidence and nothing else, while this distortion runs under, then over, then under again. Match the corrector to the shape of the distortion: Platt when the model is confidently on the wrong scale or calibration data is scarce, isotonic when the shape itself is wrong and there is enough data to estimate it.
Guidance: check calibration whenever the probability itself drives a decision — credit provisioning, option pricing, position sizing — not just the ranking, and check it against a simulated floor rather than against zero. Cross-links: this deepens the reliability curves shown in the tree-ensemble and regularized notebooks; Bayesian models (BART, Gaussian processes) and the MC-dropout capstone aim for calibration by construction via their posteriors. The final notebook, conformal prediction, takes the strongest stance of all — prediction sets with a guaranteed finite-sample coverage rate, distribution-free.