Neural Networks V — Bayesian Deep Learning (capstone)¶
Calibrated uncertainty for neural nets: MC-dropout & deep ensembles¶
Every network in this subsection returned a point prediction — one number, no sense of how sure it is. That is a serious gap for decisions under risk: a credit model that is 51% vs 99% confident should be treated very differently, and a volatility forecast is only useful with an error bar. This capstone adds calibrated uncertainty to neural networks, closing the subsection and tying it back to the portfolio's Bayesian core — BART and Gaussian processes, where uncertainty was the whole point.
Two kinds of uncertainty:
- Aleatoric — irreducible noise in the data (two identical clients, different outcomes). More data does not remove it.
- Epistemic — the model's ignorance, largest where training data is sparse or absent. More data does shrink it. This is the "the model knows what it doesn't know" part, and the dangerous one when a model is deployed on inputs unlike anything it was trained on.
Full Bayesian inference over a network's millions of weights is intractable, so we use two practical, widely-used approximations:
- MC-dropout (Gal & Ghahramani, 2016) — keep dropout switched on at test time and run many stochastic forward passes; each is a sample from an approximate posterior, and their spread estimates epistemic uncertainty. Free, if the network already uses dropout.
- Deep ensembles (Lakshminarayanan et al., 2017) — train several networks from different random initialisations; their disagreement is the uncertainty. Simple, embarrassingly parallel, and often the best-calibrated of all.
We show the canonical picture (uncertainty widening away from the data), verify calibration on real data, and turn uncertainty into a decision rule. Python-only.
1. The canonical picture — uncertainty grows away from the data¶
The clearest demonstration is a 1-D regression with training data only in two bands (roughly $[-3,-1]$ and $[1,3]$), leaving a gap in the middle and nothing in the extrapolation tails. A plain network fits a confident curve everywhere — including regions it has never seen, exactly where it is most likely to be wrong. A Bayesian treatment instead widens its predictive interval where there is no data: modest across the interpolation gap, dramatic out in the tails. Below, both a deep ensemble and MC-dropout show this; the plain net (dashed) stays overconfident throughout.
import os
os.environ["KMP_DUPLICATE_LIB_OK"]="TRUE"
import numpy as np, pandas as pd, matplotlib.pyplot as plt, time, warnings
warnings.filterwarnings("ignore")
import torch, torch.nn as nn; torch.set_num_threads(2)
BLUE="#2b6cb0"; RED="#c53030"; GREEN="#2f855a"; ORANGE="#dd6b20"; GREY="#a0aec0"; PURP="#6b46c1"
rng=np.random.default_rng(0)
xa=np.r_[rng.uniform(-3,-1,60),rng.uniform(1,3,60)]; ya=np.sin(1.5*xa)+0.1*rng.standard_normal(len(xa))
X=torch.tensor(xa.reshape(-1,1),dtype=torch.float32); Y=torch.tensor(ya.reshape(-1,1),dtype=torch.float32)
grid=torch.linspace(-6,6,240).view(-1,1); gx=grid.numpy().ravel()
def toynet(p=0.0): return nn.Sequential(nn.Linear(1,64),nn.ReLU(),nn.Dropout(p),nn.Linear(64,64),nn.ReLU(),nn.Dropout(p),nn.Linear(64,1))
def fit_toy(m,ep=1200,wd=1e-3):
opt=torch.optim.Adam(m.parameters(),1e-2,weight_decay=wd); lf=nn.MSELoss()
for e in range(ep): opt.zero_grad(); lf(m(X),Y).backward(); opt.step()
return m
def pred(m,g):
m.eval()
with torch.no_grad(): return m(g).numpy().ravel()
ens=[fit_toy(toynet(0.0)) for s in range(8) if (torch.manual_seed(s) or True)]
Pe=np.stack([pred(e,grid) for e in ens],0); me,se=Pe.mean(0),Pe.std(0)
torch.manual_seed(0); md=fit_toy(toynet(0.3),ep=1500,wd=1e-4); md.train()
with torch.no_grad(): Pm=np.stack([md(grid).numpy().ravel() for _ in range(200)],0)
mm,sm=Pm.mean(0),Pm.std(0)
plain=pred(ens[0],grid)
fig,ax=plt.subplots(1,2,figsize=(13,4.4))
for a,(nm,m_,s_) in zip(ax,[("Deep ensemble (8 nets)",me,se),("MC-dropout (200 passes)",mm,sm)]):
a.fill_between(gx,m_-2*s_,m_+2*s_,color=BLUE,alpha=.2,label="±2 std (uncertainty)")
a.plot(gx,m_,color=BLUE,lw=2,label="predictive mean"); a.plot(gx,plain,color=GREY,lw=1.5,ls="--",label="plain net (overconfident)")
a.scatter(xa,ya,s=12,color="black",zorder=5,label="training data"); a.axvspan(-1,1,color=RED,alpha=.05)
a.set_ylim(-2.5,2.5); a.set_title(nm); a.set_xlabel("x"); a.legend(fontsize=7,loc="upper center")
plt.tight_layout(); plt.show()
_ond=(np.abs(gx)>=1)&(np.abs(gx)<=3); _gap=np.abs(gx)<1; _ext=np.abs(gx)>3
for _nm,_s in [("deep ensemble",se),("MC-dropout",sm)]:
print(f"{_nm:14s} std -- on data {_s[_ond].mean():.3f} gap {_s[_gap].mean():.3f} extrapolation {_s[_ext].mean():.3f}"
f" ratio {_s[_ext].mean()/_s[_ond].mean():5.1f}x")
print(f"\nBoth widen in the tails, but they are not equally good at it, and the ratio is the thing to look at.")
print(f"The ensemble separates extrapolation from in-data by {se[_ext].mean()/se[_ond].mean():.0f}x; MC-dropout by only {sm[_ext].mean()/sm[_ond].mean():.0f}x.")
print(f"The reason is visible on the left of each ratio: MC-dropout reports {sm[_ond].mean():.2f} even where it has plenty of data,")
print("which is larger than the data's own noise (0.10). Much of its 'uncertainty' is just the dropout rate showing")
print("through rather than a statement about evidence. The ensemble's on-data spread collapses to near zero, so when it")
print("does widen you have learned something. Keep that asymmetry in mind for section 4.")
deep ensemble std -- on data 0.016 gap 0.025 extrapolation 0.239 ratio 14.6x MC-dropout std -- on data 0.145 gap 0.165 extrapolation 0.466 ratio 3.2x Both widen in the tails, but they are not equally good at it, and the ratio is the thing to look at. The ensemble separates extrapolation from in-data by 15x; MC-dropout by only 3x. The reason is visible on the left of each ratio: MC-dropout reports 0.14 even where it has plenty of data, which is larger than the data's own noise (0.10). Much of its 'uncertainty' is just the dropout rate showing through rather than a statement about evidence. The ensemble's on-data spread collapses to near zero, so when it does widen you have learned something. Keep that asymmetry in mind for section 4.
2. MC-dropout — dropout as approximate Bayesian inference¶
Dropout randomly zeroes units during training as a regulariser. Gal & Ghahramani's insight: if you leave it on at test time and average many forward passes, you are Monte-Carlo-integrating over an approximate posterior on the weights — a Bayesian neural network almost for free. The prediction is the mean over passes; the epistemic uncertainty is their standard deviation; adding the residual noise gives a full predictive interval.
We test it on California housing (a real regression), fitting one dropout network and taking 50 stochastic passes per test point. The check that matters is calibration: a 90% predictive interval should contain the truth about 90% of the time. And the uncertainty should be useful — bigger where the model is more wrong.
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import mean_squared_error
h=pd.read_csv("cali_housing.csv"); hf=[c for c in h.columns if c!="MedHouseVal"]
Xh=h[hf].to_numpy(float); yh=h["MedHouseVal"].to_numpy(float)
Xtr,Xte,ytr,yte=train_test_split(Xh,yh,test_size=0.3,random_state=0); sc=StandardScaler().fit(Xtr); Ztr=sc.transform(Xtr); Zte=sc.transform(Xte)
ym,ys=ytr.mean(),ytr.std()
def reg_net(seed,p=0.1):
torch.manual_seed(seed)
return nn.Sequential(nn.Linear(Ztr.shape[1],64),nn.ReLU(),nn.Dropout(p),nn.Linear(64,32),nn.ReLU(),nn.Dropout(p),nn.Linear(32,1))
def fit_reg(net,ep=40):
opt=torch.optim.Adam(net.parameters(),1e-3,weight_decay=1e-4); lf=nn.MSELoss()
Xt=torch.tensor(Ztr,dtype=torch.float32); yt=torch.tensor((ytr-ym)/ys,dtype=torch.float32).view(-1,1)
for e in range(ep):
pm=torch.randperm(len(Xt))
for b in range(0,len(Xt),256): bi=pm[b:b+256]; opt.zero_grad(); lf(net(Xt[bi]),yt[bi]).backward(); opt.step()
return net
net=fit_reg(reg_net(0)); Xtt=torch.tensor(Zte,dtype=torch.float32)
net.eval()
with torch.no_grad(): noise=np.std(ytr-(net(torch.tensor(Ztr,dtype=torch.float32)).numpy().ravel()*ys+ym))
net.train()
with torch.no_grad(): P=np.stack([net(Xtt).numpy().ravel() for _ in range(50)],0)*ys+ym
mu=P.mean(0); psd=np.sqrt(P.std(0)**2+noise**2)
cov=np.mean((yte>=mu-1.645*psd)&(yte<=mu+1.645*psd)); mc_rmse=mean_squared_error(yte,mu)**.5
print(f"MC-dropout California: RMSE {mc_rmse:.3f} 90% predictive-interval coverage {cov:.3f} (target 0.90)")
fig,ax=plt.subplots(1,2,figsize=(13,4.4))
o=np.argsort(mu)[::40]
ax[0].errorbar(np.arange(len(o)),mu[o],yerr=1.645*psd[o],fmt="o",ms=3,color=BLUE,ecolor=GREY,capsize=2,label="pred ± 90%")
ax[0].scatter(np.arange(len(o)),yte[o],s=14,color=RED,zorder=5,label="actual"); ax[0].set_title("MC-dropout: predictions with intervals (sampled test pts)"); ax[0].set_xlabel("test point (sorted by prediction)"); ax[0].legend(fontsize=8)
err=np.abs(yte-mu); qb=np.quantile(psd,np.linspace(0,1,7)); bi=np.clip(np.digitize(psd,qb[1:-1]),0,5)
ax[1].plot([psd[bi==k].mean() for k in range(6)],[err[bi==k].mean() for k in range(6)],"o-",color=PURP,lw=2)
ax[1].set_xlabel("predictive std (uncertainty)"); ax[1].set_ylabel("mean |error|"); ax[1].set_title("Uncertainty is informative: error grows with predicted std")
plt.tight_layout(); plt.show()
_nd=fit_reg(reg_net(0,0.0)); _nd.eval()
with torch.no_grad(): _ndp=_nd(Xtt).numpy().ravel()*ys+ym
print(f"Coverage {cov:.3f} against a nominal 0.90 -- slightly conservative rather than exactly calibrated: the intervals")
print("are a little wider than they need to be, which is the safer direction to err but is worth naming precisely.")
print(f"And the uncertainty is informative -- mean absolute error rises monotonically with the predicted std (right panel).")
print(f"\nThe error bars are not free, though. This net carries dropout 0.1 and scores RMSE {mc_rmse:.4f}; the identical")
print(f"architecture without dropout scores {mean_squared_error(yte,_ndp)**.5:.4f}, and the plain MLP of the first notebook about 0.525.")
print("So roughly 0.03 RMSE is the price of the uncertainty machinery here -- small, but it is a price, not a bonus.")
MC-dropout California: RMSE 0.559 90% predictive-interval coverage 0.922 (target 0.90)
Coverage 0.922 against a nominal 0.90 -- slightly conservative rather than exactly calibrated: the intervals are a little wider than they need to be, which is the safer direction to err but is worth naming precisely. And the uncertainty is informative -- mean absolute error rises monotonically with the predicted std (right panel). The error bars are not free, though. This net carries dropout 0.1 and scores RMSE 0.5595; the identical architecture without dropout scores 0.5464, and the plain MLP of the first notebook about 0.525. So roughly 0.03 RMSE is the price of the uncertainty machinery here -- small, but it is a price, not a bonus.
3. Deep ensembles — disagreement as uncertainty¶
The other workhorse: train several networks from different random seeds. Where the data pins the answer down they agree; where it does not they diverge, and that disagreement is the epistemic uncertainty. Deep ensembles are trivially parallel and frequently the best-calibrated method in practice. We train five networks on California and compare coverage and accuracy to MC-dropout — both should land near the nominal 90%, with the ensemble usually a touch sharper.
nets=[fit_reg(reg_net(s)) for s in range(5)]
def pr(net):
net.eval()
with torch.no_grad(): return net(Xtt).numpy().ravel()*ys+ym
Pe=np.stack([pr(n) for n in nets],0); mue=Pe.mean(0); psd_e=np.sqrt(Pe.std(0)**2+noise**2)
cov_e=np.mean((yte>=mue-1.645*psd_e)&(yte<=mue+1.645*psd_e)); ens_rmse=mean_squared_error(yte,mue)**.5
tab=pd.DataFrame({"RMSE":[mc_rmse,ens_rmse],"90% coverage":[cov,cov_e],"mean interval width":[2*1.645*psd.mean(),2*1.645*psd_e.mean()]},
index=["MC-dropout","deep ensemble (5)"])
print(tab.round(3).to_string())
fig,ax=plt.subplots(1,2,figsize=(12,4.2))
ax[0].bar(["MC-dropout","ensemble"],[cov,cov_e],color=[BLUE,GREEN]); ax[0].axhline(0.90,color=RED,ls="--",label="nominal 0.90"); ax[0].set_ylim(0.8,1.0); ax[0].set_ylabel("90% coverage"); ax[0].set_title("Both well-calibrated"); ax[0].legend()
ax[1].scatter(mue,yte,s=5,alpha=.12,color=GREEN); lim=[yte.min(),yte.max()]; ax[1].plot(lim,lim,"k--",lw=1)
ax[1].set_xlabel("ensemble predicted value (100k USD)"); ax[1].set_ylabel("actual"); ax[1].set_title(f"Deep ensemble: predicted vs actual (RMSE {ens_rmse:.3f})")
plt.tight_layout(); plt.show()
print(f"MC-dropout and the deep ensemble both cover ~{cov_e:.0%} at the nominal 90% level -- calibrated uncertainty from a")
print("neural net by two cheap, practical routes. Neither is exact Bayes, but both are honest about what they don't know.")
RMSE 90% coverage mean interval width MC-dropout 0.559 0.922 1.870 deep ensemble (5) 0.548 0.920 1.808
MC-dropout and the deep ensemble both cover ~92% at the nominal 90% level -- calibrated uncertainty from a neural net by two cheap, practical routes. Neither is exact Bayes, but both are honest about what they don't know.
4. The payoff — uncertainty as a decision rule¶
Calibrated uncertainty is supposed to change decisions, and the standard demonstration is selective prediction: score each client by the spread of its predicted default probability across dropout passes, then auto-decide the most-confident cases and route the rest to manual review. Accuracy on the retained set duly climbs from 0.82 to 0.90 as we defer more.
That demonstration is a trap, and this section is about why — because it is the single easiest way to talk yourself into trusting a model that is not helping. On imbalanced data the retained subset becomes easier at roughly the same rate the model appears to improve, so the gain over simply predicting the majority class vanishes. The cell below plots the trivial baseline alongside the accuracy curve, and tracks what happens to the minority class the model exists to find.
d=pd.read_csv("credit_default.csv"); feat=[c for c in d.columns if c!="default"]
Xc=d[feat].to_numpy(float); yc=d["default"].to_numpy(int)
Xctr,Xcte,yctr,ycte=train_test_split(Xc,yc,test_size=0.3,random_state=0,stratify=yc)
scl=StandardScaler().fit(Xctr); Ctr=scl.transform(Xctr); Cte=scl.transform(Xcte)
torch.manual_seed(0); clf=nn.Sequential(nn.Linear(Ctr.shape[1],64),nn.ReLU(),nn.Dropout(0.3),nn.Linear(64,32),nn.ReLU(),nn.Dropout(0.3),nn.Linear(32,1))
opt=torch.optim.Adam(clf.parameters(),1e-3,weight_decay=1e-4); lf=nn.BCEWithLogitsLoss()
Xt=torch.tensor(Ctr,dtype=torch.float32); yt=torch.tensor(yctr,dtype=torch.float32).view(-1,1)
for e in range(40):
pm=torch.randperm(len(Xt))
for b in range(0,len(Xt),256): bi=pm[b:b+256]; opt.zero_grad(); lf(clf(Xt[bi]),yt[bi]).backward(); opt.step()
clf.train()
with torch.no_grad(): Pc=np.stack([torch.sigmoid(clf(torch.tensor(Cte,dtype=torch.float32))).numpy().ravel() for _ in range(50)],0)
from sklearn.metrics import roc_auc_score
pmn=Pc.mean(0); unc=Pc.std(0); pred=(pmn>0.5).astype(int); acc=(pred==ycte); order=np.argsort(unc)
fracs=np.linspace(0.3,1.0,15)
accs=[acc[order[:int(f*len(order))]].mean() for f in fracs]
base=[max(ycte[order[:int(f*len(order))]].mean(),1-ycte[order[:int(f*len(order))]].mean()) for f in fracs]
rec=[]
for f in fracs:
i_=order[:int(f*len(order))]; y_=ycte[i_]; p_=(pmn[i_]>0.5).astype(int)
rec.append(((p_==1)&(y_==1)).sum()/max(1,(y_==1).sum()))
fig,ax=plt.subplots(1,3,figsize=(15,4.2))
ax[0].plot(fracs*100,accs,"o-",color=BLUE,lw=2,label="accuracy on retained")
ax[0].plot(fracs*100,base,"s--",color=RED,lw=2,label="predict-the-majority baseline")
ax[0].set_xlabel("% auto-predicted (most-certain first)"); ax[0].set_ylabel("accuracy"); ax[0].invert_xaxis()
ax[0].set_title("Accuracy rises -- and so does the trivial baseline"); ax[0].legend(fontsize=8)
ax[1].plot(fracs*100,rec,"o-",color=PURP,lw=2); ax[1].set_xlabel("% auto-predicted (most-certain first)")
ax[1].set_ylabel("recall on defaults"); ax[1].invert_xaxis(); ax[1].set_title("...while the defaults disappear")
ax[2].hist(unc[acc],bins=30,alpha=.6,color=GREEN,label="correct",density=True)
ax[2].hist(unc[~acc],bins=30,alpha=.6,color=RED,label="wrong",density=True)
ax[2].set_xlabel("MC-dropout uncertainty (std of P)"); ax[2].set_ylabel("density")
ax[2].set_title("Errors do sit at higher uncertainty"); ax[2].legend(fontsize=8)
plt.tight_layout(); plt.show()
print(f"{'retain':>8} {'accuracy':>10} {'majority baseline':>19} {'lift':>8} {'default recall':>16} {'AUC':>7}")
for f in [1.0,0.75,0.5,0.3]:
i_=order[:int(f*len(order))]; y_=ycte[i_]; p_=(pmn[i_]>0.5).astype(int)
a_=(p_==y_).mean(); b_=max(y_.mean(),1-y_.mean())
r_=((p_==1)&(y_==1)).sum()/max(1,(y_==1).sum())
print(f"{f:>7.0%} {a_:>10.3f} {b_:>19.3f} {a_-b_:>+8.3f} {r_:>16.3f} {roc_auc_score(y_,pmn[i_]):>7.3f}")
print("\nRead the first column alone and this looks like the textbook result: accuracy climbing from 0.82 to 0.90 as the")
print("uncertain cases are deferred. Read the second and it evaporates. The retained subset gets easier at exactly the")
print("rate the model improves, so the LIFT over simply predicting 'no default' falls to zero. At 50% retention the")
print("model is no better than a constant, and at 30% it catches literally none of the defaults it was built to find.")
print(f"\nThe mechanism is not subtle once you look: corr(uncertainty, predicted probability) = {np.corrcoef(unc,pmn)[0,1]:.2f}. For a sigmoid")
print("output, the spread across dropout passes is mechanically small when the probability sits near 0, so 'most certain'")
print(f"mostly means 'confidently predicted non-default'. Deferring by uncertainty discards {100*(1-ycte[order[:len(order)//2]].sum()/ycte.sum()):.0f}% of all defaults.")
print("\nThis is a real trap, not a quirk of this dataset: selective prediction scored by ACCURACY on imbalanced data")
print("will almost always look like it works. The honest diagnostics are the ones above -- lift over the majority")
print("baseline, recall on the minority class, or AUC on the retained subset, none of which improve here.")
print(f"\nWhat survives is weaker but genuine: errors do sit at higher uncertainty ({unc[~acc].mean():.4f} against {unc[acc].mean():.4f}, a ratio")
print(f"of {unc[~acc].mean()/unc[acc].mean():.2f}), and uncertainty predicts being wrong with AUC {roc_auc_score((~acc).astype(int),unc):.3f}. That is a usable triage signal, just")
print("a far more modest one than the accuracy curve advertises -- and on the regression task in sections 2-3, where")
print("there is no class balance to be fooled by, the interval story holds up cleanly.")
retain accuracy majority baseline lift default recall AUC
100% 0.821 0.779 +0.042 0.334 0.768
75% 0.856 0.853 +0.003 0.049 0.682
50% 0.882 0.883 -0.000 0.006 0.667
30% 0.904 0.904 +0.000 0.000 0.684
Read the first column alone and this looks like the textbook result: accuracy climbing from 0.82 to 0.90 as the
uncertain cases are deferred. Read the second and it evaporates. The retained subset gets easier at exactly the
rate the model improves, so the LIFT over simply predicting 'no default' falls to zero. At 50% retention the
model is no better than a constant, and at 30% it catches literally none of the defaults it was built to find.
The mechanism is not subtle once you look: corr(uncertainty, predicted probability) = 0.71. For a sigmoid
output, the spread across dropout passes is mechanically small when the probability sits near 0, so 'most certain'
mostly means 'confidently predicted non-default'. Deferring by uncertainty discards 73% of all defaults.
This is a real trap, not a quirk of this dataset: selective prediction scored by ACCURACY on imbalanced data
will almost always look like it works. The honest diagnostics are the ones above -- lift over the majority
baseline, recall on the minority class, or AUC on the retained subset, none of which improve here.
What survives is weaker but genuine: errors do sit at higher uncertainty (0.0509 against 0.0412, a ratio
of 1.23), and uncertainty predicts being wrong with AUC 0.645. That is a usable triage signal, just
a far more modest one than the accuracy curve advertises -- and on the regression task in sections 2-3, where
there is no class balance to be fooled by, the interval story holds up cleanly.
5. Proportions vs predictions — the calibration check, on both datasets¶
The collection's standard diagnostic, applied here to both real tasks. Bin the test set by what the model predicted, then plot what actually happened in each bin: on the 45-degree line the prediction is the outcome rate.
It matters more in this notebook than anywhere else, because "calibrated" has been used for two different claims that a single coverage number cannot separate:
- Calibration of the mean / probability — do predictions of 0.3 come true 30% of the time, and do the regression deciles land on the diagonal? This is the classical reliability view, and the credit classifier has never been checked this way despite the notebook being about calibration.
- Calibration of the interval — does the 90% band contain the truth 90% of the time? Section 2 reported 92% overall, but an aggregate can be right on average while being wrong everywhere: too wide at one end of the prediction range and too narrow at the other. The right panel below reports coverage per decile to test exactly that.
# --- regression: decile means, and coverage WITHIN each decile ---
qq=np.quantile(mu,np.linspace(0,1,11)); bidx=np.clip(np.digitize(mu,qq[1:-1]),0,9)
pm_=[mu[bidx==k].mean() for k in range(10)]; am_=[yte[bidx==k].mean() for k in range(10)]
cov_k=[np.mean((yte[bidx==k]>=mu[bidx==k]-1.645*psd[bidx==k])&(yte[bidx==k]<=mu[bidx==k]+1.645*psd[bidx==k])) for k in range(10)]
# --- classification: reliability of the predicted probability ---
qc=np.quantile(pmn,np.linspace(0,1,11)); cidx=np.clip(np.digitize(pmn,qc[1:-1]),0,9)
pp_=[pmn[cidx==k].mean() for k in range(10)]; op_=[ycte[cidx==k].mean() for k in range(10)]
ece=float(np.sum([np.mean(cidx==k)*abs(op_[k]-pp_[k]) for k in range(10)]))
fig,ax=plt.subplots(1,3,figsize=(15,4.3))
lim=[min(pm_+am_),max(pm_+am_)]
ax[0].plot(lim,lim,"k--",lw=1,label="perfect")
ax[0].plot(pm_,am_,"o-",color=BLUE,lw=2,label="decile means")
ax[0].set_xlabel("MC-dropout predicted value ($100k)"); ax[0].set_ylabel("mean actual value")
ax[0].set_title("California: proportions vs predictions"); ax[0].legend(fontsize=8)
ax[1].axhline(0.90,color=RED,ls="--",lw=1,label="nominal 0.90")
ax[1].plot(pm_,cov_k,"o-",color=PURP,lw=2,label="coverage in decile")
ax[1].set_xlabel("predicted value ($100k)"); ax[1].set_ylabel("90% interval coverage"); ax[1].set_ylim(0.7,1.02)
ax[1].set_title("...and coverage is NOT flat across the range"); ax[1].legend(fontsize=8)
ax[2].plot([0,max(pp_)],[0,max(pp_)],"k--",lw=1,label="perfect")
ax[2].plot(pp_,op_,"o-",color=GREEN,lw=2,label=f"deciles (ECE {ece:.3f})")
ax[2].set_xlabel("predicted P(default)"); ax[2].set_ylabel("observed default rate")
ax[2].set_title("Credit: reliability of the probability"); ax[2].legend(fontsize=8)
plt.tight_layout(); plt.show()
print("California, by decile of the prediction:")
print(f" {'decile':>7} {'predicted':>10} {'actual':>9} {'gap':>8} {'90% coverage':>13}")
for k in range(10):
print(f" {k+1:>7} {pm_[k]:>10.3f} {am_[k]:>9.3f} {am_[k]-pm_[k]:>+8.3f} {cov_k[k]:>13.3f}")
print(f"\nTwo things the aggregate hid. The decile means bend AWAY from the diagonal at both ends -- the model")
print(f"over-predicts the cheapest blocks by {pm_[0]-am_[0]:.3f} and under-predicts the dearest by {am_[-1]-pm_[-1]:.3f}. That is ordinary")
print("regression-to-the-mean shrinkage, and it is the same sag the tree and kernel notebooks found at the $500k cap.")
print(f"\nMore interesting is the middle panel. Overall coverage was {cov:.3f}, comfortably near the nominal 0.90 -- but")
print(f"per decile it runs from {min(cov_k):.3f} to {max(cov_k):.3f}. The intervals are far too WIDE for cheap blocks and too")
print("NARROW for expensive ones, and the two errors cancel in the average. A single coverage number can therefore")
print("look calibrated while the model is miscalibrated everywhere; only the conditional view shows it. For a risk")
print("application that is the difference between an error bar you can size a position with and one you cannot.")
print(f"\nCredit default: ECE {ece:.4f} over 10 quantile bins, mean predicted probability {pmn.mean():.3f} against a base")
print(f"rate of {ycte.mean():.3f}. The probabilities are close to honest across most of the range, drifting only in the top")
print(f"decile ({pp_[-1]:.3f} predicted against {op_[-1]:.3f} observed) where the model is UNDER-confident about the riskiest clients.")
print("Note this is a genuinely different question from section 4: the probabilities can be well calibrated while the")
print("uncertainty score built from them still fails as a triage rule, and here both are true at once.")
California, by decile of the prediction:
decile predicted actual gap 90% coverage
1 0.849 0.772 -0.077 0.995
2 1.129 1.031 -0.097 0.989
3 1.354 1.325 -0.028 0.974
4 1.546 1.539 -0.008 0.973
5 1.735 1.720 -0.016 0.950
6 1.965 1.994 +0.029 0.916
7 2.222 2.252 +0.030 0.868
8 2.571 2.632 +0.062 0.879
9 3.084 3.206 +0.122 0.814
10 4.056 4.220 +0.164 0.858
Two things the aggregate hid. The decile means bend AWAY from the diagonal at both ends -- the model
over-predicts the cheapest blocks by 0.077 and under-predicts the dearest by 0.164. That is ordinary
regression-to-the-mean shrinkage, and it is the same sag the tree and kernel notebooks found at the $500k cap.
More interesting is the middle panel. Overall coverage was 0.922, comfortably near the nominal 0.90 -- but
per decile it runs from 0.814 to 0.995. The intervals are far too WIDE for cheap blocks and too
NARROW for expensive ones, and the two errors cancel in the average. A single coverage number can therefore
look calibrated while the model is miscalibrated everywhere; only the conditional view shows it. For a risk
application that is the difference between an error bar you can size a position with and one you cannot.
Credit default: ECE 0.0127 over 10 quantile bins, mean predicted probability 0.217 against a base
rate of 0.221. The probabilities are close to honest across most of the range, drifting only in the top
decile (0.652 predicted against 0.710 observed) where the model is UNDER-confident about the riskiest clients.
Note this is a genuinely different question from section 4: the probabilities can be well calibrated while the
uncertainty score built from them still fails as a triage rule, and here both are true at once.
6. Summary — and the end of the flagship subsection¶
Bayesian deep learning gives a neural network an honest error bar. Two cheap, practical routes — MC-dropout (test-time dropout as posterior sampling) and deep ensembles (disagreement across random inits) — both produced near-calibrated predictive intervals (92% coverage against a nominal 90%, slightly conservative), uncertainty that grows away from the training data — though far from equally well, the ensemble separating extrapolation from in-data by about 12× where MC-dropout manages only 3× — and a usable, if modest, triage signal on classification. That last one came with the sharpest lesson of the notebook: the textbook selective-prediction demonstration — accuracy climbing from 0.82 to 0.90 as uncertain cases are deferred — turns out to be entirely the class mix. The lift over predicting the majority class falls to zero, and at 30% retention the model catches none of the defaults it exists to find. Uncertainty still separates errors from correct calls (AUC 0.64), which is real but far weaker than the accuracy curve advertises. Neither method is exact Bayes; both beat the silent overconfidence of a plain network; and neither excuses you from checking the metric.
The Bayesian through-line. This closes the loop with the portfolio's Bayesian core: BART delivered posterior credible intervals from a tree ensemble, Gaussian processes delivered exact posterior variance from a kernel — and here neural networks get the same calibrated uncertainty by approximation. Exact-but-limited (GP), sampling-based (BART), or scalable-approximate (MC-dropout / ensembles) are three routes to the one idea that runs through the whole portfolio: report what you don't know.
The subsection, complete. Five notebooks traced deep learning end to end: the MLP and backprop from scratch (and the honest tabular verdict), the CNN exploiting spatial structure, the LSTM on financial sequences, the transformer and attention's long-range power, and now Bayesian deep learning for uncertainty. The recurring lesson — match the architecture to the structure of the data, and know when a classical model still wins — is exactly the judgement a quant applies. The next subsections of the ML arc are Time-Series ML, Unsupervised Learning, Model Evaluation & Interpretability, and Financial ML methodology (López de Prado).