ML Arc, Foundations — Generative vs Discriminative Classifiers¶
Naive Bayes & LDA model p(x∣y)p(y); logistic regression & SVMs model p(y∣x) — and the trade-off between them¶
This is the natural first stop in the Machine-Learning arc, because it is the cleanest bridge from the rest of this portfolio into it. Almost everything built here so far — the Bayesian hierarchical models, the mixtures, the latent-variable models — is generative: it writes down a probability model for how the data were produced, $p(x\mid y)\,p(y)$, and reasons backward. Machine learning, by contrast, usually goes discriminative: it models the thing you actually want, $p(y\mid x)$, directly, and never bothers with a model of the features. Classification is where the two philosophies meet head-on:
- Generative classifiers — Naive Bayes, Linear/Quadratic Discriminant Analysis — estimate the class priors $p(y)$ and the class-conditional feature densities $p(x\mid y)$, then invert with Bayes' rule: $p(y\mid x)\propto p(x\mid y)\,p(y)$.
- Discriminative classifiers — logistic regression, SVMs — parameterize $p(y\mid x)$ (or just the decision boundary) and fit it directly, making no assumption about the distribution of $x$.
The contrast is not academic. Ng & Jordan (2001) proved a precise trade-off: because a generative classifier commits to a model of $x$, it has higher asymptotic error (its assumptions are usually wrong) but lower variance and faster convergence (fewer effective parameters), so it tends to win when data are scarce and lose when data are abundant. We locate that trade-off empirically — and find something more interesting than the textbook version on our data. Data: the Taiwan credit-default panel (23 features), the same one used in the Calibration notebook. Python-lead (from-scratch Naive Bayes and linear discriminant analysis, LDA + scikit-learn); R companion uses e1071, MASS, and glm.
1. The two recipes, built from scratch¶
Gaussian Naive Bayes is the purest generative classifier. It assumes the features are conditionally independent given the class and Gaussian, so $p(x\mid y)=\prod_j \mathcal N(x_j;\mu_{jy},\sigma_{jy}^2)$ — it just estimates a mean and variance per feature per class, plus the class priors. Linear Discriminant Analysis relaxes the independence assumption to a shared full covariance across classes, which (with Gaussian classes) yields a linear boundary. Both then classify by Bayes' rule. Logistic regression ignores $p(x)$ entirely and fits the log-odds of $y$ as a linear function of $x$ by maximum likelihood.
We implement Gaussian NB and LDA from scratch and compare them to logistic regression on the credit data. Two very different scorecards emerge: on ranking (AUC) the generative models are competitive — Naive Bayes even edges out logistic — but on calibration (are the probabilities right?) and threshold accuracy, Naive Bayes is much worse, because its independence assumption pushes probabilities to the extremes. Discrimination and calibration are different things, and the generative/discriminative choice trades them off.
import numpy as np, pandas as pd, matplotlib.pyplot as plt, warnings
warnings.filterwarnings("ignore")
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import roc_auc_score, accuracy_score
BLUE="#2b6cb0"; RED="#c53030"; GREEN="#2f855a"; ORANGE="#dd6b20"; GREY="#a0aec0"; PURP="#6b46c1"
d=pd.read_csv("credit_default.csv"); feat=[c for c in d.columns if c!="default"]
X=d[feat].values.astype(float); y=d["default"].values
Xtr,Xte,ytr,yte=train_test_split(X,y,test_size=0.3,random_state=0,stratify=y)
sc=StandardScaler().fit(Xtr); Ztr=sc.transform(Xtr); Zte=sc.transform(Xte)
class GaussianNBscratch: # generative: p(x|y)=prod N(mu,sigma^2), independence
def fit(s,X,y):
s.cl=np.unique(y); s.pi=np.array([np.mean(y==c) for c in s.cl])
s.mu=np.array([X[y==c].mean(0) for c in s.cl]); s.var=np.array([X[y==c].var(0)+1e-9 for c in s.cl]); return s
def log_post(s,X):
lp=[]
for k,c in enumerate(s.cl):
ll=-0.5*np.sum(np.log(2*np.pi*s.var[k])+((X-s.mu[k])**2)/s.var[k],axis=1)+np.log(s.pi[k]); lp.append(ll)
return np.array(lp).T
def predict_proba(s,X): L=s.log_post(X); L-=L.max(1,keepdims=True); P=np.exp(L); return P/P.sum(1,keepdims=True)
class LDAscratch: # generative: shared full covariance -> linear boundary
def fit(s,X,y):
s.cl=np.unique(y); s.pi=np.array([np.mean(y==c) for c in s.cl]); s.mu=np.array([X[y==c].mean(0) for c in s.cl])
S=sum(((X[y==c]-s.mu[k]).T@(X[y==c]-s.mu[k])) for k,c in enumerate(s.cl))/(len(y)-len(s.cl)); s.Si=np.linalg.pinv(S); return s
def predict_proba(s,X):
sc=np.array([X@s.Si@s.mu[k]-0.5*s.mu[k]@s.Si@s.mu[k]+np.log(s.pi[k]) for k in range(len(s.cl))]).T
sc-=sc.max(1,keepdims=True); P=np.exp(sc); return P/P.sum(1,keepdims=True)
from sklearn.linear_model import LogisticRegression
def ece(p,y,nb=10):
b=np.linspace(0,1,nb+1); e=0.0
for i in range(nb):
m=(p>=b[i])&(p<b[i+1] if i<nb-1 else p<=b[i+1])
if m.sum(): e+=m.mean()*abs(y[m].mean()-p[m].mean())
return e
models={"Naive Bayes (generative)":GaussianNBscratch().fit(Ztr,ytr),"LDA (generative)":LDAscratch().fit(Ztr,ytr),
"Logistic (discriminative)":LogisticRegression(max_iter=2000).fit(Ztr,ytr)}
rows=[]
for nm,m in models.items():
p=m.predict_proba(Zte)[:,1]; rows.append((nm,roc_auc_score(yte,p),accuracy_score(yte,(p>=.5).astype(int)),ece(p,yte)))
tab=pd.DataFrame(rows,columns=["model","AUC (ranking)","accuracy","ECE (miscalibration)"]).set_index("model")
print(tab.round(3).to_string())
from sklearn.naive_bayes import GaussianNB as _skNB
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis as _skLDA
_pn=models["Naive Bayes (generative)"].predict_proba(Zte)[:,1]; _pl=models["LDA (generative)"].predict_proba(Zte)[:,1]
print(f"\nFrom-scratch vs scikit-learn: Naive Bayes agrees to {np.abs(_pn-_skNB().fit(Ztr,ytr).predict_proba(Zte)[:,1]).max():.1e}, "
f"LDA to {np.abs(_pl-_skLDA(solver='lsqr').fit(Ztr,ytr).predict_proba(Zte)[:,1]).max():.1e}.")
_base=max(yte.mean(),1-yte.mean())
_nbacc=accuracy_score(yte,(_pn>=.5).astype(int))
_bestacc,_bestthr=max(((accuracy_score(yte,(_pn>=t).astype(int)),t) for t in np.linspace(0.01,0.99,99)))
print(f"\nNaive Bayes matches -- even beats -- logistic on AUC: it RANKS defaults well. Its accuracy is another matter,")
print(f"and the number deserves stating plainly: {_nbacc:.3f}, against {_base:.3f} for the do-nothing rule of predicting")
print(f"'no default' for everyone. Naive Bayes at the default threshold is {_base-_nbacc:.3f} WORSE than not having a model.")
print(f"\nThat is not a discrimination failure, and the fix shows it. Move the cut-off to {_bestthr:.2f} and the same model")
print(f"reaches {_bestacc:.3f} -- essentially level with logistic's {accuracy_score(yte,(models['Logistic (discriminative)'].predict_proba(Zte)[:,1]>=.5).astype(int)):.3f}. The ranking was fine all along; 0.5 is simply the wrong")
print(f"place to cut a model whose probabilities are wrong, and its ECE of {ece(_pn,yte):.2f} against logistic's {ece(models['Logistic (discriminative)'].predict_proba(Zte)[:,1],yte):.2f} says exactly")
print("how wrong. LDA, which relaxes independence to a shared covariance, sits with logistic on both counts.")
AUC (ranking) accuracy ECE (miscalibration) model Naive Bayes (generative) 0.719 0.519 0.378 LDA (generative) 0.710 0.812 0.049 Logistic (discriminative) 0.715 0.811 0.055 From-scratch vs scikit-learn: Naive Bayes agrees to 1.1e-14, LDA to 4.2e-05. Naive Bayes matches -- even beats -- logistic on AUC: it RANKS defaults well. Its accuracy is another matter, and the number deserves stating plainly: 0.519, against 0.779 for the do-nothing rule of predicting 'no default' for everyone. Naive Bayes at the default threshold is 0.259 WORSE than not having a model. That is not a discrimination failure, and the fix shows it. Move the cut-off to 0.99 and the same model reaches 0.804 -- essentially level with logistic's 0.811. The ranking was fine all along; 0.5 is simply the wrong place to cut a model whose probabilities are wrong, and its ECE of 0.38 against logistic's 0.06 says exactly how wrong. LDA, which relaxes independence to a shared covariance, sits with logistic on both counts.
2. The Ng-Jordan trade-off — generative converges faster¶
Ng & Jordan's result is about how fast each estimator learns. A generative classifier estimates a handful of simple quantities (per-feature means and variances), so its parameter estimates stabilize with very little data — low variance, fast convergence — at the cost of a wrong model (higher asymptotic error / bias). A discriminative classifier fits the boundary directly, so it is asymptotically better but needs more data to get there, and overfits when $n$ is small, especially with many features. The prediction: generative wins at small $n$, discriminative wins at large $n$, with a crossover in between.
We locate this in a controlled setting where Naive Bayes' assumptions hold (independent Gaussian features): plotting test error against training size, Naive Bayes is clearly better when data are scarce, and logistic regression catches up as $n$ grows.
That shows the small-$n$ half of the trade-off, which is the robust one — and it cannot show the other half, for a reason worth making explicit. In this design Naive Bayes' assumptions are correct, so it is asymptotically optimal too; logistic can converge to it but never beat it. Demonstrating the higher asymptotic error half requires a world where the independence assumption is actually wrong in a way that matters.
That turns out to be harder to arrange than it sounds, and the reason is instructive. Merely correlating the features does not do it. What decides a classification is the decision boundary, not the density model, and for equicorrelated features with an equal mean shift in each the optimal direction is proportional to the all-ones vector — exactly the unweighted sum Naive Bayes implicitly forms. Its variance model is wrong and its direction is right, so it stays optimal (Domingos & Pazzani, 1997: Naive Bayes can be optimal under strong dependence).
What breaks it is redundancy. Duplicate one informative feature ten times and Naive Bayes counts it as eleven independent votes, while logistic regression can learn to divide the weight among them — given enough data to do so. Both designs are run below.
from sklearn.naive_bayes import GaussianNB
rng=np.random.default_rng(0); p=25
def gen(n): # ~independent Gaussian features, class shifts each mean (NB roughly OK)
yy=(rng.uniform(size=n)<0.5).astype(int); XX=rng.normal(0,1,(n,p))+(yy[:,None]*0.45); return XX,yy
Xtst,ytst=gen(12000); ns=[30,50,80,150,300,700,2000,6000]
nb_e=[]; lg_e=[]
for n in ns:
a=[];b=[]
for r in range(40):
Xs,ys=gen(n)
if len(np.unique(ys))<2: continue
a.append(1-accuracy_score(ytst,GaussianNB().fit(Xs,ys).predict(Xtst)))
b.append(1-accuracy_score(ytst,LogisticRegression(max_iter=2000,C=1e6).fit(Xs,ys).predict(Xtst)))
nb_e.append(np.mean(a)); lg_e.append(np.mean(b))
fig,ax=plt.subplots(figsize=(8.5,4.6))
ax.plot(ns,nb_e,"o-",color=GREEN,lw=2,label="Naive Bayes (generative)")
ax.plot(ns,lg_e,"s-",color=BLUE,lw=2,label="Logistic (discriminative)")
ax.set_xscale("log"); ax.set_xlabel("training size n (log scale)"); ax.set_ylabel("test error"); ax.set_title("Ng-Jordan: generative converges faster (wins at small n)"); ax.legend()
plt.tight_layout(); plt.show()
print(f"Independent features, where Naive Bayes' assumption is TRUE:")
print(f" at n={ns[0]} Naive Bayes errs {nb_e[0]:.3f} against logistic's {lg_e[0]:.3f}; by n={ns[-1]} they are level ({nb_e[-1]:.3f} vs {lg_e[-1]:.3f}).")
print(" Fewer effective parameters, so the generative estimator stabilises on little data. But logistic only CATCHES UP")
print(" -- it never overtakes -- because here Naive Bayes is asymptotically optimal too. This design cannot show the")
print(" other half of Ng-Jordan, and pretending otherwise would be the easy mistake.")
# --- a world where the independence assumption actually costs something -------------------------------
def gen_redundant(n,rg,p_noise=10,n_copies=10,sig=0.9):
"""one informative feature, ten near-duplicates of it, plus independent noise.
the optimal rule uses that direction ONCE; Naive Bayes counts it eleven times."""
yy=(rg.uniform(size=n)<0.5).astype(int)
core=rg.normal(0,1,n)+yy*sig
copies=core[:,None]+rg.normal(0,0.15,(n,n_copies))
return np.column_stack([core,copies,rg.normal(0,1,(n,p_noise))]), yy
rg2=np.random.default_rng(3); Xr,yr=gen_redundant(20000,rg2)
ns_r=[30,60,120,300,800,2000,6000,20000]; nbr=[];lgr=[];sig_lg=None
print(f"\nRedundant features, where it is FALSE (1 informative + 10 near-copies + 10 noise):")
print(f" {'n':>7} {'NB error':>11} {'logistic':>11} {'difference':>13} verdict")
for n in ns_r:
a=[];b=[]
for _ in range(25):
Xs,ys=gen_redundant(n,rg2)
if len(np.unique(ys))<2: continue
a.append(1-accuracy_score(yr,GaussianNB().fit(Xs,ys).predict(Xr)))
b.append(1-accuracy_score(yr,LogisticRegression(max_iter=3000,C=1e6).fit(Xs,ys).predict(Xr)))
a=np.array(a);b=np.array(b); dd=a-b; se=dd.std(ddof=1)/np.sqrt(len(dd))
v="NB better" if dd.mean()<-2*se else ("logistic better" if dd.mean()>2*se else "tied")
if v=="logistic better" and sig_lg is None: sig_lg=n
nbr.append(a.mean()); lgr.append(b.mean())
print(f" {n:>7} {a.mean():>11.4f} {b.mean():>11.4f} {dd.mean():>+13.4f} {v}")
print(f"\n Now both halves are visible. Naive Bayes still wins when data are scarce -- by a wider margin than before,")
print(f" since logistic has 21 correlated features to fit -- and logistic first overtakes significantly at n = {sig_lg}.")
print(f" Naive Bayes' error flattens at {min(nbr):.3f} and stops improving: that floor IS the asymptotic bias, the price of")
print(" a wrong model of x. Logistic keeps descending past it because it never modelled x in the first place.")
fig,ax=plt.subplots(1,2,figsize=(13,4.4))
ax[0].plot(ns,nb_e,"o-",color=GREEN,lw=2,label="Naive Bayes"); ax[0].plot(ns,lg_e,"s-",color=BLUE,lw=2,label="Logistic")
ax[0].set_xscale("log"); ax[0].set_xlabel("training size n"); ax[0].set_ylabel("test error")
ax[0].set_title("Independent features: NB optimal, logistic converges to it"); ax[0].legend(fontsize=8)
ax[1].plot(ns_r,nbr,"o-",color=GREEN,lw=2,label="Naive Bayes"); ax[1].plot(ns_r,lgr,"s-",color=BLUE,lw=2,label="Logistic")
ax[1].set_xscale("log"); ax[1].set_xlabel("training size n"); ax[1].set_ylabel("test error")
ax[1].set_title("Redundant features: NB hits a floor, logistic passes it"); ax[1].legend(fontsize=8)
plt.tight_layout(); plt.show()
Independent features, where Naive Bayes' assumption is TRUE:
at n=30 Naive Bayes errs 0.263 against logistic's 0.287; by n=6000 they are level (0.128 vs 0.128).
Fewer effective parameters, so the generative estimator stabilises on little data. But logistic only CATCHES UP
-- it never overtakes -- because here Naive Bayes is asymptotically optimal too. This design cannot show the
other half of Ng-Jordan, and pretending otherwise would be the easy mistake.
Redundant features, where it is FALSE (1 informative + 10 near-copies + 10 noise):
n NB error logistic difference verdict
30 0.3407 0.4186 -0.0779 NB better
60 0.3274 0.4136 -0.0862 NB better
120 0.3250 0.3834 -0.0584 NB better
300 0.3231 0.3529 -0.0297 NB better
800 0.3223 0.3345 -0.0121 NB better
2000 0.3225 0.3262 -0.0037 NB better
6000 0.3228 0.3230 -0.0002 tied
20000 0.3232 0.3222 +0.0010 logistic better Now both halves are visible. Naive Bayes still wins when data are scarce -- by a wider margin than before, since logistic has 21 correlated features to fit -- and logistic first overtakes significantly at n = 20000. Naive Bayes' error flattens at 0.322 and stops improving: that floor IS the asymptotic bias, the price of a wrong model of x. Logistic keeps descending past it because it never modelled x in the first place.
3. On the credit data, the higher 'bias' shows up as miscalibration¶
The textbook expectation is that on any real dataset Naive Bayes should win at small $n$ and lose at large $n$. The credit data tells a subtler, more honest story. We plot both AUC (ranking) and accuracy learning curves against training size. On AUC, Naive Bayes converges fast and stays ahead of logistic at every sample size — its class-conditional Gaussians capture the ranking signal well. Yet on accuracy it is worse at every size. There is no clean accuracy crossover, because Naive Bayes' extra 'bias' here does not manifest as poor discrimination — it manifests as miscalibration: the independence assumption makes the probabilities wrong, so the default 0.5 threshold classifies badly even though the ranking is excellent.
This is exactly the phenomenon the Calibration notebook flagged (Naive Bayes: good AUC, terrible ECE) — and it is the same independence assumption from two angles. Ng-Jordan says that assumption buys Naive Bayes low variance; the Calibration notebook says it costs Naive Bayes correct probabilities. Both are consequences of committing to a (wrong) generative model of $x$.
Xall=sc.transform(X); idx=np.random.default_rng(1).permutation(len(X)); teI=idx[:8000]; pool=idx[8000:]
Zt=Xall[teI]; yt=y[teI]
ns2=[40,100,300,900,3000,10000] # 22000 would BE the whole pool: 15 identical fits, no variability
nb_auc=[];lg_auc=[];nb_acc=[];lg_acc=[];auc_se=[]
for n in ns2:
aa=[];bb=[];cc=[];dd=[]
for r in range(15):
s=np.random.default_rng(r).choice(pool,min(n,len(pool)),replace=False); Zs=Xall[s]; ys=y[s]
if len(np.unique(ys))<2: continue
nb=GaussianNB().fit(Zs,ys); lg=LogisticRegression(max_iter=1500,C=1e6).fit(Zs,ys)
aa.append(roc_auc_score(yt,nb.predict_proba(Zt)[:,1])); bb.append(roc_auc_score(yt,lg.predict_proba(Zt)[:,1]))
cc.append(accuracy_score(yt,nb.predict(Zt))); dd.append(accuracy_score(yt,lg.predict(Zt)))
nb_auc.append(np.mean(aa)); lg_auc.append(np.mean(bb)); nb_acc.append(np.mean(cc)); lg_acc.append(np.mean(dd))
_df=np.array(aa)-np.array(bb); auc_se.append(_df.std(ddof=1)/np.sqrt(len(_df)))
fig,ax=plt.subplots(1,2,figsize=(13,4.4))
ax[0].plot(ns2,nb_auc,"o-",color=GREEN,lw=2,label="Naive Bayes"); ax[0].plot(ns2,lg_auc,"s-",color=BLUE,lw=2,label="Logistic")
ax[0].set_xscale("log"); ax[0].set_xlabel("training size n"); ax[0].set_ylabel("test AUC"); ax[0].set_title("Ranking (AUC): NB converges fast, stays ahead"); ax[0].legend(fontsize=8)
ax[1].plot(ns2,nb_acc,"o-",color=GREEN,lw=2,label="Naive Bayes"); ax[1].plot(ns2,lg_acc,"s-",color=BLUE,lw=2,label="Logistic")
ax[1].set_xscale("log"); ax[1].set_xlabel("training size n"); ax[1].set_ylabel("test accuracy"); ax[1].set_title("Accuracy: NB worse throughout (miscalibration)"); ax[1].legend(fontsize=8)
plt.tight_layout(); plt.show()
print(f" {'n':>7} {'NB AUC':>9} {'logistic':>10} {'difference':>12} {'se':>8} real?")
for i,n in enumerate(ns2):
_d=nb_auc[i]-lg_auc[i]
print(f" {n:>7} {nb_auc[i]:>9.4f} {lg_auc[i]:>10.4f} {_d:>+12.4f} {auc_se[i]:>8.4f} {'yes' if abs(_d)>2*auc_se[i] else 'no'}")
print(f"\nCredit: Naive Bayes out-RANKS logistic at almost every sample size, and the gap survives its own standard error")
print(f"at all but one of them -- so this is not noise. Its accuracy ({nb_acc[-1]:.3f}) nevertheless trails logistic ({lg_acc[-1]:.3f}) throughout.")
print("There is no accuracy crossover here because Naive Bayes' extra bias does not surface as worse discrimination at all;")
print("it surfaces as miscalibration, which the default 0.5 threshold then converts into bad decisions.")
print("\n(The curve stops at n=10000 deliberately. The training pool holds 22000 rows, so a point at n=22000 would draw the")
print("same rows every time and report fifteen identical fits as an average -- a point with no variability by construction.)")
n NB AUC logistic difference se real?
40 0.6299 0.5607 +0.0692 0.0139 yes
100 0.6719 0.6321 +0.0399 0.0072 yes
300 0.7000 0.6866 +0.0134 0.0039 yes
900 0.7087 0.7081 +0.0006 0.0034 no
3000 0.7239 0.7187 +0.0051 0.0024 yes
10000 0.7312 0.7236 +0.0076 0.0014 yes
Credit: Naive Bayes out-RANKS logistic at almost every sample size, and the gap survives its own standard error
at all but one of them -- so this is not noise. Its accuracy (0.694) nevertheless trails logistic (0.798) throughout.
There is no accuracy crossover here because Naive Bayes' extra bias does not surface as worse discrimination at all;
it surfaces as miscalibration, which the default 0.5 threshold then converts into bad decisions.
(The curve stops at n=10000 deliberately. The training pool holds 22000 rows, so a point at n=22000 would draw the
same rows every time and report fifteen identical fits as an average -- a point with no variability by construction.)
4. The reliability curve makes it concrete¶
To see the miscalibration directly we plot the predictions against what actually happened, three ways.
The first two panels are the classification counterpart of a forecast-versus-actual chart: sort the test clients by predicted probability, draw each model's prediction as a line, and overlay the actual default rate among neighbouring clients in that sorted order. A well-behaved model's prediction should sit on top of the realised rate all the way along. The third panel is the standard reliability diagram — bin by predicted probability and plot observed frequency against mean prediction. Logistic regression hugs the 45° line (well-calibrated); Naive Bayes' curve sticks to the axes — it says 0.99 for cases that default far less often and 0.01 for cases that default more, the signature of the independence assumption forcing over-confident probabilities. Same excellent ranking, wildly wrong numbers. This is why, in the Calibration notebook, isotonic/Platt recalibration was needed for Naive Bayes but not for logistic regression: recalibration fixes the probabilities the generative assumption distorts, without touching the ranking it gets right.
def reliability(p,yv,nb=10):
b=np.linspace(0,1,nb+1); xs=[];ys=[]
for i in range(nb):
m=(p>=b[i])&(p<b[i+1] if i<nb-1 else p<=b[i+1])
if m.sum()>30: xs.append(p[m].mean()); ys.append(yv[m].mean())
return np.array(xs),np.array(ys)
pnb=models["Naive Bayes (generative)"].predict_proba(Zte)[:,1]; plg=models["Logistic (discriminative)"].predict_proba(Zte)[:,1]
# --- predictions against actual, sorted by predicted probability -----------------------------
def sorted_view(p, yv, win=400):
o = np.argsort(p); ps = p[o]; ys = yv[o].astype(float)
k = np.ones(win)/win
act = np.convolve(ys, k, mode="same") # local realised default rate
act[:win//2] = np.nan; act[-win//2:] = np.nan # edges of the moving window are not meaningful
return np.arange(len(ps)), ps, act
fig,ax=plt.subplots(1,3,figsize=(16,4.4))
for a,(p,c,nm) in zip(ax[:2],[(pnb,GREEN,"Naive Bayes"),(plg,BLUE,"Logistic")]):
xs,ps,act=sorted_view(p,yte)
a.plot(xs,act,color="#cbd5e0",lw=1.8,zorder=1,label="actual default rate (local)")
a.plot(xs,ps,"--",color=c,lw=1.3,zorder=3,label=f"{nm} predicted")
a.set_ylim(-0.03,1.03); a.set_xlabel("test clients, sorted by predicted probability")
a.set_ylabel("P(default)"); a.set_title(f"{nm}: prediction vs what happened"); a.legend(fontsize=8,loc="upper left")
ax[2].plot([0,1],[0,1],"k--",lw=1,label="perfect calibration")
for p,c,nm in [(pnb,GREEN,"Naive Bayes"),(plg,BLUE,"Logistic")]:
xs2,ys2=reliability(p,yte); ax[2].plot(xs2,ys2,"o-",color=c,lw=2,label=f"{nm} (ECE {ece(p,yte):.2f})")
ax[2].set_xlabel("mean predicted probability"); ax[2].set_ylabel("observed default frequency")
ax[2].set_title("Reliability diagram"); ax[2].legend(fontsize=8)
plt.tight_layout(); plt.show()
_lo=pnb<0.01; _hi=pnb>0.99
print(f"The first panel is the failure in one picture. A quarter of all clients sit at one extreme or the other:")
print(f"{np.mean(_lo):.0%} are given a predicted probability below 0.01 and {np.mean(_hi):.0%} one above 0.99, and the dashed line spends")
print("long stretches flat against the floor and the ceiling rather than tracking the grey one.")
print(f"\nBoth extremes are wrong, in opposite directions and for the same reason. Among the clients Naive Bayes calls")
print(f"essentially safe, {yte[_lo].mean():.1%} default -- not the ~0% claimed. Among those it calls essentially certain to default,")
print(f"only {yte[_hi].mean():.1%} do -- not the ~100% claimed. That is overconfidence at both ends, and it means the predictions the")
print("model is SUREST about are the ones furthest from the truth, which is the worst arrangement a probability can have.")
print(f"\nLogistic's dashed line tracks the grey one across the whole range, rising smoothly from about {plg.min():.2f} to {plg.max():.2f}.")
print("It is not a better RANKER -- the sorted order is what AUC measures, and Naive Bayes wins that. It is a better")
print("estimator of the number, which is what the third panel scores and what any decision at a threshold depends on.")
print("\nThe independence assumption multiplies many 'votes' as if independent, so the posterior saturates near 0 or 1 -- great")
print("for ranking, terrible for probabilities. Generative low-variance (Ng-Jordan) and generative miscalibration (Calibration)")
print("are the same coin. This is the bias-variance trade-off of the generative-vs-discriminative choice, made concrete.")
The first panel is the failure in one picture. A quarter of all clients sit at one extreme or the other: 12% are given a predicted probability below 0.01 and 13% one above 0.99, and the dashed line spends long stretches flat against the floor and the ceiling rather than tracking the grey one. Both extremes are wrong, in opposite directions and for the same reason. Among the clients Naive Bayes calls essentially safe, 12.7% default -- not the ~0% claimed. Among those it calls essentially certain to default, only 59.2% do -- not the ~100% claimed. That is overconfidence at both ends, and it means the predictions the model is SUREST about are the ones furthest from the truth, which is the worst arrangement a probability can have. Logistic's dashed line tracks the grey one across the whole range, rising smoothly from about 0.00 to 0.99. It is not a better RANKER -- the sorted order is what AUC measures, and Naive Bayes wins that. It is a better estimator of the number, which is what the third panel scores and what any decision at a threshold depends on. The independence assumption multiplies many 'votes' as if independent, so the posterior saturates near 0 or 1 -- great for ranking, terrible for probabilities. Generative low-variance (Ng-Jordan) and generative miscalibration (Calibration) are the same coin. This is the bias-variance trade-off of the generative-vs-discriminative choice, made concrete.
5. Summary¶
Classification is where this portfolio's generative habit meets machine learning's discriminative default, and the two make an honest trade-off:
- Generative (Naive Bayes, LDA) models $p(x\mid y)p(y)$ and inverts with Bayes' rule; discriminative (logistic, SVM) models $p(y\mid x)$ directly.
- Ng & Jordan (2001): generative classifiers have higher asymptotic error but lower variance and faster convergence — so they win at small $n$. We confirmed the robust half in a controlled setting (Naive Bayes beat logistic when data were scarce; logistic caught up as $n$ grew).
- On the credit data the story was subtler and more instructive than a clean crossover: Naive Bayes actually out-ranked logistic (higher AUC) at every sample size, but its accuracy trailed throughout — because its extra 'bias' surfaced as miscalibration, not poor discrimination. The reliability diagram showed Naive Bayes' probabilities pinned to 0 and 1.
- The unifying idea is bias-variance: committing to a (wrong) model of $x$ buys low variance and fast convergence (Ng-Jordan) but costs correct probabilities (Calibration). The same independence assumption explains both.
Placement and cross-links. This is the arc's opening precisely because it connects the whole collection: the generative modelling of the Bayesian, mixture, and latent-variable notebooks is one side; the discriminative logistic/SVM/tree methods that follow are the other. It retroactively explains the Calibration notebook (Naive Bayes' miscalibration is the flip side of its low variance), sets up the Regularized & Kernel notebook (logistic and SVMs as discriminative learners), and its bias-variance framing recurs throughout. Guidance: reach for a generative classifier (Naive Bayes) for small, wide, or streaming problems and when speed matters; reach for a discriminative one when data are plentiful and calibrated probabilities matter — and recalibrate a generative classifier before trusting its numbers.