Generative vs Discriminative — Naive Bayes vs Logistic

Python · scikit-learn · R · e1071 · MASS

Two Ways to Build a Classifier

Almost everything else in this collection is generative: it writes down a probability model for how the data were produced, $p(x\mid y)\,p(y)$ , and reasons backward with Bayes' rule. Machine learning usually goes discriminative — it models the thing you actually want, $p(y\mid x)$ , directly, and never bothers with a model of the features. Classification is where the two meet head-on, which makes it the natural entry point to the arc.

\underbrace{p(y\mid x) \propto p(x\mid y)\,p(y)}_{\text{generative: model everything, invert}} \qquad\qquad \underbrace{p(y\mid x) = \sigma(\beta^\top x)}_{\text{discriminative: model only what you need}}

What Naive Bayes actually estimates

Naive Bayes is the purest generative classifier, and the whole of it fits in one assumption. Bayes' rule turns a model of the features into a classifier, $p(y\mid x)\propto p(x\mid y)\,p(y)$ , but $p(x\mid y)$ is a joint density over all 23 features — expensive to estimate and hopeless in high dimensions. The naive step is to assume the features are conditionally independent given the class, which factorises that joint into a product of one-dimensional pieces.

p(x\mid y) \;=\; \prod_{j=1}^{p} p(x_j \mid y), \qquad p(x_j\mid y=k) \;=\; \mathcal{N}\!\left(x_j;\ \mu_{jk},\ \sigma_{jk}^{2}\right)

In the Gaussian version used here, each piece is a normal density, so fitting the model means estimating a mean and a variance per feature per class, plus the class priors — $2pK+K$ numbers, every one of them a one-dimensional summary. Nothing has to be inverted, nothing is iterative, and each parameter is estimated from the full sample rather than from a shrinking slice of it. That is precisely why it converges so fast on little data, and it is the mechanical reason behind the Ng–Jordan result below.

\log p(y=k\mid x) \;=\; \log \pi_k \;-\; \tfrac12\sum_{j=1}^{p}\left[\log\!\left(2\pi\sigma_{jk}^{2}\right) + \frac{(x_j-\mu_{jk})^{2}}{\sigma_{jk}^{2}}\right] \;+\; \text{const}

The assumption is also, on this data, plainly false — the six bill-amount columns correlate at about 0.89. That is not fatal, and the reason it is not fatal is worth carrying: what determines a classification is the decision boundary, not the density model, and a wrong density can still imply a right boundary. Where it does hurt is in the numbers. Multiplying 23 dependent signals as though they were independent evidence compounds the same information over and over, and the posterior saturates at 0 or 1. Linear discriminant analysis (LDA) is the halfway house: it keeps the Gaussian model but replaces independence with a single shared covariance matrix across classes, which costs $p(p+1)/2$ more parameters and buys back a linear boundary and well-behaved probabilities.

The trade-off it implies

Ng & Jordan (2001) proved a precise trade-off. Committing to a model of $x$ means higher asymptotic error — the assumptions are usually wrong — but lower variance and faster convergence, since there are fewer effective parameters. So the generative classifier should win when data are scarce and lose when they are abundant.

Showing Both Halves of the Trade-Off

The controlled experiment shows the small-n half cleanly: with independent Gaussian features, Naive Bayes is clearly better when data are scarce and logistic catches up as $n$ grows. It cannot show the other half, and the reason is worth stating — in that design Naive Bayes' assumptions are correct, so it is asymptotically optimal too. Logistic converges to it and can never overtake.

Demonstrating the higher-asymptotic-error half is harder than it sounds, and the first attempt fails instructively. Merely correlating the features does not break Naive Bayes. What decides a classification is the decision boundary, not the density model, and for equicorrelated features with an equal mean shift the optimal direction is proportional to the all-ones vector — exactly the unweighted sum Naive Bayes implicitly forms. Its variance model is wrong and its direction is right, so it stays optimal (Domingos & Pazzani, 1997).

What breaks it is redundancy. Duplicate one informative feature ten times and Naive Bayes counts it as eleven independent votes, while logistic can learn to divide the weight — given enough data. There, Naive Bayes' error flattens at a floor while logistic keeps descending past it and overtakes at $n=20{,}000$ . That floor is the asymptotic bias, and both halves of the result are finally visible.

redundant-feature design	Naive Bayes error	logistic error	verdict
n = 30	0.341	0.419	NB better
n = 300	0.323	0.352	NB better
n = 2,000	0.323	0.327	NB better
n = 6,000	0.323	0.323	tied
n = 20,000	0.323 the floor	0.322	logistic better

On Real Data, the Bias Shows Up as Miscalibration

On the real credit data the story is subtler than a crossover, and better for it. Naive Bayes out-ranks logistic at almost every sample size — the gap survives its own standard error at all but one — yet its accuracy trails throughout. The extra bias does not surface as worse discrimination at all. It surfaces as miscalibration.

The number deserves stating plainly: Naive Bayes classifies at 0.519 accuracy against 0.779 for the do-nothing rule of predicting "no default" for everyone — 0.259 worse than having no model. And that is a threshold artefact, not a discrimination failure: move the cut-off to 0.99 and the same model reaches 0.804, level with logistic. The ranking was fine all along; 0.5 is simply the wrong place to cut a model whose probabilities are wrong, and its expected calibration error (ECE) — the average gap between a predicted probability and the frequency actually observed at that prediction — of 0.38 against logistic's 0.06 says how wrong. The table below reports the three together: AUC (area under the receiver-operating-characteristic curve) scores the ranking alone, accuracy scores a decision taken at a threshold, and ECE scores whether the probabilities themselves are true.

credit default, 30,000 clients	AUC (ranking)	accuracy	ECE (miscalibration)
Naive Bayes (generative)	0.719	0.519	0.378
LDA (generative)	0.710	0.812	0.049
Logistic (discriminative)	0.715	0.811	0.055
do-nothing baseline	0.500	0.779	—

What the R companion adds

The R companion makes the same point from an angle the Python side cannot. Its table differs from Python's — and not because the implementations differ. Given Python's exact train/test indices, e1071 reproduces its numbers to three decimals. The whole gap is the split. Across twenty random splits Naive Bayes' AUC — the area under the receiver-operating-characteristic (ROC) curve, which scores only the ranking and never looks at a threshold — is stable to a standard deviation of 0.007 while its accuracy swings with a standard deviation of 0.069 — sixteen times logistic's 0.004. A model whose probabilities pile up at 0 and 1 has an accuracy hinging on how many points fall either side of the cut. The ranking metric replicates and the threshold metric does not, which is the miscalibration story in one line.

R also surfaces something the Python optimiser hides. At the small-n end of the learning curve, nineteen of twenty logistic fits at $n=80$ fail to converge — with 25 features and a few dozen rows the classes are linearly separable and unregularised maximum likelihood has no finite solution. That is not an artefact to suppress: separation is the discriminative small-sample failure in its most extreme form, and Naive Bayes has no equivalent, since a mean and a variance always exist. It is the Ng–Jordan variance argument taken to its limit.

Where this sits

This retroactively explains Calibration, where Naive Bayes scored 29× the ECE noise floor: its miscalibration is the flip side of its low variance, and the same independence assumption explains both. The discriminative end of the ladder continues in SVM and the Discriminative Spectrum, and the logistic/SVM machinery is developed in Ridge, Lasso & Elastic Net and SVM and Kernel Methods. The generative side is the whole Bayesian core of the collection — the latent class, mixture and hierarchical models all write down $p(x\mid y)p(y)$ .

Notebooks

Python Gaussian Naive Bayes and LDA built from scratch and validated against scikit-learn, the Ng-Jordan trade-off demonstrated in both directions using a redundant-feature design that a merely correlated one cannot achieve, and the credit-data result diagnosed as miscalibration rather than a missing crossover (Python) Open → R e1071, MASS and glm reproducing the conclusions, plus two things the Python side leaves implicit: the cross-language gap traced entirely to the train/test split, and the separation behind the small-n logistic fits counted rather than left in the warning stream (R) Open →

Downloads

credit_default.csv Default of credit-card clients — 30,000 clients, 23 features, 22.1% default rate (Yeh & Lien 2009; UCI). A credit limit, four demographics, and eighteen months of payment history: six of repayment status, six of bill amounts, six of payments made. Shared by both examples in this section, and by the tree-ensemble, calibration and interpretability sections

References

Ng, A. Y. & Jordan, M. I. (2001). On discriminative vs. generative classifiers: a comparison of logistic regression and naive Bayes. NeurIPS 14. — the trade-off tested here
Domingos, P. & Pazzani, M. (1997). On the optimality of the simple Bayesian classifier under zero-one loss. Machine Learning 29, 103–130. — why correlation alone does not break Naive Bayes
Efron, B. (1975). The efficiency of logistic regression compared to normal discriminant analysis. JASA 70(352), 892–898. — the original efficiency comparison
Albert, A. & Anderson, J. A. (1984). On the existence of maximum likelihood estimates in logistic regression models. Biometrika 71(1), 1–10. — separation, and why the small-n fits do not converge
Yeh, I.-C. & Lien, C.-H. (2009). The comparisons of data mining techniques for the predictive accuracy of probability of default of credit card clients. Expert Systems with Applications 36(2), 2473–2480. — the data