← All examples

Machine Learning: Generative vs Discriminative

This is the bridge from the rest of the collection into machine learning, and it is worth crossing deliberately. Almost everything else here is generative: the hierarchical models, the mixtures, the latent-variable models all write down a probability model for how the data were produced and reason backward. Machine learning usually goes discriminative — it models the thing you actually want and never bothers with a model of the features at all.

Classification is where the two philosophies meet head-on, and they make a genuine trade rather than one being better. Committing to a model of the features buys low variance and fast convergence and costs asymptotic accuracy when the model is wrong. Committing to less is more robust and more data-hungry. Arranged by how much each method commits to, the classifiers form a ladder: generative models everything, logistic models the conditional, the SVM models only the boundary.

One thing to be clear about, because the pairing invites the opposite conclusion. Logistic regression is discriminative: it estimates p(y | x) directly and never models how the features are distributed. The confusion is understandable, though, because a generative model implies it — assume Gaussian class-conditionals with one shared covariance matrix, which is exactly linear discriminant analysis (LDA), and the posterior that falls out is precisely the logistic function. Same functional form, different estimator: LDA fits class means and a covariance and derives the boundary, while logistic fits the boundary coefficients directly by conditional likelihood and learns nothing about the features at all. That shared form with different fitting is what makes them a matched pair, and it is the whole reason the convergence comparison below is a fair one.

That ladder frames every classifier in the arc that follows, and it turns out to be a calibration story as well as a convergence one — with the ordering coming out differently from the way it is usually told.

The data. Both examples run on the same cross-section: 30,000 Taiwanese credit-card clients, 23 features, and a 22.1% default rate (Yeh & Lien, 2009; UCI). The features are a credit limit, four demographics — sex, education, marital status and age — and then eighteen columns of payment history: six months of repayment status (PAY_1…PAY_6, coded in months of delay), six of bill amounts and six of payments made. The target is whether the client defaults the following month.

It is a good dataset for this comparison for two reasons. The features are strongly dependent — the six bill amounts correlate at about 0.89, and repayment status persists from month to month — so Naive Bayes' conditional-independence assumption is emphatically false, which is exactly the condition under which the generative/discriminative trade-off is supposed to bite. And the classes are imbalanced and heavily overlapping, so ranking and thresholded accuracy come apart, which is what lets the miscalibration story surface at all. The same cross-section is used by the tree ensembles, calibration and interpretability sections, so results here can be read directly against those, and the support vector machine gets its full treatment in SVM and Kernel Methods.

Three numbers, three different questions. Every comparison below is reported in the same three metrics, and the whole point of the section is that they do not have to agree. AUC is the area under the ROC curve — receiver operating characteristic, a name inherited from wartime radar and worth ignoring; the curve plots the true-positive rate against the false-positive rate as the threshold sweeps from one extreme to the other. The area under it has a clean reading: it is the probability that a randomly chosen defaulter is scored above a randomly chosen non-defaulter. It asks only whether the ordering is right, so it never looks at a threshold and never looks at whether the numbers themselves mean anything; 0.5 is a coin flip. Accuracy is the fraction of correct calls once a threshold is fixed, conventionally at 0.5, and it is therefore a statement about a decision rather than about the model. It has to be read against what you would get for nothing: 22.1% of these clients default, so predicting “no default” for everybody scores 0.779, and any accuracy below that is worse than having no model at all. ECE, the expected calibration error, asks the third question: are the probabilities true? Group the predictions into bins by what they claimed, compare each bin’s claim with what actually happened in it, and average the gaps. Zero is perfect. Because it lives on the probability scale it reads directly — an ECE of 0.38 means the stated probabilities are off by about 0.38 on average, which is not a subtle defect.

Why that matters here. Naive Bayes on this data scores an AUC of 0.719, the best of the three models, and an accuracy of 0.519, well below the 0.779 you get by predicting nothing. Both are true at once, and they are not in tension: the model ranks clients correctly and then attaches nonsense probabilities to that ranking, so a cut at 0.5 falls in the wrong place. Moving the threshold recovers 0.804 without changing the model at all. The ECE of 0.378 is what names the actual defect, and the notebook makes it concrete: among the clients Naive Bayes calls essentially safe, 12.7% default; among those it calls essentially certain, only 59.2% do. It is most wrong exactly where it is most confident, which is the worst arrangement a probability can have. Report only AUC and this model looks like the best of the three; report only accuracy and it looks broken; you need all three to say what is actually happening.

Native probabilities, and supplied ones. The models differ in whether they produce probabilities at all. Naive Bayes and logistic regression have them built in — they are fitted by maximising a likelihood, so a probability falls out of the fit, which is what “native” means below. A support vector machine has none: it returns a signed distance from the boundary, which orders cases but is not on the probability scale and does not pretend to be. Platt scaling supplies one after the fact by fitting a one-dimensional logistic regression of the outcome on those distances, using held-out data. That sets up the section’s least expected result: the SVM’s bolted-on probabilities come out better calibrated than logistic’s built-in ones, 0.017 against 0.054. Having a probability natively is not the same as having a good one, and an explicit calibration step fitted for that one job can beat a link function that was fitted for a different one.