← All examples

Machine Learning: Model Evaluation & Interpretability

Every earlier subsection built models. This one asks the questions that decide whether a model is usable, and they are all versions of the same question: does this number mean what it appears to mean?

A selection criterion that recovers the true model on one dataset may do so 70% of the time. A feature-importance ranking can be produced by an algorithm's internal accounting rather than by the data. A classifier with excellent discrimination can emit probabilities that are badly wrong as probabilities. And a point prediction says nothing at all about how far off it might be.

The methods here are the standard answers — and because they are diagnostics, each one is itself put through the check it recommends. A criterion is asked to repeat its verdict on fresh draws; an attribution is compared against a permutation baseline and against what the model actually uses; a probability is scored against observed frequency; and an interval is checked for whether its stated coverage is delivered.

Choosing complexity, and the four criteria that do it. A more flexible model always fits the data it was trained on better, so training error cannot decide how complex a model should be — past some point the extra flexibility is spent reproducing noise that will not recur, which is overfitting. Every honest criterion therefore estimates performance on data the model has not seen. Cross-validation does it directly, refitting on subsets and scoring the held-out part. AIC (the Akaike information criterion) and BIC (the Bayesian information criterion) do it from a single fit, taking the likelihood and subtracting a penalty for the number of parameters — the difference between them is the size of that penalty, and it is not a detail: AIC charges a fixed amount per parameter, BIC charges an amount that grows with the sample size. PSIS-LOO is leave-one-out cross-validation computed from a Bayesian posterior without refitting, using Pareto-smoothed importance sampling to reweight the existing draws. It reports an ELPD, the expected log pointwise predictive density — the average log probability the model assigns to a new observation, where higher is better — and comes with a self-diagnostic: the Pareto-k statistic flags when that reweighting is unreliable, conventionally at 0.7, so an ELPD is only worth comparing once the k values are below it.

Two questions that sound like one. The criteria above split into two camps, and the split explains almost everything that happens on the first page. CV, AIC and LOO all estimate how well the model will predict. BIC estimates which model generated the data. These come apart whenever a model slightly too large predicts about as well as the right one, which is usually. A criterion is consistent if the probability it picks the true model goes to 1 as the sample grows, and efficient if it picks the model that predicts best. BIC is consistent, AIC is not — more data does not rescue it, because its penalty per parameter never grows — and no criterion is both. That is a property of the question each one asks, not a bug in either.

Attribution: three ways to ask which features mattered. Impurity importance, also called MDI (mean decrease in impurity), adds up how much each feature improved the splitting criterion every time a tree used it. It is free, because the model recorded it while fitting — and it is computed entirely from the training data, which is the source of its trouble. Permutation importance asks the question empirically instead: shuffle one column, re-score on held-out data, and see how much performance drops. A column carrying nothing should cost nothing, and can score slightly negative. SHAP (SHapley Additive exPlanations) comes from cooperative game theory: a Shapley value divides a payout among players by averaging each player's marginal contribution over every possible order of joining, and here the players are features and the payout is the prediction. That gives it a property the others lack — contributions for a single prediction that sum exactly to it — at much greater cost. One distinction runs through all three: SHAP is faithful to the model, describing what the model does; permutation is faithful to the data, describing what actually helps. When a model has learned something spurious those are different answers, and both are correct.

Calibration: ranking well and being right are different achievements. Discrimination is whether a model orders cases correctly, and is what AUC — the area under the receiver-operating-characteristic curve — measures: the probability a randomly chosen positive case is ranked above a randomly chosen negative one. Calibration is whether the numbers mean what they say: among cases given a 30% chance, does about 30% actually happen? A model can be excellent at the first and badly wrong at the second, because nothing in a ranking metric inspects the values themselves. The usual summary is ECE, the expected calibration error: sort predictions into bins, compare each bin's average claim with the frequency observed in it, and average the gaps weighted by bin size. Because it averages absolute gaps it can never be zero on finite data, which is why a simulated floor matters. The Brier score is the mean squared error of the probabilities, mixing both properties into one number, and a reliability diagram is the picture: claimed probability on one axis, observed frequency on the other, with the diagonal as perfection. Two repairs recur. Platt scaling fits a sigmoid to the model's scores — two parameters, so it can stretch or shift confidence but not change the shape of the distortion. Isotonic regression fits any non-decreasing function, so it can absorb a distortion that reverses direction, at the price of far more flexibility and a tendency to overfit small samples.

Conformal prediction, and the assumption nobody states. A point prediction says nothing about how wrong it might be. Conformal prediction wraps any model in an interval — or, for classification, a prediction set of labels — whose coverage is guaranteed in finite samples. It works by choosing a nonconformity score, a measure of how unusual an observation looks to the fitted model (for regression, typically just the absolute error), computing it on a held-out calibration set, and taking the appropriate quantile as the threshold. It is distribution-free: nothing is assumed about the shape of the errors, and nothing is assumed about the model being any good — a terrible model simply produces wide intervals. But distribution-free is not assumption-free. The guarantee requires exchangeability: that the calibration data and the new case are drawn in a way where the order does not matter, so a future point is no more surprising than a calibration point was. Independent draws from one population satisfy it; a time series with drift, or a deployment on a population that has shifted since calibration, does not — and when exchangeability fails the guarantee fails with it, silently. The second limit is what the guarantee covers. Marginal coverage is an average over all cases; conditional coverage would hold within every subgroup. Only the first is promised, and the gap between them is measurable. CQR (conformalized quantile regression) narrows it by conformalising a quantile model, so interval width varies with the case, and APS (adaptive prediction sets) is the classification analogue, accumulating predicted probabilities until they cross the calibrated threshold.