← All examples

Machine Learning: Trees and Ensembles

A decision tree is the rare model whose failure is as instructive as its success. It fits any shape you like — axis-aligned rectangles, arbitrarily fine — and that flexibility is exactly the problem: grown to purity it memorises the training set, and a single deep tree here pins a client's default probability only to within an SD of 0.26 across resamples. Nearly everything in this section is a response to that one fact. Pruning trades flexibility back for stability; bagging averages the variance away; the random subspace de-correlates the things being averaged; boosting abandons averaging altogether and fits trees to what the previous ones got wrong; and BART puts a prior on the whole ensemble so the answer comes with a posterior attached.

The examples are deliberately paired with a plain regression benchmark, because the interesting comparison is rarely the one the leaderboard reports. On this data a tuned tree beats a logistic regression by about 0.03 in AUC — area under the receiver-operating-characteristic curve, a ranking score defined below — — and the ordering flips when the tree is pruned by R's default rule instead of by cross-validation, which says more about tuning than about trees. A recurring theme is measuring the mechanism rather than asserting it: the de-correlation that random forests are supposed to buy does not show up in the correlation statistic everyone reaches for, but shows up clearly in the variance of the ensemble.

How a tree decides where to split. A tree grows by asking, at each node, which single yes/no question about one feature best separates the outcome — and “best” needs a definition. That is impurity: a measure of how mixed the classes are in a node, zero when every case in it shares a label. Two are in common use and they rarely disagree. Gini impurity is the chance that two cases drawn at random from the node have different labels; entropy is the information-theoretic version of the same idea. The split chosen is whichever gives the largest drop in impurity, weighted by how many cases go each way, and the tree keeps going until a stopping rule bites. That greedy, one-question-at-a-time construction is the source of both the method’s interpretability and its instability: nothing forces the second split to be sensible given the first, and a small change in the data can change the first split and everything below it.

Why averaging trees works, and why the forest goes further. A deep tree has low bias and high variance — it fits the training data almost exactly and would look quite different on a different sample. Averaging many such trees cancels that variance, which is bagging: fit each tree on a bootstrap resample and take the mean. But averaging only helps to the extent the trees make different mistakes, and bootstrapped trees on the same features tend to pick the same strong splitter at the top and end up highly correlated. The random forest’s addition is to restrict each split to a random subset of features, which forces the trees apart — decorrelation, and the actual reason a forest beats bagging. It also gets a free validation set: each tree omits about a third of the rows, so those out-of-bag cases can be predicted by the trees that never saw them.

Boosting’s three knobs. Boosting attacks the other half of the error. Instead of averaging independent trees it fits them in sequence, each one on what the ensemble still gets wrong, so it reduces bias where the forest reduces variance. Three settings govern it and they interact. The learning rate (or shrinkage) multiplies each tree’s contribution before adding it: small values need more trees but generalise better, and the pairing of a small rate with early stopping — halting when held-out performance stops improving — is what production practice actually does. Depth controls how much interaction each tree can express; boosting favours shallow trees precisely because it will fit thousands of them. And subsampling rows or columns per tree adds the forest’s randomisation on top. Turn the rate up and the ensemble climbs fast and then declines, which is overfitting made visible.

Reading the numbers. Two metrics recur. AUC — the area under the receiver-operating-characteristic curve — is the probability that a randomly chosen defaulter is scored above a randomly chosen non-defaulter; it measures ranking only and ignores whether the probabilities are believable. ECE, the expected calibration error, measures exactly what AUC ignores: bin the predictions, compare each bin’s claim with the frequency actually observed in it, and average the gaps. The pair matters here because the section’s most counterintuitive result is a calibration one — gradient boosting, which textbooks warn is badly calibrated, comes out the best of three on this data at 0.012, while the logistic regression that estimates probabilities by maximum likelihood is the worst at 0.059. The boosting pathology is a symptom of overfitting, and a regularised fit simply does not have it.