XGBoost, LightGBM & CatBoost — Production Gradient Boosting

What the Engineering Buys

The three libraries that actually get used. Gradient boosting from scratch establishes what they compute; XGBoost, LightGBM and CatBoost establish what it takes to run it at scale — histogram binning, sparsity-aware splits, leaf-wise growth, ordered target statistics for categoricals, and early stopping as a first-class feature rather than a manual sweep. The question this example actually answers is what that engineering buys, measured rather than assumed.

Head to head

Fit times are one run on one machine and move by tens of percent between executions; the accuracy columns are deterministic and do not. Read the time column for its order of magnitude, not its digits.

Not accuracy, mostly. On classification the three are near-identical — AUC 0.773 to 0.776, a spread of 0.003, which is inside the noise of a single train/test split. On regression they are not quite: RMSE 0.494 / 0.494 / 0.521, with CatBoost 6% behind the other two at default settings. What genuinely separates them is fit time, which differs by 8× between the fastest and slowest on the same task.

Early stopping

Early stopping is the production essential, and it is the direct answer to the tuning burden the from-scratch example ran into: training AUC climbs toward 1.0 while validation AUC peaks near round 98 and then flattens, so the boosting stops itself there. Held-out AUC at that point is 0.7724, with no manual tree-count sweep. R's xgboost lands on the same answer independently — 97 rounds, AUC 0.7756.

Explaining the model

Interpretability gets the modern treatment. Where the forest example showed that default impurity importance is biased toward continuous features, SHAP is per-prediction, signed and additively consistent: it confirms the whole family's story — recent repayment status dominates — while also showing the direction, with arrears pushing predicted log-odds of default sharply up. That is the form a risk committee or a regulator can actually interrogate.

Reading past the top feature

The beeswarm ranks ten features and it is worth reading past the first. Scoring each by mean |SHAP| and pairing it with the correlation between the feature's value and its own SHAP value gives the direction as well as the magnitude — and the second tier is entirely made of payment amounts, all pushing the other way. PAY_AMT2, the sum actually paid two months back, is the clearest: high values lower predicted risk at a correlation of −0.64.

Taken back to the raw data that direction holds and sharpens. Default rates fall monotonically across quintiles of PAY_AMT2, from 32.7% in the lowest to 13.2% in the highest against a base rate of 22.1%. But the signal sits at zero: 18% of clients paid nothing that month and they default at 33.3% against 19.7% for everyone who paid something, so the negative SHAP mass is largely the mirror of a large positive push for the non-payers. Nor is it standing in for account size — the correlation with the bill is 0.10 and with the credit limit 0.18, so the amount paid carries information of its own. And because SHAP is conditional on the rest of the model, this is what PAY_AMT2 adds on top of the delinquency flags, which is why it sits in the second tier rather than the first.

Calibration, and a ceiling in the data

Calibration matters more than ranking for anything priced off the prediction, and the boosters hug the 45-degree line: a predicted 30% really does default about 30% of the time, with the logistic slightly worse in the high-risk deciles. On the California regression the decile means track the diagonal until they flatten at the top — which is not model failure but the dataset's known $500k price cap, a ceiling no model can predict past.

The scoreboard

Across everything in the section, on the same split, the ordering is clear and the margins are not. The single tree and the linear baseline trail; every ensemble sits at the top within 0.003 AUC and 0.029 RMSE of the others. On clean tabular data of this size the choice between them is about speed, tooling and uncertainty — not a headline accuracy race.

Notebooks

Downloads

References

• Chen, T. & Guestrin, C. (2016). XGBoost: a scalable tree boosting system. KDD '16, 785–794. — sparsity-aware splitting and the regularised objective
• Ke, G. et al. (2017). LightGBM: a highly efficient gradient boosting decision tree. NeurIPS 30. — histogram binning and leaf-wise growth
• Prokhorenkova, L. et al. (2018). CatBoost: unbiased boosting with categorical features. NeurIPS 31. — ordered boosting and target statistics
• Lundberg, S. M. & Lee, S.-I. (2017). A unified approach to interpreting model predictions. NeurIPS 30. — SHAP values
• Niculescu-Mizil, A. & Caruana, R. (2005). Predicting good probabilities with supervised learning. ICML '05. — why calibration and ranking are different questions