Interpretability — Impurity vs Permutation vs SHAP

Three Measures, Three Answers

A model that predicts well is half of what a risk committee needs; the other half is why. But "feature importance" is not one quantity. A tree's built-in impurity importance, or MDI (mean decrease in impurity), adds up how much each feature improved the split criterion wherever a tree used it. It is free, because the model recorded it while fitting, and it is biased. Permutation importance shuffles a column and measures the drop in held-out performance. SHAP (SHapley Additive exPlanations) borrows the Shapley value from cooperative game theory — a way of dividing a payout among players by averaging each one's marginal contribution across every possible order of joining, with features as the players and the prediction as the payout. It assigns each feature a signed contribution to each individual prediction, exactly and quickly for trees. The three disagree, and the disagreement is systematic rather than random.

The test is to inject columns of pure noise into the Taiwan credit-default data — one high-cardinality, one binary — and see where each measure puts them. A trustworthy measure should place both at the bottom.

Read as ranks, the three look similar and the story is muddled: permutation puts the binary noise at rank 15 of 25 while impurity puts it dead last, which would suggest impurity is the better measure. Read as magnitudes, the ordering is unambiguous. Impurity awards the high-cardinality noise 10% of the strongest real feature's importance and SHAP 5%, while permutation returns a negative value: shuffling the column slightly improves held-out AUC, which is the correct answer for a column containing nothing. (AUC is the area under the receiver-operating-characteristic curve, and measures only whether cases are ranked correctly.)

The reason ranks mislead here is worth stating on its own. With enough repeats to attach a standard error, 18 of the 25 features have permutation importance within two standard errors of zero. Ordering that group is close to arbitrary, and a rank inside it can move ten places between random seeds while the underlying number does not move at all. A table of ranks conveys a precision the numbers do not have.

Why They Disagree

The mechanism behind impurity's bias is testable rather than merely citable. Five columns of pure noise, identical in every respect except how many distinct values they take, produce steadily more impurity importance as that number rises — a feature with more split points has more chances to reduce impurity by luck. Permutation stays flat at zero throughout.

SHAP's residual mass on the noise is not a bug, and this is the most useful distinction on the page. SHAP is faithful to the model. The forest genuinely did split on those columns, so a correct attribution has to report that it did. SHAP is not answering "is this feature useful?" — it is answering "what does this model do with it," and those come apart precisely when the model is imperfect. Permutation is the only one of the three scored on held-out data, which is why it is the only one that answers the generalisation question.

A second caveat is usually stated and rarely shown. Permutation asks what happens when one column is scrambled — but if a second column carries nearly the same information, the model leans on the substitute and the measured drop is small. The credit data has six monthly bill-amount columns correlating at 0.89. Permuted one at a time they cost 0.0090 AUC in total; permuted as a block they cost 0.0137, about 1.5× as much. A low permutation importance means "this feature adds nothing given the others," which is not the same as "this feature does not matter."

Explaining One Prediction

What no ranking can provide is the per-decision explanation. SHAP's beeswarm gives the global picture — recent repayment status $\texttt{PAY\_1}$ dominates, and arrears push risk up — while a local attribution decomposes one client's 83% predicted default probability into its specific drivers, which is the form a credit decision or a regulator actually requires. Partial dependence and ICE (individual conditional expectation) plots — the average effect of a feature across everyone, and the same curve drawn separately for each individual — then supply effect shapes: risk falling with the credit limit, jumping once a client is months behind, with the spread of the individual curves flagging interactions the average hides.

Where this sits

The impurity-bias caveat is raised in Random Forests — Averaging Away the Variance and made explicit here; the SHAP machinery is used in XGBoost, LightGBM & CatBoost, and the same credit data is modelled in Gradient Boosting — Correcting Errors in Sequence and Decision Trees — CART from Scratch. The habit of separating "what the model does" from "what is true" recurs throughout this subsection — it is the same distinction that makes an in-sample anomaly score invert in Autoencoders, and the same reason Model Selection grades criteria on fresh draws rather than one.

Notebook

Downloads

References

• Strobl, C., Boulesteix, A.-L., Zeileis, A. & Hothorn, T. (2007). Bias in random forest variable importance measures: illustrations, sources and a solution. BMC Bioinformatics 8, 25. — the cardinality bias reproduced here
• Breiman, L. (2001). Random forests. Machine Learning 45(1), 5–32. — permutation importance
• Lundberg, S. M. & Lee, S.-I. (2017). A unified approach to interpreting model predictions. NeurIPS 30. — SHAP
• Lundberg, S. M. et al. (2020). From local explanations to global understanding with explainable AI for trees. Nature Machine Intelligence 2, 56–67. — TreeSHAP
• Hooker, G., Mentch, L. & Zhou, S. (2021). Unrestricted permutation forces extrapolation: variable importance requires at least one more model. Statistics and Computing 31, 82. — why correlated features break permutation importance
• Goldstein, A., Kapelner, A., Bleich, J. & Pitkin, E. (2015). Peeking inside the black box: visualizing statistical learning with plots of individual conditional expectation. JCGS 24(1), 44–65. — ICE
• Yeh, I.-C. & Lien, C.-H. (2009). The comparisons of data mining techniques for the predictive accuracy of probability of default of credit card clients. Expert Systems with Applications 36(2), 2473–2480. — the data