A random forest (Breiman 2001) is an ensemble of decision trees (CART) in which each tree is grown on a bootstrap sample of the data and each node is split using a random subset of the candidate features; predictions are formed by majority vote (classification) or averaging (regression). The double randomization — bootstrap rows plus random features — decorrelates the trees so that averaging sharply reduces variance while keeping bias low.
Key Ideas
Two sources of randomness. (1) Bagging — each tree sees a bootstrap resample of the training rows; (2) random feature subspace — at each split only m of the p features (e.g. m=p) are considered. Both reduce correlation between trees.
Strength–correlation error bound. Breiman's generalization-error bound is ≤ρˉ(1−s2)/s2, with s the strength (expected margin) of individual trees and ρˉ their mean pairwise correlation. Accuracy improves by making trees strong and mutually uncorrelated — the random features trade a little individual strength for a large drop in correlation.
No overfitting in the number of trees. As the number of trees grows, the generalization error converges almost surely to a limit; more trees never hurt (unlike boosting), so the forest size is a computational, not a tuning, choice.
Out-of-bag (OOB) estimates. Each tree's held-out ~1/3 of rows give internal, near-unbiased estimates of test error, strength, and correlation — cross-validation "for free".
Permutation variable importance. Importance of a feature = the increase in OOB error when that feature's values are randomly permuted; a widely used but biased measure (see caveat).
Why It Matters
A default nonlinear predictor. Robust, low-tuning, handles mixed-type features, interactions, and high dimensions — the go-to off-the-shelf method and a strong baseline across domains.
Free diagnostics. OOB error and permutation importance make forests a practical tool for prediction and feature screening without a separate validation set.
Biased variable importance. Impurity- and permutation-based importance favour high-cardinality / continuous predictors and are distorted by correlated features; Strobl et al. (2007) diagnose this and propose conditional-inference forests with subsampling-without-replacement as a fix.
Interpretability. The ensemble sacrifices the single tree's flowchart interpretability; partial-dependence and SHAP-type tools only partly recover it.
Extrapolation. Like all trees, forests cannot predict outside the range of the training response, limiting use for genuinely out-of-sample regimes.