Causal Forests

The Average Is the Wrong Object

Every design in this arc so far estimated an effect — one number, averaged over whoever happened to be in the study. That is often the wrong object. A training programme with an average effect of zero may help the young a great deal and harm the old, and a policy built on the average would be indefensible either way. The question here is $\tau(x)$: the effect as a function of who you are.

The setting is a simulated enrolment programme on 6,000 workers where the true $\tau(x)$ is known, which is the only honest way to grade an estimator of something never observed. Enrolment is confounded — older, higher-earning workers are likelier to enrol — and the true effect runs from −2.49 for older high earners to +5.96 for the young and low-paid.

The benchmarks set the bar honestly, and one of them clears it. The naive difference in means gives 4.757 against a true average of 2.068. Ordinary least squares with a treatment dummy recovers the average at 1.837 but returns a single number. And OLS with age and prior-earnings interactions reproduces the true $\tau(x)$ with a correlation of 1.000.

That last result deserves to be stated plainly rather than buried. A parametric model with the right interactions beats everything that follows, because the true effect really is linear in age and prior earnings. The forest has to discover that shape with no hint about which variables matter. If you already know the functional form, use it — the machinery on this page is for when you do not, and it earns its place by finding the shape rather than by beating a correctly specified model.

Honest Trees, and What a Forest Adds

The building block is an honest causal tree: split the sample to maximise heterogeneity in the treatment effect, then estimate the effect in each leaf using different observations from the ones that chose the split. That separation is what makes leaf estimates unbiased rather than a restatement of the noise the splitter chased.

Built from scratch, it splits on age and prior earnings — exactly the true modifiers — with no guidance about which variables matter, reaching a correlation of 0.744 with the truth on held-out data.

A forest averages many such trees, and the result exposes a distinction that matters more than the headline. Two forests, two very different failure modes:

Correlation measures whether the ranking is right; RMSE measures whether the magnitudes are.

The from-scratch forest gets the shape right and the level wrong. Its correlation with the truth is 0.930 — it ranks workers by benefit almost as well as the orthogonalized version, which strips the confounders out of both outcome and treatment before splitting — while its average is 3.350 against a true 2.061, and its RMSE is three times econml's. It has no orthogonalization, so it inherits the confounding wholesale. For targeting, which needs only the ranking, that is survivable. For reporting an effect size it is disqualifying, and nothing in the forest's own output announces which of the two you are doing.

The Intervals Are the Weak Part

Then the part that does not work as advertised. The nominal 90% confidence intervals cover the true $\tau(x)$ 85.4% of the time. That is not sampling noise, and it is not an artefact of one package: the R companion's grf — the reference implementation, written by the authors of the method — covers 76.4% on the same design. Two languages, two implementations, both short, and the reference one shorter.

The reason is structural. Honest splitting gives an unbiased leaf estimate and a variance for it; what it does not capture is the smoothing bias from approximating a continuous effect surface with piecewise-constant leaves. So the intervals are honest about the noise and silent about the approximation. The point estimates are good enough to target with; the intervals should be read as indicative rather than as calibrated 90% statements, and none of this page's ranking results depend on them.

Variable importance concentrates on prior earnings and age, and the partial-dependence curves track the true downward slopes — the forest recovers not just that effects differ but along which dimensions, which is the part a programme can act on. The R companion adds grf's test_calibration, a formal check whose differential-prediction coefficient near 1 with a tiny p-value is evidence the heterogeneity is real rather than fitted noise.

Turning τ(x) Into a Decision

The payoff is a decision. Treating the top 20% by estimated effect returns a realised average of 4.07 against 2.06 for treating everyone — twice the gain per enrollee. Enrolling only where $\hat\tau(x) > 0$ covers 90% of workers and avoids the roughly 9% the programme would actively harm, who are invisible in any average.

Where this sits

This is where the causal arc meets the machine-learning arc: honest splitting is the same out-of-sample discipline that purged cross-validation enforces, applied to a quantity that is never observed. The orthogonalization that fixes the from-scratch forest's level is double machine learning, and turning $\hat\tau(x)$ into an actual rule is policy learning. The estimand itself — an effect that varies by covariate — is what meta-learners attack from a different direction.

Notebooks

References

• Wager, S. & Athey, S. (2018). Estimation and Inference of Heterogeneous Treatment Effects using Random Forests. Journal of the American Statistical Association 113(523), 1228–1242. — causal forests, and their asymptotics
• Athey, S., Tibshirani, J. & Wager, S. (2019). Generalized random forests. The Annals of Statistics 47(2). — generalized random forests
• Nie, X. & Wager, S. (2020). Quasi-oracle estimation of heterogeneous treatment effects. Biometrika 108(2), 299–319. — the R-learner objective