Double Machine Learning

Why the Obvious Thing Fails

The obvious way to use machine learning for a causal estimate is to fit a flexible model of the confounders, subtract it, and regress what is left on treatment. It does not work, and the reason is instructive rather than incidental.

In the partially linear model $Y = \theta T + g(X) + \varepsilon$, subtracting a fitted $\hat g(X)$ looks harmless. But $\hat g$ is regularized, and regularization means it absorbs some of the signal that belongs to $T$ — because $T$ itself depends on $X$. On a simulation with a true $\theta = 1.0$, the naive plug-in returns 0.750. A 25% bias, from a procedure that looks like careful modelling.

Neyman orthogonality is the fix, and it is one line: residualize $T$ as well. Regress $T$ on $X$, keep the residual, and use only the variation in treatment that the confounders cannot explain. That estimate comes back at 0.986. The reason it works is that the moment condition becomes insensitive to first-order errors in the nuisance functions — the parts of the model nobody cares about, fitted only so that they can be removed — a slightly wrong $\hat g$ or $\hat m$ no longer biases $\theta$.

Cross-fitting completes it. Fit the nuisance models on one fold and residualize on another, so no observation contributes to both the model that predicts it and the estimate that uses it. Otherwise the estimate inherits the overfitting. The full procedure gives 1.003 with a standard error of 0.023.

The Coverage Experiment, and What It Took to See It

Then the coverage experiment, which is where this page earns its length. Over 200 replications the DML interval covers the truth 89.0% of the time, give or take 4.3 points, against the naive plug-in's 0%. The comparison is not close, and that is the headline: orthogonality plus cross-fitting take an interval that essentially never contains the truth and make it mostly right.

But 89% is not 95%, and the gap is worth reading rather than rounding away. Two things cause it, and the diagnostics separate them: a residual bias of −0.0199 — about 0.75 standard errors — and reported standard errors around 5% smaller than the actual spread of the estimates. Neither is large alone; together they cost six points of coverage. Orthogonality removes the first-order regularization bias, which is exactly what the 0.750 → 0.986 improvement demonstrates. What remains is a second-order term that has not vanished at $n = 1500$ with these learners.

An earlier version of this experiment got that wrong, and how it got it wrong is the transferable part. It ran 40 replications, observed 40 out of 40 covering, and concluded that coverage was essentially perfect. Those forty were genuine — the first hundred replications run at about 94%. But a run of forty cannot measure this quantity at all:

Even if coverage were exactly the nominal 95%, a clean sweep of forty has probability 0.13 — about one in eight. The experiment could not have distinguished 95% from 100% whatever it returned, so the observed sweep carried almost no information, and reading it as confirmation was reading noise. Under the coverage the longer run actually finds, seeing 40/40 is roughly a one-in-a-hundred event, which is what a lucky stretch looks like.

Which is the arc's recurring lesson arriving from one more direction: a diagnostic has to be powered to detect the thing it is being used to rule out, and “it passed” means nothing until you know what it could have caught. The pre-trend test, the independence test on the Sachs network, and this coverage check all fail in the same way.

401(k) Eligibility and Household Wealth

The application is 401(k) eligibility and household wealth on 9,915 households from the 1991 SIPP — the canonical DML dataset, because eligibility is plausibly unconfounded given income, age and saving proxies while the relationship between those controls and wealth is hopelessly nonlinear.

The naive eligible-minus-ineligible gap is USD 19,559. Controlling flexibly, DML attributes USD 8,841 to eligibility itself, with a 95% interval of [5,937, 11,745] — and econml's LinearDML independently gives USD 9,080. Over half the raw gap was confounding, and the two implementations agree on what remains.

Where this sits

The residualize-both-sides trick is what separates the orthogonalized causal forest from the from-scratch one that got the level wrong, and applied pointwise it is the R-learner. Cross-fitting is the same discipline as purged cross-validation in the machine-learning arc — keeping the data that fits a model away from the data that evaluates it. And the coverage shortfall here is the counterpart of the interval undercoverage the causal forests page found in two independent implementations.

Notebooks

Downloads

References

• Chernozhukov, V. et al. (2018). Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal 21(1), C1–C68. — double/debiased machine learning
• Robinson, P. M. (1988). Root-N-Consistent Semiparametric Regression. Econometrica 56(4), 931. — the partially linear model it generalises
• Wager, S. & Athey, S. (2018). Estimation and Inference of Heterogeneous Treatment Effects using Random Forests. Journal of the American Statistical Association 113(523), 1228–1242. — forests as the nuisance learner