← All examples

Causal Inference: Frontier — ML & Dynamics

The arc so far has been about identification — what a design entitles you to claim. This group changes the subject to estimation: identification is granted, and the question is what flexible machine learning can add once it is.

Two things, it turns out. The first is the conditional effect τ(x) rather than an average, which is what a programme actually needs — an average of zero can hide a large benefit for the young and real harm to the old, and no amount of care about identification surfaces that. The second is robustness to nuisance — the confounder-to-outcome relationship that has to be modelled but is not the thing being estimated: when the relationship between confounders and outcome is genuinely nonlinear, a parametric adjustment is a guess, and orthogonalization lets a flexible model absorb it without the flexibility contaminating the estimate.

The last two examples change the subject again, to dynamics. Time-to-event outcomes are censored, so there is nothing to subtract; and when treatment repeats, confounders respond to it, and the instruction to adjust for confounders becomes self-contradictory. Both need machinery that has no analogue earlier in the arc.

What unites the six is a discipline the flexibility makes necessary. Every one of them is graded against a known truth, because an estimator of something never observed cannot be validated any other way — and doing that consistently is what turned up the results below.

What flexibility buys, and what it costs

Every value is committed notebook output. Five of the six run on simulations with known individual effects — the only way to grade an estimator of a quantity that is never observed — and the sixth on the 1991 SIPP, where the comparison is between implementations rather than against a truth.

A · Ranking and magnitude are separate properties, and a flexible estimator can have one without the other Both forests estimate the same τ(x) on the same 6,000 workers. Correlation says whether the ORDER is right; RMSE says whether the NUMBERS are. correlation with true τ RMSE econml CausalForestDML orthogonalized 0.955 0.460 forest, from scratch no orthogonalization 0.930 1.421 honest tree, from scratch a single tree 0.744 1.434 The middle row ranks people almost as well as the top row and mis-states their effects by three times as much. For targeting, which needs only the order, that is survivable. For reporting an effect size it is disqualifying — and nothing in the output says which you are doing. B · The intervals are where flexibility is still being paid for Every row is a confidence interval that does not contain the truth as often as it claims. The last one is the reason the others were checked at all. interval nominal actual why CATE intervals — econml 90% 85.4% smoothing bias not in the variance CATE intervals — grf, the reference implementation 90% 76.4% same cause, larger DML interval, 200 replications 95% 89.0% SEs 5% too narrow, bias 0.75 SE And the check that missed it: an earlier DML experiment ran 40 replications, saw 40 of 40 cover, and called the inference valid. At exactly nominal 95% coverage, a clean sweep of forty has probability 0.13. The experiment could not have distinguished 95% from 100% whatever it returned, so the sweep carried almost no information. A diagnostic has to be powered to detect the thing it is being used to rule out. C · And two settings where adjusting for the confounder is not merely insufficient but wrong Everything above assumes the usual apparatus applies. When the outcome is censored, or when treatment repeats, it does not. A hazard ratio is two kinds of wrong at once Cox reports a single 0.601. Adjusting for the confounder gives 0.352, so confounding explains 0.249 of the gap — and adjustment fixes that part. The true hazard ratio runs 0.139 to 0.750 over follow-up, so the corrected constant is still a summary of a curve that changed shape. RMST instead: 1.69 by weighting, 1.57 by outcome modelling, against a true 1.62. Adjusting for the confounder makes it worse When treatment repeats, a marker confounds it AND is changed by it. The naive estimate is confounded at −0.232; the textbook adjustment is worse at −0.065, because conditioning blocks the pathway the treatment works through. Reweighting rather than conditioning recovers −0.342 — and no specification of that regression would have worked.

The first block is the group's central distinction. A causal forest without orthogonalization correlates 0.930 with the truth and has an RMSE three times the orthogonalized version's. It ranks people correctly and states their effects wrongly. For targeting — which needs only the order — that is survivable; for reporting an effect size it is disqualifying, and nothing in the output tells you which of the two you are doing. Ranking and magnitude are separate properties, and flexible estimators routinely have one without the other.

The interval rows are the group's least comfortable result. CATE confidence intervals undercover in both implementations, and grf — written by the authors of the method — undercovers more than econml. DML's intervals miss for a different and more diagnosable reason: a residual bias of about 0.74 standard errors plus standard errors roughly 5% too narrow. Orthogonality removes the first-order regularization bias, which is exactly what taking 0.750 to 1.003 demonstrates, and a second-order term survives at realistic sample sizes.

The DML coverage row also carries a methodological finding about the check itself. An earlier version of that experiment ran 40 replications, observed 40 out of 40, and concluded coverage was perfect. Even at exactly nominal coverage, a clean sweep of forty has probability 0.13 — so the experiment could not have distinguished 95% from 100% whatever it returned. The sweep was a lucky stretch, and reading it as confirmation was reading noise. It is the same failure as an underpowered pre-trend test or an independence test that finds nothing because it can find nothing.

The meta-learner rows are a warning about benchmark results generally. The X-learner wins the simulation it was designed for and drops to third on IHDP, where poor overlap punishes anything dividing by a propensity and the simplest learner is left standing. The R companion flips the ranking too — and disagrees with Python about which learner leads in each setting. The ordering is not a property of the learners; it is a property of the learners crossed with the overlap in your data.

And the last two rows are where adjustment stops working at all. A Cox hazard ratio is confounded and a time-average; adjusting fixes the first and leaves a single constant standing in for something that runs from 0.14 to 0.75. Then in the longitudinal case, conditioning on a confounder that past treatment caused takes the estimate from −0.232 to −0.065 — the textbook instruction, applied literally, erases most of a real effect. Reweighting recovers it. There is no specification of that regression that works, because the variable's two roles cannot be separated by including or excluding it.

How the six examples relate

Four on heterogeneity and nuisance, running from an estimate to a decision; two on dynamics, where the earlier machinery has no analogue.

Estimating τ(x)

Who benefits, not how much on average?

1 · Causal forestshonest splitting, and a forest that ranks well while getting the level wrong 2 · Meta-learnersS, T, X and R — and a ranking that inverts between datasets and implementations
Making it trustworthy

How does flexible ML avoid poisoning the estimate?

3 · Double machine learningresidualize both sides, cross-fit — and the coverage that still falls short
Making it a decision

Who should actually be treated, given a cost?

4 · Policy learningtreat-all destroys value; a depth-2 rule matches the oracle
When time is involved

Censored outcomes, and confounders that respond to treatment.

5 · Causal survival analysisRMST instead of a hazard ratio that is two kinds of wrong 6 · Marginal structural modelswhere adjusting for the confounder is worse than not adjusting

The thread is that flexibility moves the failure rather than removing it. Parametric adjustment fails visibly — a wrong functional form, a coefficient that moves when you add a term. These methods fail quietly: a forest that ranks correctly and mis-states magnitudes, an interval that is 11% too narrow, a benchmark ranking that does not transfer, a coverage check too small to detect its own failure. Every one of those was found here by holding the estimator against a known truth, which is available in a simulation and never available in the application you care about. That asymmetry is the honest summary of the group: the tools are real and worth having, and the confidence they invite is not something the data supplies.

Causal Forests — Heterogeneous Treatment Effects

Estimating τ(x) on 6,000 workers where the truth is known. Honest causal trees built from scratch recover the true effect modifiers with no hint about which variables matter, and the forest reaches a correlation of 0.955. Then the distinction that matters: the from-scratch forest, lacking orthogonalization, correlates 0.930 with an RMSE of 1.421 against econml's 0.460 — it ranks correctly and mis-states magnitudes, which is survivable for targeting and disqualifying for reporting. And the intervals undercover in both languages: 85.4% in econml and 76.4% in grf, the reference implementation, against a nominal 90%. Targeting the top 20% still doubles the gain per enrollee.

View example →

Meta-Learners — S, T, X and R

The same base learner wired four ways gives four answers. On a confounded simulation with a 30% treated arm the X-learner wins at PEHE 0.118, which is exactly the imbalance it was designed for. On IHDP — 747 units, 25 real covariates, 19% treated with genuinely poor overlap — the ranking inverts: the simple S-learner wins at 0.447, the X-learner drops to third, and the R-learner does not merely lose but fails at 2.721, because its pseudo-outcome divides by a propensity that is near 0.17. The R companion flips too, and disagrees with Python about which learner leads in each setting. There is no universal winner, and the ranking is a property of the learners crossed with your overlap.

View example →

Double Machine Learning — Orthogonality and Cross-Fitting

Subtracting a flexible ĝ(X) biases a true 1.0 down to 0.750, because the regularized model absorbs signal belonging to the treatment. Residualizing the treatment as well — Neyman orthogonality — restores 0.986, and cross-fitting completes it at 1.003. Then the coverage experiment, which is where the page earns its length: 86% against a nominal 95%, from a residual bias of about 0.7 standard errors plus standard errors 11% too narrow. An earlier version of that check ran 40 replications, saw 40 of 40, and called it perfect — but even at exactly 95% coverage a clean sweep of forty has probability 0.13, so it could never have detected the shortfall. On the 401(k) data the naive USD 19,559 gap becomes USD 8,841.

View example →

Policy Learning — From τ(x) to a Rule

An estimate is not a decision. With a treatment costing 0.4 against a true average effect of 0.25, treating everyone destroys value — a gain of −0.153, so an evaluation reporting a positive average and recommending rollout would be recommending a loss. The oracle that treats where τ(x) exceeds cost returns +0.238. A depth-2 policy tree fit on doubly-robust rewards recovers that same +0.238 with a rule of x₁ > 0.579 against the value-maximising 0.560 — conservative by 0.019, costing nothing measurable, and unlike the forest it approximates, it is a sentence a programme can put in an eligibility document.

View example →

Causal Survival Analysis — RMST over Hazard Ratios

Censoring means there is no outcome to subtract. The field's default answer, a Cox hazard ratio, comes back at 0.601 — and has two things wrong with it. It is unadjusted: including the confounder gives 0.352, so confounding accounts for 0.249 of the gap and adjustment fixes that part. What adjustment cannot fix is that the true hazard ratio runs from 0.139 to 0.750 over follow-up, so even the corrected constant is a summary of a curve that changed shape. RMST is the estimand that survives: IPTW gives 1.69 years and g-computation 1.57 against a true 1.62 — two routes, one modelling treatment and one the outcome, agreeing.

View example →

Marginal Structural Models — Treatment–Confounder Feedback

When treatment repeats, the confounder stops holding still: a marker determines treatment and is changed by past treatment, so it confounds and mediates at once. Both standard analyses fail in opposite directions. The naive regression is confounded at −0.232; the adjusted one — the thing every methods course teaches — is worse at −0.065, because conditioning blocks the pathway through which treatment does its work. Reweighting rather than conditioning recovers −0.342. And the residual is stated honestly: the implied survival gap is +0.31 against a true +0.39, or 79% — a shortfall that is not confounding but the structural model's own specification error.

View example →