← All examples

Causal Inference: Selection on Observables

The previous group had randomization doing the identifying work, which is why it came first: it is the one setting where a causal effect is recovered with almost no assumptions. Everything here is an attempt to earn that back without the experiment.

The assumption doing the earning is unconfoundedness — conditional on the covariates, treatment is as good as random. It is the workhorse of applied policy evaluation and it has a permanent defect: it asserts that no unobserved confounder remains, and no amount of data can check that claim. What can be checked is balance on the covariates in hand, which is a weaker thing than it looks.

So the group is built around that gap. It starts on the dataset designed to expose it, where the true answer is known from a randomized trial and the observational methods can be graded against it. It then asks what a failed assumption would have to look like to matter, and finally what changes when balance is targeted directly rather than through a model of who gets treated.

What adjustment recovers, what the diagnostic misses, and what neither can reach

Every value is committed notebook output. The group is unusual in having an answer key: the true effect is known from a randomized trial, so for once the observational estimators can be graded rather than merely compared.

A · One dataset, seven answers — and only the design-based benchmark is known to be right LaLonde NSW/CPS, estimated effect of job training on 1978 earnings (USD). Dashed line = the randomized experiment’s USD 1,794. naive comparison 185 NSW trainees vs 429 CPS controls, no adjustment −$635 Mahalanobis matching matched on the full covariate vector $719 -60% TMLE doubly robust, efficient, with a targeting step $1,199 -33% AIPW doubly robust: propensity model + outcome model $1,225 -32% entropy balancing exact covariate-mean balance by construction $1,273 -29% IPW reweight controls by e(X)/(1-e(X)) $1,316 -27% NN-propensity matching nearest neighbour on the propensity score $1,792 essentially exact randomized benchmark $1,794 B · The group’s signature finding: balance improves down the column, accuracy does not follow Standardized mean differences after adjustment, against the same estimate’s distance from the benchmark. worst |SMD| outside ±0.1 estimate vs benchmark NN-propensity matching leaves age at 0.228 and married at 0.152 outside the band 0.228 2 of 8 $1,792 essentially exact Mahalanobis matching balances age and the dummies almost perfectly — lagged earnings less well 0.138 1 of 8 $719 −60% entropy balancing exact by construction — and 77% of the effective sample is gone 5×10⁻⁶ 0 of 8 $1,273 −29% C · None of it touches the assumption. How much hidden confounding would each result tolerate? Γ = how far a hidden confounder could skew treatment odds within a matched pair. Γ = 1 is a randomized experiment. the LaLonde matched estimate landed on the truth at $1,792; significant at Γ=1 with p = 0.0033 Γ* = 1.21 a 1.21× odds shift erases it large effect, 30 pairs d = 1.07, t-test p = 2.4×10⁻⁶ Γ* = 4.2 robust small effect, 480 pairs d = 0.23, t-test p = 9.3×10⁻⁷ — the MORE significant of the two Γ* = 1.4 fragile smoking-scale association risk ratio 3.9, VanderWeele’s benchmark E-value 7.26 no known confounder is that strong weak association risk ratio 1.3 E-value 1.92 a modest confounder would do it

A is the case for the toolkit, and it is a strong one. The naive comparison does not merely mis-measure the programme — it gets the sign wrong, reporting that job training cost trainees USD 635 when the experiment says it earned them USD 1,794. Adjusting for eight observed covariates moves every estimator onto the correct side of zero, and nearest-neighbour propensity matching lands on the benchmark almost exactly. That is a real accomplishment, and the reason the field takes unconfoundedness seriously.

It is also where the caveat lives. The seven answers span USD 719 to USD 1,792, a factor of two and a half, and every adjusted estimate falls at or below the truth rather than scattering around it. A practitioner without the answer key would have several defensible numbers and no principled way to choose between them — which is precisely the complaint LaLonde made in 1986, restated with better tools.

B is the finding that reorganises the group. Read the balance column downward and it improves monotonically: 0.228, then 0.138, then exact to six decimal places. Read the accuracy column and it does not follow at all. Mahalanobis matching produces the tidier balance table and misses by 60%; entropy balancing achieves perfect mean balance and misses by 29%, having spent 77% of its effective sample to get there. The scheme with the worst balance table is the one that lands on the truth.

The explanation is that which covariates a scheme balances matters more than how well it balances them on average. Lagged earnings are what predict 1978 earnings, and nearest-neighbour matching gets re75 to 0.008 where Mahalanobis leaves it at 0.138, buying its tidy aggregate on age and the demographic dummies instead. So the standard advice — check balance, do not just report a number — is necessary and not sufficient: applied to the aggregate, as it usually is, it selects the wrong estimator here. Judging balance properly requires knowing which covariates drive the outcome, which is a claim about the outcome model rather than about the matching, and is exactly the knowledge unconfoundedness already assumes. The diagnostic leans on the assumption it was brought in to police.

C is the limit on all of it. Everything in A and B assumes no unobserved confounder, and nothing in A or B can check that. Sensitivity analysis does not test the assumption — that is impossible — but it prices it. The matched estimate that landed on the truth is genuinely significant (p = 0.0033 with no hidden bias assumed) and would be erased by a confounder shifting within-pair treatment odds by a factor of 1.21. Landing on the truth and being robust are separate properties, and this estimate has only the first.

The middle two rows of C carry the transferable lesson. Both studies are overwhelmingly significant and the small effect is the more significant of the pair — p = 9.3×10−7 against 2.4×10−6 — yet it breaks at Γ = 1.4 while the large effect holds to 4.2. Significance can always be bought with sample size; robustness to hidden bias cannot. A p-value and a sensitivity number answer different questions, and reporting only the first is how a fragile result comes to look settled.

How the three examples relate

A single dataset runs through all three, which is deliberate: holding the data fixed is what makes the methods comparable, and LaLonde is the dataset built for exactly that purpose.

Make the assumption work

Can observed covariates alone recover a randomized benchmark?

1 · Potential outcomes & matchingpropensity scores, matching, IPW and AIPW — and the balance table that picks the wrong one 3 · Modern balancingtarget balance directly — exact, and 77% of the effective sample gone
Price the assumption

If it fails, how badly would it have to fail to matter?

2 · Sensitivity analysisRosenbaum bounds and the E-value — the estimate that hit the truth is fragile at Γ = 1.21
Where it goes next

What to do when no covariate set is enough.

Instrumental variables, and the rest of the arcfind variation in treatment that is as good as randomly assigned, instead of assuming it away

One thread runs the length of the group and is worth following deliberately. Every diagnostic here measures something other than what you want to know. Balance measures the covariates you happened to collect; a p-value measures incompatibility with a null, not robustness; an exact balance table says nothing about the variance it cost. The three examples are, in order, a case where the diagnostic picks the wrong estimator, a case where it is silent on fragility, and a case where it hides its own price. That is not an argument against measuring — it is an argument for knowing what each measurement leaves out, which on this dataset can be shown rather than asserted, because the answer is known.

Potential Outcomes & Matching — the LaLonde Problem

The famous test with an answer key: take a programme whose effect is known from a randomized trial, discard the experimental controls, and try to recover it from survey comparisons. The naive comparison gets the sign wrong — −USD 635 against a true +USD 1,794 — and propensity matching recovers the benchmark almost exactly at USD 1,792. Then the diagnostic turns on itself. Mahalanobis matching produces the tidier balance table and the worse estimate: a worst standardized mean difference of 0.138 against nearest-neighbour’s 0.228, one covariate outside the ±0.1 band against two, and an estimate of USD 719. It balances age and the demographic dummies almost perfectly while balancing lagged earnings less well — and lagged earnings are what predict the outcome. Checking aggregate balance, as it is usually done, would have chosen the wrong estimator.

View example →

Sensitivity Analysis — Rosenbaum Bounds and the E-value

Unconfoundedness cannot be tested, but its fragility can be measured. Two simulated studies built to share a p-value separate cleanly on robustness. Writing Γ for how far a hidden confounder could skew the treatment odds within a matched pair, a large effect on 30 pairs survives to Γ = 4.2 while a small effect on 480 breaks at 1.4 — and the more significant of the two is the less robust, because significance can be bought with sample size and robustness to hidden bias cannot. Then the previous page’s success story is reassessed: the matched estimate that landed on the experimental truth at USD 1,792 turns out fragile at Γ = 1.21. Landing on the truth and being robust are separate properties. The E-value carries the same logic to any risk ratio — a smoking-scale RR of 3.9 gives 7.26, a weak 1.3 gives 1.92.

View example →

Modern Balancing — Entropy Balancing, CBPS and TMLE

Rather than fit a propensity model and check balance afterwards, target balance directly. Entropy balancing delivers on the promise — maximum imbalance collapses from 1.76 standardized units to 5×10−6 in one convex optimization, no fit-check-refit loop. Then the bill: the effective control sample falls to 98 of 429, a 77% loss, with the median control carrying under a fifth of uniform weight and the heaviest 25 holding 38% of the total. Exact balance is paid for in variance, and the balance table does not show it. TMLE — targeted maximum likelihood estimation — adds a targeting step for double robustness and efficiency, and on this sample moves the estimate USD 350 further from the benchmark — a reminder that efficiency is a claim about repeated sampling, not about one answer.

View example →