Potential Outcomes & Matching

An Answer Key for an Observational Problem

The previous group had randomization doing the identifying work for free: treatment was independent of the potential outcomes by design. Almost no policy question comes with that luxury. When we observe only who happened to be treated, the treated and untreated differ systematically, and a naive comparison confounds the effect with those pre-existing differences. This subsection introduces the assumption that stands in for randomization — unconfoundedness — and the family of estimators built on it.

The test case is the most famous in the field, and it has a built-in answer key. LaLonde (1986) asked a devastating question: take a programme whose true effect is known from a randomized experiment, throw away the experimental controls, try to recover that effect from observational comparison data — do econometric methods succeed? His answer was largely no. Dehejia & Wahba (1999) revived the debate by showing propensity-score methods could recover the benchmark after all. The notebook replays the whole argument.

The National Supported Work Demonstration randomized 6–18 months of guaranteed employment for severely disadvantaged workers. Its experimental sample — 185 treated against 260 experimental controls — gives the truth by a simple difference in means: an average treatment effect on the treated (ATT) of USD 1,794 — the effect for those who actually took the training, which is the policy-relevant quantity and what the experiment measures. LaLonde’s challenge discards those 260 controls and substitutes 429 respondents from the Current Population Survey, who are nothing like the trainees. The naive comparison then says the programme reduced earnings by USD 635 — a bias of USD 2,429, and a sign error.

Unconfoundedness asserts that conditional on covariates $X$, treatment is as good as random, together with overlap ($0<e(X)<1$). It is what replaces randomization, and unlike randomization it is not testable: it claims no unobserved confounder remains, and no diagnostic can check that. All that can be checked is balance on the covariates in hand — a distinction the last section of this page turns out to depend on.

Conditioning on eight covariates at once is hard. Rosenbaum & Rubin (1983) proved the simplification that makes the field practical: the scalar propensity score $e(X)=P(W=1\mid X)$ is a balancing score, so if unconfoundedness holds given $X$ it also holds given $e(X)$ alone. Eight dimensions collapse to one. Estimating it by logistic regression also exposes the overlap problem directly — the CPS controls pile up near $e \approx 0$, and only 273 of the 429 fall inside the [0.05, 0.95] common-support band.

Matching, and What Balance Does Not Tell You

Two classic matching schemes are built from scratch, both estimating the ATT with replacement, meaning a control unit may serve as the match for more than one treated unit: nearest-neighbour on the propensity score, and Mahalanobis matching on the full covariate vector. The conventional diagnostic is covariate balance — standardized mean differences collapsing inside a ±0.1 band after matching. Before matching, 7 of 8 covariates sit outside it, with the black indicator at 1.76 and married at −0.83.

The two schemes then disagree by a factor of two and a half: nearest-neighbour propensity matching gives USD 1,792 against the experimental truth of USD 1,794, while Mahalanobis gives USD 719. The reason is where this page stops being a demonstration and starts being a warning.

Mahalanobis produces the tidier balance table and the worse estimate. On every aggregate summary it is the better-balanced match — a maximum $|{\rm SMD}|$ of 0.138 against nearest-neighbour’s 0.228, and 1 of 8 covariates outside the ±0.1 band against 2 of 8. And it is the one that misses, by more than a thousand dollars.

Standardized mean differences, treated-group standard deviation in the denominator. Experimental truth USD 1,794.

What separates them is which covariates each balances. Lagged earnings are the strongly prognostic variables here, and on re75 nearest-neighbour achieves $|{\rm SMD}| = 0.008$ against Mahalanobis’s 0.138. Mahalanobis buys its tidy aggregate on age and the demographic dummies — balancing them almost perfectly — and pays for it on the variables that actually predict 1978 earnings. No overall balance summary would have revealed that.

So the standard advice, check balance rather than just reporting a number, is necessary and not sufficient. Applied to the aggregate, as it usually is, it would have selected the wrong estimator here. Balance has to be judged on the covariates that drive the outcome — which is a claim about the outcome model rather than about the matching, and it is precisely the knowledge unconfoundedness already assumes you have. The diagnostic leans on the assumption it was brought in to police.

Weighting, Double Robustness, and the Spread

Matching discards most of the control sample, since each treated unit consumes one control. Inverse-probability weighting instead uses every unit, weighting controls by $e(X)/(1-e(X))$ so the reweighted controls resemble the treated — consistent if the propensity model is right. Doubly-robust AIPW adds an outcome regression $m_0(X)=E[Y\mid X, W=0]$ and is consistent if either model is right: two chances instead of one.

Every method moves the estimate off the naive −USD 635 and onto the correct side of zero, which is the headline and a genuine success. The spread among them is the caveat, and it is wide: three of the four land between USD 1,200 and USD 1,800, Mahalanobis sits well below, and all four fall at or below the benchmark rather than scattering around it. A practitioner without the answer key would face four defensible numbers spanning a factor of two and no principled way to choose. That is LaLonde’s original complaint, restated with better tools.

Where this sits

Choosing which covariates enter the propensity model is the confounder-selection face of the Variable Selection arc — include every confounder, exclude post-treatment variables and instruments. The doubly-robust idea returns, generalized with machine-learning nuisance models and cross-fitting, in Double/Debiased ML. And the untestable assumption at the centre of this page is what the next subsection interrogates directly: if unconfoundedness fails, by how much would it have to fail to overturn the conclusion?

Notebooks

Downloads

References

• Rosenbaum, P. R. & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika 70(1), 41–55. — the propensity score
• LaLonde, R. J. (1986). Evaluating training programs in business and industry. Performance + Instruction 25(5), 32–35. — the benchmark that discredited the estimators
• Dehejia, R. H. & Wahba, S. (1999). Causal Effects in Nonexperimental Studies: Reevaluating the Evaluation of Training Programs. Journal of the American Statistical Association 94(448), 1053–1062. — and the reply
• Abadie, A. & Imbens, G. W. (2006). Large Sample Properties of Matching Estimators for Average Treatment Effects. Econometrica 74(1), 235–267. — large-sample properties of matching