RD Validity & Falsification

A Number Is Not a Result

The regression-discontinuity page produced a number and one test. That is not enough to earn a causal reading. An RD estimate becomes credible only after a falsification suite — a set of checks each designed to break the design, which the design then fails to be broken by. This page assembles the standard four and, more usefully, separates what each one actually certifies.

Covariate continuity is the RD analogue of covariate balance in matching. A predetermined covariate — something fixed before treatment — cannot legitimately jump at the cutoff, so running the same RD with that covariate as the outcome should return nothing. To show the test has teeth it is run on two designs: a valid one and one deliberately manipulated so that high-covariate units sort just above the threshold.

The test discriminates cleanly. In the valid design the covariate jump is −0.150 with $p = 0.256$; in the manipulated design it is +1.099 with $p < 0.001$. The density test agrees, passing the valid design at $p = 0.994$ and rejecting the manipulated one at $p = 0.0004$. Both are worth running: the covariate test can catch sorting subtle enough to leave the density smooth, since units can select on something correlated with the outcome without piling up visibly at the threshold.

Placebo Cutoffs

Placebo cutoffs ask whether the estimator finds a jump wherever it looks. If the local-linear machinery would manufacture a discontinuity at any arbitrary point, the one at zero means nothing. So the same estimate is computed at four fake thresholds either side of the real one.

Only the true cutoff produces anything. The effect is specific to the winning threshold rather than spurious curvature the estimator would find anywhere — which is what a valid design requires and what a global-polynomial fit would have struggled to demonstrate.

Where the Checks Stop Agreeing

Donut-hole re-estimates after excluding observations within $\pm\delta$ of the threshold. Manipulation and heaping concentrate right at the cutoff, so a result driven by those suspicious points should change or vanish when they are punched out. Bandwidth sensitivity re-estimates across a range of windows.

Positive and significant throughout — and spanning a factor of two. Across bandwidths the estimate runs 0.0584 to 0.0842, a spread of 41% of the estimate.

Here the four checks stop agreeing, and the disagreement is the point. The effect stays positive and significant at every donut radius and every bandwidth — that is robustness, and it is a real result. But the magnitude is not settled: the donut estimate runs 0.046 to 0.093, a factor of two, and the bandwidth estimate spans 41% of its own size.

Both readings are true, and keeping them apart matters because the four checks are not asking the same question. Covariate continuity, the density test and the placebo cutoffs interrogate the design: is the discontinuity real, and is it specific to the cutoff? Donut and bandwidth interrogate the estimate. On Lee's data the design passes cleanly and the estimate is sensitive.

Treating a passed falsification test as though it certified the point estimate is how a paper comes to claim more robustness than it has. The honest summary here — the design is credible and the magnitude is uncertain — is a stronger statement than it sounds. It is exactly what a real incumbency advantage, estimated locally from a few thousand close races, ought to look like.

Which gives the discipline the whole subsection is built around: a single RD number is not a result. The result is the number together with the suite of checks that failed to break it — and each check read for what it licenses, rather than banked as generic reassurance.

Where this sits

Covariate continuity is the RD cousin of the balance diagnostics in matching — and inherits their limitation, since the last group established that balance on measured covariates cannot speak for unmeasured ones. Placebo cutoffs are the RD form of the placebo and permutation inference used in the randomized experiments group and, later, in synthetic control. A related design worth knowing is the regression kink design, which identifies effects from a discontinuity in the slope of a policy rule rather than its level, using the same local-polynomial machinery.

Notebooks

Downloads

The manipulated design used for contrast is simulated in the notebook, since demonstrating that a test catches sorting requires a dataset where sorting is known to exist.

References

• McCrary, J. (2008). Manipulation of the running variable in the regression discontinuity design: A density test. Journal of Econometrics 142(2), 698–714. — the density test for manipulation
• Cattaneo, M. D., Jansson, M. & Ma, X. (2019). Simple Local Polynomial Density Estimators. Journal of the American Statistical Association 115(531), 1449–1455. — its modern local-polynomial replacement
• Lee, D. S. & Lemieux, T. (2010). Regression Discontinuity Designs in Economics. Journal of Economic Literature 48(2), 281–355. — the validity checks a design should pass