Causal Inference IV(b) — RD Validity, Falsification & Robustness¶
The credibility checklist that turns a discontinuity into causal evidence¶
The regression-discontinuity notebook estimated the incumbency advantage from the jump in Lee's (2008) data and checked the density of the running variable. But a number at a cutoff is only causal if the design is valid, and RD's credibility rests entirely on assumptions that can — and must — be stress-tested. A referee's first questions about any RD are always the same: Could units have sorted across the cutoff? Are the units on either side really comparable? Is the jump special to the true threshold, or would you find one anywhere? Does the estimate survive dropping the points right at the boundary? This notebook builds the standard falsification and robustness suite that answers them.
- Covariate-continuity (placebo-outcome) tests — predetermined covariates must not jump at the cutoff. If they do, units differ across the threshold for reasons other than treatment, and the design is invalid. This is the RD analogue of covariate balance in matching.
- The manipulation (density) test — sorting across the cutoff leaves a footprint in the density of the running variable (McCrary / Cattaneo-Jansson-Ma).
- Placebo cutoffs — there should be no jump at fake thresholds where nothing happens; a real effect appears only at the true cutoff.
- Donut-hole and bandwidth robustness — the estimate should survive excluding the points nearest the boundary (where heaping/manipulation would concentrate) and reasonable changes in bandwidth.
We demonstrate on a valid vs a deliberately manipulated simulated design (so the tests can be shown catching a real violation), then apply the placebo-cutoff and donut-hole checks to the real Lee (2008) data. Python-lead (rdrobust, rddensity); R companion mirrors it.
1. Covariate continuity — the key validity test¶
The core RD assumption is that units just above and just below the cutoff are comparable — so any predetermined covariate (measured before treatment) should vary smoothly through the threshold, with no jump. Testing it is simple and powerful: run the RD with the covariate as the "outcome." A significant discontinuity in a covariate is a red flag that units sorted across the cutoff.
We simulate two designs with the same true effect. In the valid design there is no sorting: a predetermined covariate $W$ is continuous at the cutoff and the RD-on-$W$ finds no jump. In the manipulated design, motivated (high-$W$) units just below the cutoff push themselves just above it — so the units just above are systematically different, and the RD-on-$W$ detects a large, significant jump, correctly condemning the design. The covariate-continuity test is what catches this.
import numpy as np, pandas as pd, matplotlib.pyplot as plt, warnings
warnings.filterwarnings("ignore")
from rdrobust import rdrobust
from rddensity import rddensity
BLUE="#2b6cb0"; RED="#c53030"; GREEN="#2f855a"; ORANGE="#dd6b20"; GREY="#a0aec0"; PURP="#6b46c1"
rng=np.random.default_rng(0); n=5000; tau=5.0
X=rng.uniform(-1,1,n); W=1.5*X+rng.normal(0,1,n) # predetermined covariate (continuous)
Y=2*X+tau*(X>=0)+0.5*W+rng.normal(0,1,n)
# manipulated: high-W units just below the cutoff sort just above
Xm=X.copy(); mask=(X>-0.2)&(X<0)&(W>np.median(W[(X>-0.2)&(X<0)])); Xm[mask]=Xm[mask]+0.22
def rd(y,x,c=0): r=rdrobust(y,x,c=c); return r.coef.iloc[0,0], r.pv.iloc[2,0]
cv_valid=rd(W,X); cv_manip=rd(W,Xm)
mc_valid=float(rddensity(X=X,c=0).test["p_jk"]); mc_manip=float(rddensity(X=Xm,c=0).test["p_jk"])
print("COVARIATE-CONTINUITY test (RD with covariate W as the outcome; jump should be ~0):")
print(f" VALID design : W jump {cv_valid[0]:+.3f} (robust p={cv_valid[1]:.3f}) -> PASS (no jump)")
print(f" MANIPULATED design: W jump {cv_manip[0]:+.3f} (robust p={cv_manip[1]:.3f}) -> FAIL (covariate jumps => sorting)")
print(f"\nMANIPULATION (density) test: VALID p={mc_valid:.3f} (pass) MANIPULATED p={mc_manip:.4f} (fail)")
fig,ax=plt.subplots(1,2,figsize=(13,4.4))
for a,xx,lab in zip(ax,[X,Xm],["VALID (no sorting)","MANIPULATED (sorting at cutoff)"]):
bins=np.linspace(-1,1,41); mid=(bins[:-1]+bins[1:])/2; idx=np.digitize(xx,bins)-1
wm=[W[idx==b].mean() if (idx==b).sum()>5 else np.nan for b in range(len(mid))]
a.scatter(mid,wm,s=16,color=GREY); a.axvline(0,color=RED,ls="--")
a.set_xlabel("running variable"); a.set_ylabel("mean of covariate W"); a.set_title(f"{lab}")
plt.tight_layout(); plt.show()
print("A predetermined covariate must be continuous at the cutoff. In the manipulated design W jumps (the units just above are")
print("the high-W sorters), which the covariate-continuity RD flags. This is the RD twin of covariate balance in matching.")
COVARIATE-CONTINUITY test (RD with covariate W as the outcome; jump should be ~0): VALID design : W jump -0.150 (robust p=0.256) -> PASS (no jump) MANIPULATED design: W jump +1.099 (robust p=0.000) -> FAIL (covariate jumps => sorting) MANIPULATION (density) test: VALID p=0.994 (pass) MANIPULATED p=0.0004 (fail)
A predetermined covariate must be continuous at the cutoff. In the manipulated design W jumps (the units just above are the high-W sorters), which the covariate-continuity RD flags. This is the RD twin of covariate balance in matching.
2. Placebo cutoffs — the effect must be special to the true threshold¶
If the discontinuity at the true cutoff is really caused by treatment, then running the same RD at fake cutoffs — points away from the threshold where nothing changes — should find no jump. A significant "effect" at a placebo cutoff would mean the RD estimator is picking up ordinary curvature in the outcome, not a treatment effect. We apply this to the real Lee (2008) data: the incumbency jump appears only at the true cutoff (margin = 0); at fake cutoffs of ±0.15 and ±0.30 there is nothing. This is strong evidence the estimated effect is a genuine discontinuity, not an artifact.
d=pd.read_csv("lee2008.csv"); x=d.margin.values; y=d.dem_voteshare_next.values
cutoffs=[-0.30,-0.15,0.0,0.15,0.30]; ests=[]; pvs=[]
for c in cutoffs:
sub=np.ones(len(x),bool) if c==0 else (x>c-0.35)&(x<c+0.35)
e,p=rd(y[sub],x[sub],c=c); ests.append(e); pvs.append(p)
tab=pd.DataFrame({"cutoff":cutoffs,"RD jump":np.round(ests,3),"robust p":np.round(pvs,3)})
print(tab.to_string(index=False))
fig,ax=plt.subplots(figsize=(8.5,4.2))
cols=[RED if c==0 else GREY for c in cutoffs]
ax.bar([str(c) for c in cutoffs],ests,color=cols)
for i,(e,p) in enumerate(zip(ests,pvs)): ax.text(i,e+0.003,f"p={p:.2f}",ha="center",fontsize=8)
ax.axhline(0,color="k",lw=.7); ax.set_xlabel("cutoff (0 = true winning threshold)"); ax.set_ylabel("estimated RD jump")
ax.set_title("Placebo cutoffs: a significant jump only at the true threshold"); plt.tight_layout(); plt.show()
print("Only the true cutoff (0, red) shows a significant jump; the fake cutoffs are flat. The incumbency effect is specific to")
print("the winning threshold, exactly as a valid RD requires -- it is not spurious curvature the estimator would find anywhere.")
Mass points detected in the running variable. Mass points detected in the running variable. Mass points detected in the running variable. Mass points detected in the running variable. Mass points detected in the running variable. cutoff RD jump robust p -0.30 -0.001 0.951 -0.15 -0.007 0.747 0.00 0.064 0.000 0.15 0.012 0.539 0.30 -0.027 0.180
Only the true cutoff (0, red) shows a significant jump; the fake cutoffs are flat. The incumbency effect is specific to the winning threshold, exactly as a valid RD requires -- it is not spurious curvature the estimator would find anywhere.
3. Donut-hole and bandwidth robustness¶
Two final checks. Manipulation or heaping (units bunched at round numbers) tends to concentrate right at the cutoff; the donut-hole RD re-estimates after excluding the observations within $\pm\delta$ of the threshold, so a result driven by those suspicious points would change or vanish. And because RD is a local estimator, the estimate is re-computed across a range of bandwidths.
On the Lee data the incumbency estimate stays positive and significant at every donut radius and every bandwidth, which is what robustness means here and is a genuine result. Its magnitude, though, is not settled: the donut estimate runs 0.046 to 0.093, a factor of two, and the bandwidth estimate spans 41% of its own size. Both statements are true, and it is worth keeping them apart — the sign and significance survive, the point estimate is not pinned down.
That distinction matters more than it first appears, because the four checks in this notebook are not all asking the same question. Covariate continuity, the density test and the placebo cutoffs interrogate the design: is the discontinuity real and is it specific to the cutoff? Donut and bandwidth interrogate the estimate. Here the design passes cleanly and the estimate is sensitive, and treating a passed falsification test as though it certified the point estimate is how a paper comes to claim more robustness than it has.
fig,ax=plt.subplots(1,2,figsize=(13,4.4))
# donut-hole
deltas=[0.0,0.005,0.01,0.02,0.05]; de=[]; dp=[]
for dl in deltas:
keep=np.abs(x)>=dl; e,p=rd(y[keep],x[keep]); de.append(e); dp.append(p)
ax[0].plot(deltas,de,"o-",color=BLUE,lw=2)
for i,(e,p) in enumerate(zip(de,dp)): ax[0].annotate(f"p={p:.2f}",(deltas[i],de[i]),textcoords="offset points",xytext=(0,7),fontsize=7,ha="center")
ax[0].axhline(0,color="k",lw=.6); ax[0].set_xlabel("donut radius δ (exclude |margin|<δ)"); ax[0].set_ylabel("RD estimate"); ax[0].set_title("Donut-hole: estimate survives dropping near-cutoff points")
# bandwidth sensitivity
hs=np.linspace(0.05,0.4,15); be=[]
for h in hs:
r=rdrobust(y,x,c=0,h=h); be.append(r.coef.iloc[0,0])
ax[1].plot(hs,be,"s-",color=GREEN,lw=2); r0=rdrobust(y,x,c=0); ax[1].axvline(r0.bws.iloc[0,0],color=RED,ls="--",label=f"CCT bw {r0.bws.iloc[0,0]:.2f}")
ax[1].axhline(0,color="k",lw=.6); ax[1].set_xlabel("bandwidth h"); ax[1].set_ylabel("RD estimate"); ax[1].set_title("Bandwidth sensitivity"); ax[1].legend(fontsize=8)
plt.tight_layout(); plt.show()
be=np.array(be)
print(f" {'donut delta':>12} {'estimate':>10} {'robust p':>10}")
for dl,e,pv in zip(deltas,de,dp):
print(f" {dl:>12.3f} {e:>10.4f} {pv:>10.4f}")
print(f"\n {'bandwidth h':>12} {'estimate':>10}")
for h_,e_ in list(zip(hs,be))[::3]:
print(f" {h_:>12.3f} {e_:>10.4f}")
print(f"\n donut range {min(de):.4f} to {max(de):.4f}")
print(f" bandwidth range {be.min():.4f} to {be.max():.4f} "
f"({100*(be.max()-be.min())/r0.coef.iloc[0,0]:.0f}% of the CCT estimate)")
print()
print("Read those two blocks separately from the three above them, because they answer a different")
print("question. Covariate continuity, the density test and the placebo cutoffs interrogate the DESIGN:")
print("is the discontinuity at zero real, and is it specific to zero? On all three the answer is a clean")
print("yes -- the covariate does not jump, the density does not jump, and of five cutoffs only the true")
print("one produces an effect.")
print()
print("The donut and bandwidth curves interrogate the ESTIMATE, and there the news is more mixed. The")
print("effect is positive and significant at every donut radius and every bandwidth, which is what")
print("robustness means here and is a genuine result. But the magnitude is not settled: the donut")
print(f"estimate runs {min(de):.3f} to {max(de):.3f}, a factor of two, and the bandwidth estimate spans {100*(be.max()-be.min())/r0.coef.iloc[0,0]:.0f}% of its own")
print("size. Calling that 'stable' would overstate it.")
print()
print("Both readings are true and they are not in tension: THE DESIGN IS CREDIBLE AND THE MAGNITUDE IS")
print("UNCERTAIN. Conflating the two -- treating a passed falsification test as though it certified the")
print("point estimate -- is how a paper comes to claim more robustness than it has.")
Mass points detected in the running variable. Mass points detected in the running variable. Mass points detected in the running variable. Mass points detected in the running variable. Mass points detected in the running variable. Mass points detected in the running variable. Mass points detected in the running variable. Mass points detected in the running variable. Mass points detected in the running variable. Mass points detected in the running variable. Mass points detected in the running variable. Mass points detected in the running variable. Mass points detected in the running variable. Mass points detected in the running variable. Mass points detected in the running variable. Mass points detected in the running variable. Mass points detected in the running variable. Mass points detected in the running variable. Mass points detected in the running variable. Mass points detected in the running variable. Mass points detected in the running variable.
donut delta estimate robust p
0.000 0.0637 0.0000
0.005 0.0580 0.0005
0.010 0.0459 0.0327
0.020 0.0541 0.0261
0.050 0.0931 0.0276
bandwidth h estimate
0.050 0.0682
0.125 0.0618
0.200 0.0740
0.275 0.0790
0.350 0.0821
donut range 0.0459 to 0.0931
bandwidth range 0.0584 to 0.0842 (41% of the CCT estimate)
Read those two blocks separately from the three above them, because they answer a different
question. Covariate continuity, the density test and the placebo cutoffs interrogate the DESIGN:
is the discontinuity at zero real, and is it specific to zero? On all three the answer is a clean
yes -- the covariate does not jump, the density does not jump, and of five cutoffs only the true
one produces an effect.
The donut and bandwidth curves interrogate the ESTIMATE, and there the news is more mixed. The
effect is positive and significant at every donut radius and every bandwidth, which is what
robustness means here and is a genuine result. But the magnitude is not settled: the donut
estimate runs 0.046 to 0.093, a factor of two, and the bandwidth estimate spans 41% of its own
size. Calling that 'stable' would overstate it.
Both readings are true and they are not in tension: THE DESIGN IS CREDIBLE AND THE MAGNITUDE IS
UNCERTAIN. Conflating the two -- treating a passed falsification test as though it certified the
point estimate -- is how a paper comes to claim more robustness than it has.
4. Summary¶
An RD estimate earns causal interpretation only after a falsification and robustness suite, and this notebook assembled the standard one:
- covariate-continuity — predetermined covariates must not jump at the cutoff; the test caught a deliberately manipulated design (a large, significant covariate jump) while passing the valid one — the RD analogue of covariate balance;
- manipulation / density — sorting shows up as a density discontinuity (McCrary / Cattaneo-Jansson-Ma), which flagged the manipulated design; (note that the covariate test can catch subtler sorting the density test misses, so run both);
- placebo cutoffs — on Lee's data the incumbency jump appeared only at the true threshold, ruling out spurious curvature;
- donut-hole and bandwidth — the estimate stayed positive and significant when near-cutoff points were excluded and when the bandwidth was varied, though its magnitude moved substantially: 0.046 to 0.093 across donut radii, and a 41% spread across bandwidths.
The discipline: a single RD number is not a result; the result is the number plus the suite of checks that fail to break it. And the checks should be read for what each one tests — the first three license the design, the last two describe how firmly the magnitude is pinned. On Lee's data the honest summary is that the design is credible and the magnitude is uncertain, which is a stronger claim than it sounds: it is exactly what a real incumbency advantage, estimated locally from a few thousand close races, ought to look like. A related design worth knowing is the regression kink design (RKD), which identifies effects from a discontinuity in the slope of a policy rule rather than its level, using the same local-polynomial machinery. Cross-links: covariate-continuity is the RD cousin of matching's balance checks (subsection 2); placebo cutoffs are the RD form of the placebo/permutation inference in the experiments and synthetic-control notebooks; and the whole suite embodies the arc's recurring lesson that a credible causal claim is one that has been attacked and survived. This completes the depth of the Regression Discontinuity subsection.