Causal Inference III(b) — Weak-Instrument-Robust Inference¶
When the first-stage F is not enough: Anderson-Rubin confidence sets and the tF procedure¶
The instrumental-variables notebook diagnosed weak instruments with the first-stage F and the "F > 10" rule of thumb. But diagnosing weakness is not the same as doing valid inference despite it. This notebook is about the fix. The problem is sharp: when instruments are weak, the ordinary 2SLS Wald confidence interval ($\hat\beta\pm1.96\,\widehat{\text{se}}$) is invalid — the 2SLS estimate is biased toward OLS, its standard error understates the true uncertainty, and the interval can cover the truth far less than 95% of the time. With many weak instruments (the Angrist-Krueger quarter-of-birth setting, hundreds of interactions), the failure is catastrophic.
Two tools do inference that stays valid regardless of instrument strength:
- The Anderson-Rubin (AR) test and confidence set — invert a test of the structural parameter that is valid for any instrument strength (and any number of instruments). Its confidence set has exact coverage, and — honestly — becomes wide or even unbounded when the instrument carries little information.
- The tF procedure (Lee, McCrary, Moon & Weidner 2022) — a simple adjustment to the 2SLS t-ratio's critical value as a function of the first-stage F, which recalibrates the old "F > 10" folklore into a valid rule (the true threshold for using 1.96 is F > 104.7).
We show the coverage collapse, build the AR confidence set from scratch, and apply everything to the Card returns-to-schooling data — where valid inference makes the estimate notably more tentative than the naive CI suggests. Python-lead (from-scratch AR); R companion uses ivmodel (AR/CLR) and ivDiag (tF, effective-F).
1. The coverage collapse — why the 2SLS Wald CI fails¶
We simulate the endogenous-regressor model with a known effect $\beta=1$ and a confounder that biases OLS upward. Two regimes: a strong single instrument, and many (30) weak instruments — the structural feature of the Angrist-Krueger design, where quarter-of-birth was interacted into hundreds of weak instruments. We record, over many replications, how often the nominal-95% 2SLS Wald interval actually contains the truth.
With a strong instrument, everything is fine. With many weak instruments, 2SLS is badly biased toward OLS (the many-instruments bias) and the Wald interval's coverage collapses — it excludes the truth almost always, advertising precision it does not have. This is exactly the Bound-Jaeger-Baker critique quantified: a confidence interval you must not trust.
import numpy as np, pandas as pd, matplotlib.pyplot as plt, warnings
warnings.filterwarnings("ignore")
from scipy.stats import f as fdist
BLUE="#2b6cb0"; RED="#c53030"; GREEN="#2f855a"; ORANGE="#dd6b20"; GREY="#a0aec0"; PURP="#6b46c1"
def sim(seed,L,strength,n=500,beta=1.0,rho=2.0):
r=np.random.default_rng(seed); Z=r.normal(0,1,(n,L)); u=r.normal(0,1,n)
D=Z@np.full(L,strength)+u+r.normal(0,1,n); Y=beta*D+rho*u+r.normal(0,1,n); return Z,D,Y
def tsls(Z,D,Y):
n=len(Y); Zc=np.column_stack([np.ones(n),Z]); Xc=np.column_stack([np.ones(n),D])
PZ=Zc@np.linalg.solve(Zc.T@Zc,Zc.T@Xc); b=np.linalg.solve(PZ.T@PZ,PZ.T@Y); res=Y-Xc@b
s2=res@res/(n-2); return b[1], np.sqrt((s2*np.linalg.inv(PZ.T@PZ))[1,1])
def AR_covers(Z,D,Y,b0,alpha=0.05): # F-test all instruments =0 in OLS(Y-b0 D ~ 1+Z)
n,L=Z.shape; e=Y-b0*D; X1=np.column_stack([np.ones(n),Z]); X0=np.ones((n,1))
r1=e-X1@np.linalg.lstsq(X1,e,rcond=None)[0]; r0=e-X0@np.linalg.lstsq(X0,e,rcond=None)[0]
F=((r0@r0-r1@r1)/L)/(r1@r1/(n-L-1)); return F<fdist.ppf(1-alpha,L,n-L-1)
rows=[]
for L,st,lab in [(1,1.0,"strong (1 instrument)"),(30,0.03,"many weak (30, AK-style)")]:
wc=[]; ac=[]; bs=[]; ols=[]
for s in range(800):
Z,D,Y=sim(s,L,st); b,se=tsls(Z,D,Y); wc.append(abs(b-1)<=1.96*se); ac.append(AR_covers(Z,D,Y,1.0)); bs.append(b); ols.append(np.polyfit(D.mean(1) if L>1 else D,Y,1)[0] if False else np.polyfit(D,Y,1)[0])
rows.append((lab,np.mean(bs),np.mean(ols),100*np.mean(wc),100*np.mean(ac)))
print(f"{lab:26s}: 2SLS={np.mean(bs):.2f} (true 1, OLS={np.mean(ols):.2f}) | Wald cov {100*np.mean(wc):.0f}% | AR cov {100*np.mean(ac):.0f}%")
fig,ax=plt.subplots(figsize=(8,4.2))
lab=[r[0] for r in rows]; x=np.arange(len(lab)); w=0.35
ax.bar(x-w/2,[r[3] for r in rows],w,color=RED,label="2SLS Wald")
ax.bar(x+w/2,[r[4] for r in rows],w,color=GREEN,label="Anderson-Rubin")
ax.axhline(95,color="k",ls="--",label="95% nominal"); ax.set_xticks(x); ax.set_xticklabels(lab,fontsize=9)
ax.set_ylabel("CI coverage of the true effect"); ax.set_title("Many weak instruments: 2SLS-Wald coverage collapses; AR holds")
for i,r in enumerate(rows): ax.text(i-w/2,r[3]+2,f"{r[3]:.0f}%",ha="center",fontsize=8); ax.text(i+w/2,r[4]+2,f"{r[4]:.0f}%",ha="center",fontsize=8)
ax.legend(); plt.tight_layout(); plt.show()
print("With 30 weak instruments the 2SLS estimate is dragged from the truth (1) toward OLS, and its Wald interval covers the")
print("truth almost never -- the many-weak-instruments trap behind the Bound-Jaeger-Baker critique. Anderson-Rubin stays valid.")
strong (1 instrument) : 2SLS=1.00 (true 1, OLS=1.67) | Wald cov 95% | AR cov 95%
many weak (30, AK-style) : 2SLS=1.84 (true 1, OLS=1.99) | Wald cov 4% | AR cov 94%
With 30 weak instruments the 2SLS estimate is dragged from the truth (1) toward OLS, and its Wald interval covers the truth almost never -- the many-weak-instruments trap behind the Bound-Jaeger-Baker critique. Anderson-Rubin stays valid.
2. The Anderson-Rubin confidence set — from scratch¶
Anderson & Rubin's idea (1949) sidesteps estimating $\beta$ altogether. To test a candidate value $\beta_0$, form the residual $Y-\beta_0 D$; if $\beta_0$ is the true effect, this residual is exogenous, so the instruments should not predict it. So we regress $Y-\beta_0 D$ on the instruments (and controls) and run an $F$-test that all instrument coefficients are zero. The AR confidence set is every $\beta_0$ the test does not reject. Because the test is exact for any first-stage strength, the set has correct coverage no matter how weak the instruments — the property 2SLS lacks.
We invert the test on a grid. When the instrument is informative the set is a tight interval; when it is weak the set widens, and can become a half-line or the whole real line — an honest signal that the data cannot pin down the effect.
On the single weak-instrument sample below, the contrast to look for is width, not coverage. The Wald interval comes out at [−1.28, 3.12] and the AR set runs to the edge of the search grid, [−4.00, 6.00], which means effectively unbounded. Both happen to contain the true value of 1 in this particular draw — coverage is a repeated-sampling property, and no single sample can display it. That is exactly why section 1 measured it over many replications, where the Wald interval covered 4% of the time. What one sample can show is that the Wald interval advertises a precision the instruments do not support, while the AR set reports honestly that the data cannot pin the effect down at all.
def AR_pval(Z,D,Y,b0):
n=len(Y); Z=Z.reshape(n,-1); L=Z.shape[1]; e=Y-b0*D
X1=np.column_stack([np.ones(n),Z]); r1=e-X1@np.linalg.lstsq(X1,e,rcond=None)[0]
r0=e-np.mean(e); F=((r0@r0-r1@r1)/L)/(r1@r1/(n-L-1)); return 1-fdist.cdf(F,L,n-L-1)
grid=np.linspace(-4,6,1201)
# a strong and a weak single-instrument sample
Zs,Ds,Ys=sim(0,1,1.0); Zw,Dw,Yw=sim(0,1,0.06)
def ar_set(Z,D,Y): inc=grid[[AR_pval(Z,D,Y,g)>0.05 for g in grid]]; return inc
def wald(Z,D,Y): b,se=tsls(Z,D,Y); return b-1.96*se,b+1.96*se
fig,ax=plt.subplots(1,2,figsize=(13,4.4))
for a,(Z,D,Y),lab in zip(ax,[(Zs,Ds,Ys),(Zw,Dw,Yw)],["strong instrument","weak instrument"]):
pv=np.array([AR_pval(Z,D,Y,g) for g in grid]); a.plot(grid,pv,color=BLUE,lw=2)
a.axhline(0.05,color=RED,ls="--"); a.axvline(1.0,color=GREEN,ls=":",label="true β=1")
lo,hi=wald(Z,D,Y); a.axvspan(lo,hi,color=ORANGE,alpha=.2,label=f"2SLS Wald CI")
inc=ar_set(Z,D,Y); a.axvspan(inc.min(),inc.max(),color=BLUE,alpha=.12,label="AR set")
a.set_xlabel("β₀"); a.set_ylabel("AR p-value"); a.set_title(f"{lab}"); a.legend(fontsize=7)
plt.tight_layout(); plt.show()
lo,hi=wald(Zw,Dw,Yw); inc=ar_set(Zw,Dw,Yw)
print(f"weak sample: 2SLS-Wald CI [{lo:.2f}, {hi:.2f}] vs AR set [{inc.min():.2f}, {inc.max():.2f}]"+(" (touches grid edge = effectively unbounded)" if inc.min()<=grid[1] or inc.max()>=grid[-2] else ""))
print("The AR set is the range of effect values the instruments cannot rule out. Under a weak instrument it is wide/unbounded")
print("-- correctly reporting little information -- whereas the Wald CI stays narrow and can exclude the truth.")
weak sample: 2SLS-Wald CI [-1.28, 3.12] vs AR set [-4.00, 6.00] (touches grid edge = effectively unbounded) The AR set is the range of effect values the instruments cannot rule out. Under a weak instrument it is wide/unbounded -- correctly reporting little information -- whereas the Wald CI stays narrow and can exclude the truth.
3. The tF procedure — recalibrating "F > 10" to "F > 104.7"¶
The AR set is fully robust but, for a single instrument, many practitioners still want a $\hat\beta\pm(\text{critical value})\times\widehat{\text{se}}$ interval. Lee, McCrary, Moon & Weidner (2022) provide it: the tF procedure keeps the 2SLS estimate and standard error but replaces the fixed critical value 1.96 with an adjusted value $c_F$ that depends on the first-stage F-statistic. When the first stage is strong, $c_F\to1.96$; when it is weak, $c_F$ grows, widening the interval to restore validity. Their headline recalibrates the folklore: to justify the usual $t>1.96$ at the 5% level, the first-stage F must exceed 104.7 — not 10. The old rule of thumb was far too lenient.
The consequence is sobering for many published IV studies whose first-stage F sat between 10 and 100: their conventional confidence intervals were too narrow. We illustrate on the Card data next.
One note on provenance, since it matters for reading the numbers. The critical value $c_F \approx 2.934$ used below is not derived in this notebook — it is Lee et al.'s critical-value function evaluated at Card's first-stage F, as implemented in the ivDiag package and used in the R companion. Everything else here (the AR inversion, the coverage simulation, the 2SLS fit) is computed from scratch.
# the memorable thresholds from Lee, McCrary, Moon & Weidner (2022)
print("Lee-McCrary-Moon-Weidner (2022) tF, 5% level:")
print(" first-stage F > 104.7 -> the usual t > 1.96 is valid (c_F = 1.96)")
print(" first-stage F ~ 10 -> c_F is much larger than 1.96; the conventional CI badly understates uncertainty")
print(" the classic 'F > 10' rule of thumb is NOT enough for valid conventional inference.")
# Card first-stage F for context (computed from scratch)
d=pd.read_csv("card.csv"); ctrl=["exper","expersq","black","south","smsa","reg661","reg662","reg663","reg664","reg665","reg666","reg667","reg668","smsa66"]
d=d.dropna(subset=["lwage","educ","nearc4"]+ctrl).reset_index(drop=True); n=len(d)
C=np.column_stack([np.ones(n)]+[d[c].values for c in ctrl]); Z=d["nearc4"].values; D=d["educ"].values
Xf=np.column_stack([Z,C]); rf=D-Xf@np.linalg.lstsq(Xf,D,rcond=None)[0]; r0=D-C@np.linalg.lstsq(C,D,rcond=None)[0]
F_card=((r0@r0-rf@rf)/1)/(rf@rf/(n-Xf.shape[1]))
print(f"\nCard first-stage F = {F_card:.1f} -> ABOVE the old 'F>10' rule, but FAR BELOW 104.7:")
print(f" the conventional 2SLS CI is too narrow; valid inference (AR / tF) must be wider. (ivDiag: c_F ~ 2.93.)")
Lee-McCrary-Moon-Weidner (2022) tF, 5% level: first-stage F > 104.7 -> the usual t > 1.96 is valid (c_F = 1.96) first-stage F ~ 10 -> c_F is much larger than 1.96; the conventional CI badly understates uncertainty the classic 'F > 10' rule of thumb is NOT enough for valid conventional inference. Card first-stage F = 13.3 -> ABOVE the old 'F>10' rule, but FAR BELOW 104.7: the conventional 2SLS CI is too narrow; valid inference (AR / tF) must be wider. (ivDiag: c_F ~ 2.93.)
4. Card returns-to-schooling under valid inference¶
We put the three intervals side by side on Card's data (instrument: grew up near a four-year college; first-stage F ≈ 13–14). The naive 2SLS Wald interval excludes zero — the tidy "significant ~13% return" of the IV notebook. But that F is well below 104.7, so valid inference tells a more tentative story: the Anderson-Rubin set is wider and asymmetric (its upper end stretches higher), and the tF interval — with its inflated critical value $c_F\approx2.9$ — is wider still and includes zero. The AR set (more powerful) just excludes zero; tF (a simple, conservative correction) does not. Either way, the honest conclusion is that Card's estimate is imprecise and borderline once weak-instrument uncertainty is taken seriously — a very different message from the naive interval, and a caution that applies to a large swath of the applied IV literature.
# 2SLS Wald and AR set on Card (from scratch)
Zmat=np.column_stack([Z,C]); Xmat=np.column_stack([D,C]); Y=d["lwage"].values
PZ=Zmat@np.linalg.solve(Zmat.T@Zmat,Zmat.T@Xmat); b2=np.linalg.solve(PZ.T@PZ,PZ.T@Y); iv=b2[0]
res=Y-Xmat@b2; s2=res@res/(n-Xmat.shape[1]); se=np.sqrt((s2*np.linalg.inv(PZ.T@PZ))[0,0])
def AR_p_ctrl(b0):
e=Y-b0*D; X1=np.column_stack([Z,C]); r1=e-X1@np.linalg.lstsq(X1,e,rcond=None)[0]; r0=e-C@np.linalg.lstsq(C,e,rcond=None)[0]
F=((r0@r0-r1@r1)/1)/(r1@r1/(n-X1.shape[1])); return 1-fdist.cdf(F,1,n-X1.shape[1])
g=np.linspace(-0.1,0.5,3001); inc=g[[AR_p_ctrl(x)>0.05 for x in g]]
wald_ci=(iv-1.96*se,iv+1.96*se); ar_ci=(inc.min(),inc.max()); cF=2.934; tf_ci=(iv-cF*se,iv+cF*se)
print(f"2SLS estimate: educ = {iv:.4f} (SE {se:.4f})")
print(f" 2SLS Wald 95% CI : [{wald_ci[0]:.4f}, {wald_ci[1]:.4f}] (excludes 0 -- but F<<104.7, so invalid/too narrow)")
print(f" Anderson-Rubin : [{ar_ci[0]:.4f}, {ar_ci[1]:.4f}] (robust; wider; just excludes 0)")
print(f" tF (c_F={cF}) : [{tf_ci[0]:.4f}, {tf_ci[1]:.4f}] (robust & conservative; INCLUDES 0)")
fig,ax=plt.subplots(figsize=(9,3.6))
for i,(nm,ci,c) in enumerate([("2SLS Wald",wald_ci,ORANGE),("Anderson-Rubin",ar_ci,BLUE),("tF (Lee et al)",tf_ci,PURP)]):
ax.plot(ci,[i,i],color=c,lw=4,solid_capstyle="round"); ax.plot([iv],[i],"o",color="k",ms=6)
ax.text(ci[1]+0.005,i,nm,va="center",fontsize=9)
ax.axvline(0,color=RED,ls="--",label="no effect"); ax.axvline(iv,color=GREY,ls=":",label=f"2SLS point {iv:.2f}")
ax.set_yticks([]); ax.set_ylim(-0.5,2.8); ax.set_xlabel("return to a year of schooling (log points)"); ax.set_title("Card: valid weak-IV inference is wider — and borderline")
ax.legend(loc="upper left",fontsize=8); plt.tight_layout(); plt.show()
print("The naive Wald interval's neat significance is an artifact of treating F=13 as 'strong'. Robust inference widens the")
print("interval; the effect is real by AR but not by the conservative tF -- honestly borderline, as the IV-schooling debate has long held.")
2SLS estimate: educ = 0.1315 (SE 0.0550) 2SLS Wald 95% CI : [0.0238, 0.2392] (excludes 0 -- but F<<104.7, so invalid/too narrow) Anderson-Rubin : [0.0250, 0.2848] (robust; wider; just excludes 0) tF (c_F=2.934) : [-0.0298, 0.2928] (robust & conservative; INCLUDES 0)
The naive Wald interval's neat significance is an artifact of treating F=13 as 'strong'. Robust inference widens the interval; the effect is real by AR but not by the conservative tF -- honestly borderline, as the IV-schooling debate has long held.
5. The instruments that were never there — Angrist & Krueger on the real data¶
Section 1 simulated the many-weak-instruments failure in the Angrist–Krueger style. This runs it on the study itself: 329,509 men born 1930–1939, from the 1980 census, the sample behind one of the most cited papers in applied economics.
The logic is elegant. Compulsory schooling laws let you leave at 16, but you start school in the year you turn 6 — so children born early in the year hit their sixteenth birthday having completed less schooling than those born late. Quarter of birth therefore shifts education, and nothing about the season of your birth should touch your wage except through schooling.
Two specifications matter. The simple one uses three quarter-of-birth dummies. The celebrated one interacts quarter with year of birth and with state of birth, giving 180 instruments — done to squeeze out more precision, and it works: the standard error halves.
Then Bound, Jaeger & Baker (1995) asked the question that turned the paper into a cautionary tale. Throw the real quarters away, replace them with random draws, and run exactly the same specification. If the machinery is doing what it claims, this should return nothing.
import zipfile, scipy.sparse as sp
ak = pd.read_csv("qob.csv.gz")
n = len(ak); y = ak.lwklywge.values; educ = ak.educ.values.astype(float)
print(f"Angrist & Krueger (1991): {n:,} men, born 19{ak.yob.min()}-19{ak.yob.max()}, "
f"{ak.pob.nunique()} states of birth")
print(f" mean log weekly wage {y.mean():.4f}; mean schooling {educ.mean():.3f} years")
print()
# quarter of birth really does move schooling -- the first stage is visible in the raw means
qm = ak.groupby("qob").educ.mean()
print("mean years of schooling by quarter of birth:")
for q, v in qm.items(): print(f" Q{q} {v:.4f}")
print(f" Q4 - Q1 = {qm[4]-qm[1]:+.4f} years -- small, real, and the whole basis of the design")
print()
def dummies(codes, drop_first=True):
u = np.unique(codes); u = u[1:] if drop_first else u
idx = {v: j for j, v in enumerate(u)}
r, c = [], []
for i, v in enumerate(codes):
if v in idx: r.append(i); c.append(idx[v])
return sp.csc_matrix((np.ones(len(r)), (r, c)), shape=(n, len(u)))
ones = sp.csc_matrix(np.ones((n, 1)))
D_yob, D_pob = dummies(ak.yob.values), dummies(ak.pob.values)
W = sp.hstack([ones, D_yob, D_pob]).tocsc() # exogenous controls: YOB and POB main effects
def interacted(qob): # QOB x YOB and QOB x POB -- the 180-instrument set
blocks = []
for q in (2, 3, 4):
M = sp.diags((qob == q).astype(float))
blocks += [M @ sp.hstack([ones, D_yob]).tocsc(), M @ D_pob]
return sp.hstack(blocks).tocsc()
def qob_only(qob): # the simple set: three quarter dummies
return sp.hstack([sp.csc_matrix((qob == q).astype(float).reshape(-1, 1)) for q in (2, 3, 4)]).tocsc()
def ak_tsls(Zx):
"""2SLS of log wage on schooling, instrumented by Zx, controlling for W."""
A = sp.hstack([W, Zx]).tocsc(); X = sp.hstack([sp.csc_matrix(educ.reshape(-1, 1)), W]).tocsc()
AA = (A.T @ A).toarray(); AX = np.asarray((A.T @ X).todense()); Ay = np.asarray(A.T @ y).ravel()
pi = np.linalg.lstsq(AA, AX, rcond=None)[0]
XPX = AX.T @ pi; b = np.linalg.lstsq(XPX, pi.T @ Ay, rcond=None)[0]
res = y - np.asarray(X @ b).ravel(); k = np.linalg.matrix_rank(XPX)
se = np.sqrt((res @ res / (n - k)) * np.linalg.pinv(XPX)[0, 0])
pe = np.linalg.lstsq(AA, np.asarray(A.T @ educ).ravel(), rcond=None)[0]
rss1 = ((educ - np.asarray(A @ pe).ravel()) ** 2).sum()
WW = (W.T @ W).toarray(); pw = np.linalg.lstsq(WW, np.asarray(W.T @ educ).ravel(), rcond=None)[0]
rss0 = ((educ - np.asarray(W @ pw).ravel()) ** 2).sum()
q = np.linalg.matrix_rank(AA) - np.linalg.matrix_rank(WW)
F = ((rss0 - rss1) / q) / (rss1 / (n - np.linalg.matrix_rank(AA)))
return b[0], se, F, q
X0 = sp.hstack([sp.csc_matrix(educ.reshape(-1, 1)), W]).tocsc()
XX = (X0.T @ X0).toarray()
b_ols = np.linalg.lstsq(XX, np.asarray(X0.T @ y).ravel(), rcond=None)[0]
r0 = y - np.asarray(X0 @ b_ols).ravel()
se_ols = np.sqrt((r0 @ r0 / (n - np.linalg.matrix_rank(XX))) * np.linalg.pinv(XX)[0, 0])
b3, se3, F3, q3 = ak_tsls(qob_only(ak.qob.values))
b180, se180, F180, q180 = ak_tsls(interacted(ak.qob.values))
print(f" {'specification':32} {'return to schooling':>20} {'SE':>8} {'t':>7} {'first-stage F':>15}")
print(f" {'OLS':32} {b_ols[0]:>20.4f} {se_ols:>8.4f} {b_ols[0]/se_ols:>7.1f} {'--':>15}")
print(f" {'2SLS, 3 quarter dummies':32} {b3:>20.4f} {se3:>8.4f} {b3/se3:>7.2f} {F3:>15.2f}")
print(f" {'2SLS, ' + str(q180) + ' interactions':32} {b180:>20.4f} {se180:>8.4f} {b180/se180:>7.2f} {F180:>15.2f}")
print()
print(f"Adding {q180-q3} instruments cut the standard error from {se3:.4f} to {se180:.4f} -- a {100*(1-se180/se3):.0f}% gain in")
print(f"apparent precision -- while the first-stage F fell from {F3:.1f} to {F180:.2f}. Section 3's threshold for")
print(f"valid conventional inference is 104.7. The celebrated specification is short of it by a factor of {104.7/F180:.0f}.")
print()
print("Note also where the estimate went. The simple specification says", f"{b3:.4f};", "OLS says", f"{b_ols[0]:.4f}.")
frac = (b180 - b_ols[0]) / (b3 - b_ols[0])
print(f"The {q180}-instrument version lands at {b180:.4f}, {100*(1-frac):.0f}% of the way back toward OLS -- the direction")
print("many weak instruments always push, because a first stage that is mostly noise reproduces the")
print("endogenous variation it was supposed to purge.")
print()
print("THE PLACEBO. Quarter of birth replaced by a uniform random draw, same specification, 200 times.")
print()
rng = np.random.default_rng(20260813)
pl = np.array([ak_tsls(interacted(rng.integers(1, 5, size=n)))[:3] for _ in range(200)])
tpl = pl[:, 0] / pl[:, 1]
print(f" estimates mean {pl[:,0].mean():.4f} range [{pl[:,0].min():.4f}, {pl[:,0].max():.4f}]")
print(f" standard errors mean {pl[:,1].mean():.4f} (the real specification reports {se180:.4f})")
print(f" first-stage F mean {pl[:,2].mean():.3f} max {pl[:,2].max():.3f} (pure noise gives about 1)")
print()
print(f" significant at 5%: {(np.abs(tpl)>1.96).sum()} of {len(pl)}")
print(f" |t| range {np.abs(tpl).min():.2f} to {np.abs(tpl).max():.2f}, median {np.median(np.abs(tpl)):.2f}")
print(f" larger than the real 2SLS estimate: {(pl[:,0]>b180).sum()} of {len(pl)}")
print(f" larger than OLS: {(pl[:,0]>b_ols[0]).sum()} of {len(pl)}")
print()
print(f"Random numbers, run through the published specification, return {pl[:,0].mean():.4f} with a t-statistic")
print(f"whose median is {np.median(np.abs(tpl)):.1f}. Nearly every draw clears the 5% bar. {(pl[:,0]>b180).sum()} of {len(pl)} produce an estimate")
print("LARGER than the real one. Nothing in the output announces that the instruments are noise.")
print()
print(f"And the standard errors are not lying. The spread of the placebo estimates is {pl[:,0].std(ddof=1):.4f}, against a")
print(f"mean reported standard error of {pl[:,1].mean():.4f} -- calibrated almost exactly. The estimator has an")
print("accurate view of its own sampling variability and no view whatsoever of the fact that it is")
print("estimating nothing. That is the failure to internalise: a standard error answers 'how much would")
print("this move across samples', never 'is this quantity identified'.")
print()
print(f"The one honest signal is the first stage. The placebo F never exceeds {pl[:,2].max():.2f}; the real one is {F180:.2f}.")
print("So the real instruments do carry information -- genuinely more than nothing, and nowhere near")
print("enough. Which is the whole subject of this page: F = 2.58 against a requirement of 104.7.")
fig, ax = plt.subplots(1, 2, figsize=(12, 4.2))
ax[0].hist(pl[:, 0], bins=28, color=GREY, edgecolor="white")
for v, c, lab in [(b_ols[0], BLUE, f"OLS {b_ols[0]:.4f}"), (b180, RED, f"real 2SLS {b180:.4f}"),
(b3, GREEN, f"3 instruments {b3:.4f}")]:
ax[0].axvline(v, color=c, lw=2.2, label=lab)
ax[0].set_xlabel("estimated return to schooling"); ax[0].set_ylabel("placebo runs")
ax[0].set_title("200 runs with randomly generated quarters of birth"); ax[0].legend(fontsize=8)
ax[1].hist(np.abs(tpl), bins=28, color=GREY, edgecolor="white")
ax[1].axvline(1.96, color=RED, lw=2.2, ls="--", label="|t| = 1.96")
ax[1].axvline(abs(b180 / se180), color=BLUE, lw=2.2, label=f"real specification {abs(b180/se180):.1f}")
ax[1].set_xlabel("|t| on the placebo estimate"); ax[1].set_ylabel("placebo runs")
ax[1].set_title(f"{(np.abs(tpl)>1.96).sum()} of {len(pl)} random-instrument runs are 'significant'")
ax[1].legend(fontsize=8)
plt.tight_layout(); plt.show()
print("Left: the placebo estimates straddle OLS, which is where irrelevant instruments send 2SLS.")
print("Right: the significance the specification manufactures out of random numbers.")
Angrist & Krueger (1991): 329,509 men, born 1930-1939, 51 states of birth mean log weekly wage 5.8999; mean schooling 12.770 years mean years of schooling by quarter of birth: Q1 12.6881 Q2 12.7447 Q3 12.8054 Q4 12.8394 Q4 - Q1 = +0.1514 years -- small, real, and the whole basis of the design
specification return to schooling SE t first-stage F OLS 0.0673 0.0003 194.4 -- 2SLS, 3 quarter dummies 0.1077 0.0195 5.52 36.04 2SLS, 180 interactions 0.0928 0.0093 9.98 2.58 Adding 177 instruments cut the standard error from 0.0195 to 0.0093 -- a 52% gain in apparent precision -- while the first-stage F fell from 36.0 to 2.58. Section 3's threshold for valid conventional inference is 104.7. The celebrated specification is short of it by a factor of 41. Note also where the estimate went. The simple specification says 0.1077; OLS says 0.0673. The 180-instrument version lands at 0.0928, 37% of the way back toward OLS -- the direction many weak instruments always push, because a first stage that is mostly noise reproduces the endogenous variation it was supposed to purge. THE PLACEBO. Quarter of birth replaced by a uniform random draw, same specification, 200 times.
estimates mean 0.0651 range [0.0250, 0.1129] standard errors mean 0.0150 (the real specification reports 0.0093) first-stage F mean 0.994 max 1.284 (pure noise gives about 1) significant at 5%: 197 of 200 |t| range 1.52 to 7.27, median 4.44 larger than the real 2SLS estimate: 8 of 200 larger than OLS: 87 of 200 Random numbers, run through the published specification, return 0.0651 with a t-statistic whose median is 4.4. Nearly every draw clears the 5% bar. 8 of 200 produce an estimate LARGER than the real one. Nothing in the output announces that the instruments are noise. And the standard errors are not lying. The spread of the placebo estimates is 0.0150, against a mean reported standard error of 0.0150 -- calibrated almost exactly. The estimator has an accurate view of its own sampling variability and no view whatsoever of the fact that it is estimating nothing. That is the failure to internalise: a standard error answers 'how much would this move across samples', never 'is this quantity identified'. The one honest signal is the first stage. The placebo F never exceeds 1.28; the real one is 2.58. So the real instruments do carry information -- genuinely more than nothing, and nowhere near enough. Which is the whole subject of this page: F = 2.58 against a requirement of 104.7.
Left: the placebo estimates straddle OLS, which is where irrelevant instruments send 2SLS. Right: the significance the specification manufactures out of random numbers.
6. Summary¶
Diagnosing weak instruments (subsection 3's first-stage F) is not enough; inference must be made robust to them. The 2SLS Wald interval is invalid when instruments are weak — and with many weak instruments (the Angrist-Krueger design) its coverage collapsed to near zero in our simulation, the Bound-Jaeger-Baker critique made quantitative. Two fixes restore validity:
- Anderson-Rubin inverts an exact test of the structural parameter, giving a confidence set with correct coverage at any instrument strength (and any number of instruments); it is wide or unbounded precisely when the data are uninformative — an honest confidence set.
- The tF procedure (Lee-McCrary-Moon-Weidner) adjusts the 2SLS critical value by the first-stage F, and recalibrates the folklore: valid conventional inference needs F > 104.7, not 10.
On Card, valid inference (AR wider and just-significant; tF wider still and including zero) revealed the returns-to-schooling estimate as borderline — a far more tentative conclusion than the naive Wald interval, and a warning for the many IV studies whose first-stage F sat between 10 and 100.
Cross-links. This is the honest completion of the Instrumental Variables notebook (subsection 3): there we diagnosed weak instruments, here we do valid inference despite them. The test-inversion logic (invert a test to build a confidence set) is the same idea behind conformal prediction in the ML arc and the placebo/randomization inference of the experiments and synthetic-control notebooks; and the many-weak-instruments bias is a cautionary cousin of the overfitting the cross-fitting in DML guards against.
And the empirical demonstration behind all of it. On Angrist & Krueger's own 329,509 observations, the celebrated 180-instrument specification has a first-stage F of 2.58 — against the 104.7 that conventional t-inference requires. Replacing quarter of birth with random draws and rerunning the same specification returns an average estimate of 0.065 with a median |t| of 4.4, and 197 of 200 draws are significant at 5%. The reported standard errors are correctly calibrated to sampling variation the whole time. A standard error answers how much would this move across samples; it never answers is this quantity identified, and no amount of care in computing one will make it.