Causal Inference I(c) — Noncompliance and Cluster Designs¶
When the treatment isn't the assignment, and when the unit isn't the individual — on two real experiments¶
Randomization identifies causal effects, but two features of real field and clinical trials complicate the simple difference in means:
- Noncompliance. Subjects are assigned to treatment, but some don't take it (and some controls find a way to get it). Now "assignment" and "treatment received" differ, and there are two distinct estimands — the effect of offering treatment (ITT) and the effect of taking it among those who comply (CACE/LATE). Comparing people by what they took (as-treated) throws randomization away.
- Clustered assignment. Treatment is often assigned to whole groups — schools, villages, households — not individuals. Outcomes within a cluster are correlated, so the effective sample size is far smaller than the row count, and standard errors that ignore clustering are badly overconfident.
We ground both in real experiments: the Sommer–Zeger vitamin A trial in Indonesia (a landmark encouragement design with real refusal), and Project STAR, the Tennessee class-size experiment (real within-school clustering). We keep one compact known-truth simulation at the end — the only way to see that naive standard errors under-cover. Python leads (from-scratch ITT/CACE/IV and cluster-robust SEs); the R companion uses estimatr. This is the third foundations notebook, extending the RCM (1a) and covariate adjustment (1b).
1. Noncompliance and the intention-to-treat effect — the vitamin A trial¶
Sommer & Zeger (1991) studied whether vitamin A supplementation reduces child mortality in Indonesia. Villages were randomized to a vitamin A program or control; but of the children assigned to receive it, about 20% never did (remote access, refusal, absence). Controls had no access. This is the canonical one-sided noncompliance setup, with three latent types:
- Compliers — take the vitamin if and only if assigned to it;
- Never-takers — never take it (the ~20% in the treatment arm who didn't);
- Always-takers — would take it regardless (absent here, since controls had no access).
The published cell counts give us everything. The intention-to-treat (ITT) effect compares everyone as assigned, ignoring what they took — the effect of the program, and always estimable from randomization alone.
import numpy as np, pandas as pd, matplotlib.pyplot as plt, statsmodels.formula.api as smf, warnings
warnings.filterwarnings("ignore")
BLUE="#2b6cb0"; RED="#c53030"; GREEN="#2f855a"; ORANGE="#dd6b20"; GREY="#a0aec0"; PURP="#6b46c1"
# Published Sommer-Zeger cell counts: (assigned Z, received D, N children, deaths)
cells=[(1,1,9675,12),(1,0,2419,34),(0,0,11588,74)]
rows=[]
for Z,D,N,dead in cells: rows += [(Z,D,1)]*dead + [(Z,D,0)]*(N-dead)
sz=pd.DataFrame(rows,columns=["Z","D","death"])
print(f"Sommer-Zeger vitamin A trial: {len(sz):,} children ({(sz.Z==1).sum():,} assigned vitamin A, {(sz.Z==0).sum():,} control)")
print(f" of those assigned vitamin A, {100*sz[sz.Z==1].D.mean():.0f}% actually received it (compliers); {100*(1-sz[sz.Z==1].D.mean()):.0f}% never-takers")
m_vit=sz[sz.Z==1].death.mean(); m_ctl=sz[sz.Z==0].death.mean()
itt_y=m_vit-m_ctl; itt_d=sz[sz.Z==1].D.mean()-sz[sz.Z==0].D.mean()
print(f"\n mortality: vitamin-A arm {1000*m_vit:.2f} per 1,000 control arm {1000*m_ctl:.2f} per 1,000")
print(f" ITT (effect of the PROGRAM) = {1000*itt_y:+.2f} per 1,000 (a {100*itt_y/m_ctl:+.0f}% change vs control)")
print(f" compliance gap ITT_D = {itt_d:.2f} (share who take up when offered)")
fig,ax=plt.subplots(1,2,figsize=(13,4.4))
ax[0].bar(["control\n(assigned)","vitamin A\n(assigned)"],[1000*m_ctl,1000*m_vit],color=[BLUE,GREEN])
ax[0].set_ylabel("mortality per 1,000"); ax[0].set_title(f"ITT: compare AS ASSIGNED = {1000*itt_y:+.2f} per 1,000")
for i,v in enumerate([1000*m_ctl,1000*m_vit]): ax[0].text(i,v+0.1,f"{v:.2f}",ha="center")
comp=[sz[sz.Z==1].D.mean(),1-sz[sz.Z==1].D.mean()]
ax[1].bar(["compliers\n(took it)","never-takers\n(didn't)"],[100*comp[0],100*comp[1]],color=[GREEN,GREY])
ax[1].set_ylabel("% of vitamin-A arm"); ax[1].set_title("One-sided noncompliance: ~20% never-takers")
for i,v in enumerate([100*comp[0],100*comp[1]]): ax[1].text(i,v+1,f"{v:.0f}%",ha="center")
plt.tight_layout(); plt.show()
print("ITT answers a real policy question -- 'what happens if we LAUNCH the program?' -- and needs no assumption beyond randomization.")
print("But it is diluted by the 20% who never take the vitamin: it understates the effect of the vitamin ITSELF.")
Sommer-Zeger vitamin A trial: 23,682 children (12,094 assigned vitamin A, 11,588 control) of those assigned vitamin A, 80% actually received it (compliers); 20% never-takers mortality: vitamin-A arm 3.80 per 1,000 control arm 6.39 per 1,000 ITT (effect of the PROGRAM) = -2.58 per 1,000 (a -40% change vs control) compliance gap ITT_D = 0.80 (share who take up when offered)
ITT answers a real policy question -- 'what happens if we LAUNCH the program?' -- and needs no assumption beyond randomization. But it is diluted by the 20% who never take the vitamin: it understates the effect of the vitamin ITSELF.
2. As-treated is biased; CACE (= IV) recovers the complier effect¶
The tempting fix — compare children by what they actually took (as-treated), or compare compliers to all controls (per-protocol) — breaks randomization. The 20% who refused the vitamin are not a random subset: they were harder to reach, poorer, sicker, with higher baseline mortality. Comparing takers to non-takers therefore conflates the vitamin with who takes it, and here it overstates the benefit.
The valid target is the Complier Average Causal Effect (CACE), a.k.a. LATE — the effect among compliers. With one-sided noncompliance, Bloom's estimator gives it exactly: $$\text{CACE}=\frac{\text{ITT}_Y}{\text{ITT}_D}=\frac{\text{effect of assignment on the outcome}}{\text{effect of assignment on take-up}},$$ which is precisely the instrumental-variables (2SLS) estimate using assignment $Z$ as an instrument for receipt $D$. A noncompliant RCT is the textbook valid instrument: assignment is randomized (independent), moves take-up (relevant), and affects the outcome only through it (exclusion).
at = sz[sz.D==1].death.mean() - sz[sz.D==0].death.mean() # as-treated (naive)
pp = sz[(sz.Z==1)&(sz.D==1)].death.mean() - sz[sz.Z==0].death.mean() # per-protocol
cace = itt_y/itt_d # Bloom
iv = smf.ols("death~Z",sz).fit().params['Z'] / smf.ols("D~Z",sz).fit().params['Z'] # 2SLS
print("Effect of vitamin A on child mortality (per 1,000), four ways:")
print(f" ITT (program, as assigned) {1000*itt_y:+.2f} valid, but diluted by noncompliance")
print(f" as-treated (took vs didn't) {1000*at:+.2f} BIASED: refusers were sicker -> overstates benefit")
print(f" per-protocol (compliers vs ctrl) {1000*pp:+.2f} BIASED: same selection problem")
print(f" CACE = ITT_Y/ITT_D (Bloom) {1000*cace:+.2f} valid effect AMONG COMPLIERS")
print(f" CACE via 2SLS (Z instruments D) {1000*iv:+.2f} identical -- noncompliant RCT = valid instrument")
print(f"\n Among compliers, vitamin A cut mortality by ~{100*(-cace)/m_ctl:.0f}% relative to the control rate.")
print(f" Why as-treated misleads: non-takers' mortality {1000*sz[(sz.Z==1)&(sz.D==0)].death.mean():.2f}/1,000 >> takers' {1000*sz[(sz.Z==1)&(sz.D==1)].death.mean():.2f}/1,000 -- selection, not the vitamin.")
fig,ax=plt.subplots(figsize=(8.5,4.4))
labs=["ITT\n(program)","as-treated\n(BIASED)","per-protocol\n(BIASED)","CACE=IV\n(compliers)"]
vals=[1000*itt_y,1000*at,1000*pp,1000*cace]; cols=[BLUE,RED,ORANGE,GREEN]
b=ax.bar(labs,vals,color=cols); ax.axhline(0,color="k",lw=.6)
for r,v in zip(b,vals): ax.text(r.get_x()+r.get_width()/2, v-0.25, f"{v:+.2f}", ha="center", color="white", fontweight="bold")
ax.set_ylabel("effect on mortality per 1,000"); ax.set_title("ITT (diluted) < CACE (complier effect); as-treated exaggerates")
plt.tight_layout(); plt.show()
print("ITT and CACE answer DIFFERENT valid questions (launch-the-program vs take-the-vitamin); as-treated/per-protocol answer neither.")
Effect of vitamin A on child mortality (per 1,000), four ways: ITT (program, as assigned) -2.58 valid, but diluted by noncompliance as-treated (took vs didn't) -6.47 BIASED: refusers were sicker -> overstates benefit per-protocol (compliers vs ctrl) -5.15 BIASED: same selection problem CACE = ITT_Y/ITT_D (Bloom) -3.23 valid effect AMONG COMPLIERS CACE via 2SLS (Z instruments D) -3.23 identical -- noncompliant RCT = valid instrument Among compliers, vitamin A cut mortality by ~51% relative to the control rate. Why as-treated misleads: non-takers' mortality 14.06/1,000 >> takers' 1.24/1,000 -- selection, not the vitamin.
ITT and CACE answer DIFFERENT valid questions (launch-the-program vs take-the-vitamin); as-treated/per-protocol answer neither.
3. Cluster designs — Project STAR and the correlated-outcomes problem¶
In Project STAR (Tennessee, 1985–89), students and teachers were randomized to small vs regular kindergarten classes. Because a class shares a teacher, a room, and peers, the treatment is effectively assigned at the classroom/school level, and reading scores are correlated within school — children in the same school resemble each other far more than children in different schools. That within-cluster correlation (the intraclass correlation, ICC) means each additional child in an already-sampled school adds less than one child's worth of new information.
Ignoring it makes the standard error far too small. We estimate the small-class effect on kindergarten reading with (a) a naive heteroskedasticity-robust SE and (b) a cluster-robust SE grouped by school. The point estimate is the same; the honest SE is nearly twice as large.
star=pd.read_csv("star_k.csv")
print(f"Project STAR kindergarten: {len(star):,} students in {star.school.nunique()} schools; {int(star.small.sum())} small-class, {int((1-star.small).sum())} regular")
m=smf.ols("read~small",star).fit(cov_type="HC1")
mc=smf.ols("read~small",star).fit(cov_type="cluster",cov_kwds={"groups":star.school})
# crude ICC of reading across schools
grp=star.groupby("school")["read"]; icc=grp.mean().var()/star["read"].var()
print(f"\n small-class effect on kindergarten reading = {m.params['small']:+.2f} points")
print(f" naive (HC1) SE = {m.bse['small']:.3f} 95% CI [{m.conf_int().loc['small',0]:.2f}, {m.conf_int().loc['small',1]:.2f}]")
print(f" cluster-robust SE = {mc.bse['small']:.3f} 95% CI [{mc.conf_int().loc['small',0]:.2f}, {mc.conf_int().loc['small',1]:.2f}] ({mc.bse['small']/m.bse['small']:.2f}x wider)")
print(f" between-school variance share of reading (ICC proxy) = {icc:.3f} -> strong clustering")
fig,ax=plt.subplots(1,2,figsize=(13,4.4))
est=m.params['small']
ax[0].errorbar([0],[est],yerr=[1.96*m.bse['small']],fmt="o",color=GREY,capsize=6,label="naive HC1 SE")
ax[0].errorbar([1],[est],yerr=[1.96*mc.bse['small']],fmt="o",color=GREEN,capsize=6,label="cluster-robust SE")
ax[0].axhline(0,color="k",lw=.6); ax[0].set_xlim(-.5,1.5); ax[0].set_xticks([0,1]); ax[0].set_xticklabels(["naive","clustered"])
ax[0].set_ylabel("small-class effect (95% CI)"); ax[0].set_title(f"Same estimate {est:+.1f}, honest CI ~{mc.bse['small']/m.bse['small']:.1f}x wider"); ax[0].legend(fontsize=8)
sm=star.groupby("school").agg(read=("read","mean"),small_share=("small","mean"),n=("read","size")).reset_index()
ax[1].scatter(sm.index,sm.read,s=18,color=BLUE); ax[1].axhline(star.read.mean(),color=RED,lw=1.5,ls="--",label="grand mean")
ax[1].set_xlabel("school"); ax[1].set_ylabel("mean reading score"); ax[1].set_title(f"School means vary widely (ICC~{icc:.2f}) -> outcomes cluster"); ax[1].legend(fontsize=8)
plt.tight_layout(); plt.show()
print("The naive SE pretends 3,700 independent students; clustering says the real information is closer to '79 schools'.")
Project STAR kindergarten: 3,743 students in 79 schools; 1738 small-class, 2005 regular small-class effect on kindergarten reading = +5.82 points naive (HC1) SE = 1.042 95% CI [3.78, 7.86] cluster-robust SE = 1.850 95% CI [2.19, 9.44] (1.78x wider) between-school variance share of reading (ICC proxy) = 0.225 -> strong clustering
The naive SE pretends 3,700 independent students; clustering says the real information is closer to '79 schools'.
4. Why it matters — coverage under a cluster-randomized design (known truth)¶
Real data shows the SE changes, but only a simulation with a known effect can show that the naive SE is wrong — that its confidence intervals fail to cover the truth. We simulate a cluster-randomized experiment (whole clusters assigned to treatment) with a true effect of zero, $G=30$ clusters of $m=40$, and ICC $=0.15$, and check how often each 95% interval actually contains 0.
The naive intervals cover far below 95% — false precision — while cluster-robust intervals are calibrated. The gap is governed by the design effect $1+(m-1)\,\text{ICC}$: here $1+39(0.15)\approx6.8$, so honest SEs are about $\sqrt{6.8}\approx2.6$ times larger, and the effective sample size is the total divided by the design effect. The rule is blunt: randomize clusters, analyze clusters.
def one(G=30,m=40,icc=0.15,tau=0.0,seed=0):
r=np.random.default_rng(seed); su=np.sqrt(icc); se=np.sqrt(1-icc)
tg=r.permutation(np.r_[np.ones(G//2),np.zeros(G-G//2)]); rows=[]
for g in range(G):
a=r.normal(0,su); y=tau*tg[g]+a+r.normal(0,se,m)
rows += [(g,tg[g],yi) for yi in y]
d=pd.DataFrame(rows,columns=["cl","T","y"])
nb=smf.ols("y~T",d).fit(); cb=smf.ols("y~T",d).fit(cov_type="cluster",cov_kwds={"groups":d.cl})
return nb.params['T'], nb.bse['T'], cb.bse['T']
R=np.array([one(seed=s) for s in range(500)])
cov_n=np.mean(np.abs(R[:,0])<1.96*R[:,1]); cov_c=np.mean(np.abs(R[:,0])<1.96*R[:,2])
deff=1+(40-1)*0.15
print(f"Cluster-randomized, true effect = 0, 500 simulations (G=30 clusters x m=40, ICC=0.15):")
print(f" naive-SE 95% CI coverage = {100*cov_n:.0f}% <- should be 95%: FALSE precision")
print(f" cluster-SE 95% CI coverage = {100*cov_c:.0f}% <- calibrated")
print(f" mean SE: naive {R[:,1].mean():.3f} vs clustered {R[:,2].mean():.3f} ({R[:,2].mean()/R[:,1].mean():.2f}x)")
print(f" design effect 1+(m-1)ICC = {deff:.1f}; sqrt = {np.sqrt(deff):.2f} (matches the SE ratio)")
fig,ax=plt.subplots(1,2,figsize=(13,4.4))
ax[0].bar(["naive SE","cluster SE"],[100*cov_n,100*cov_c],color=[RED,GREEN]); ax[0].axhline(95,color="k",ls="--",lw=1)
ax[0].set_ylabel("actual 95% CI coverage (%)"); ax[0].set_title("Naive SEs badly under-cover under clustering")
for i,v in enumerate([100*cov_n,100*cov_c]): ax[0].text(i,v-6,f"{v:.0f}%",ha="center",color="white",fontweight="bold")
iccg=np.linspace(0,0.4,50); ax[1].plot(iccg,np.sqrt(1+(40-1)*iccg),color=PURP,lw=2)
ax[1].scatter([0.15],[np.sqrt(deff)],color=RED,s=80,zorder=3); ax[1].set_xlabel("ICC"); ax[1].set_ylabel("SE inflation = sqrt(design effect)")
ax[1].set_title("Design effect grows with ICC and cluster size (m=40)")
plt.tight_layout(); plt.show()
print("Effective n = total / design effect. With m=40 and ICC=0.15, 1,200 individuals carry the information of ~175.")
Cluster-randomized, true effect = 0, 500 simulations (G=30 clusters x m=40, ICC=0.15): naive-SE 95% CI coverage = 54% <- should be 95%: FALSE precision cluster-SE 95% CI coverage = 94% <- calibrated mean SE: naive 0.057 vs clustered 0.146 (2.55x) design effect 1+(m-1)ICC = 6.8; sqrt = 2.62 (matches the SE ratio)
Effective n = total / design effect. With m=40 and ICC=0.15, 1,200 individuals carry the information of ~175.
5. Summary¶
Two features of real experiments break the naive difference in means, and each has a principled fix:
- Noncompliance (vitamin A trial): assignment ≠ receipt. The ITT (−2.6 per 1,000) is the valid, assumption-light effect of launching the program, but is diluted by the 20% who never took the vitamin. Comparing children by what they took (as-treated, −6.5 per 1,000) is biased — refusers were sicker — and exaggerates the benefit. The CACE = ITT_Y/ITT_D = 2SLS (−3.2 per 1,000, a ~50% reduction among compliers) is the valid effect of the vitamin itself, with assignment serving as a textbook instrument for receipt.
- Cluster designs (Project STAR): when treatment is effectively assigned to groups and outcomes are correlated within them, standard errors that ignore clustering are drastically overconfident — STAR's honest CI was ~1.8× wider, and a known-truth simulation showed naive 95% intervals covering only ~54% of the time. The design effect $1+(m-1)\text{ICC}$ sets the penalty; cluster-robust inference restores calibration.
Guidance: report ITT as the primary estimand and CACE/IV as the per-protocol complement (never as-treated); and always cluster standard errors at the level of randomization. Cross-links: CACE is the instrumental-variables/LATE estimator (subsection 3) with randomized assignment as the instrument — the cleanest possible instrument; cluster-robust SEs recur in panel data (subsection 5, Bertrand-Duflo-Mullainathan) and in purged/blocked cross-validation (Financial ML); the selection that biases as-treated is the same confounding that matching (subsection 2) confronts in observational data. The R companion runs both experiments through estimatr (iv_robust for CACE, difference_in_means(clusters=) for cluster-robust inference).