Noncompliance and Cluster Designs

When Assignment Is Not Receipt

Randomization identifies the effect, but two features of real field and clinical trials get between the design and the simple difference in means. Subjects assigned to a treatment do not always take it, so assignment and receipt come apart and there are suddenly two different things one could mean by "the effect". And treatment is often assigned to whole groups — schools, villages, households — rather than individuals, so outcomes within a group are correlated and the row count badly overstates how much independent information there is. Both are grounded here on real experiments rather than described.

Sommer and Zeger (1991) studied whether vitamin A supplementation reduces child mortality in Indonesia: 23,682 children, 12,094 assigned the programme and 11,588 control. Of those assigned, about 80% actually received it. Controls had no access at all, which makes this the canonical one-sided noncompliance design — there are compliers and never-takers, but no always-takers. Mortality ran 3.80 per 1,000 in the vitamin arm against 6.39 in control.

The intention-to-treat effect compares everyone as assigned, ignoring what they took: −2.58 per 1,000, a 40% reduction. It needs no assumption beyond randomization, and it answers a real question — what happens if we launch the programme. It is also, unavoidably, diluted: a fifth of the treatment arm never took the vitamin, so ITT understates what the vitamin itself does.

Four Estimates, Two of Them Meaningless

The tempting repair is to compare children by what they actually took. This throws randomization away, and the notebook shows exactly how much damage that does. The 20% who refused were not a random subset — they were harder to reach, poorer and sicker. Their mortality was 14.06 per 1,000 against 1.24 among takers, a factor of eleven. Almost none of that gap is the vitamin; nearly all of it is who takes vitamins. So the as-treated comparison returns −6.47 and per-protocol −5.15, both overstating the benefit by a wide margin, and both answering no well-posed question at all.

The valid target for "what does the vitamin do" is the complier average causal effect, and with one-sided noncompliance Bloom's estimator gives it exactly: divide the effect of assignment on the outcome by the effect of assignment on take-up. That yields −3.23 per 1,000 — about a 51% reduction relative to the control rate, among compliers. Estimating the same thing by two-stage least squares, using assignment as an instrument for receipt, returns −3.23 as well: identical, because a noncompliant randomized trial is the textbook valid instrument. Assignment is randomized, it moves take-up, and it can only affect mortality through the vitamin.

The important thing is that ITT and CACE are not competing estimates of one quantity. They are correct answers to two different questions — launch the programme versus take the vitamin — and which one a decision needs depends on whether the decision is a policy or a prescription. As-treated and per-protocol are not a third option; they answer neither.

When the Unit Is Not the Individual

The second problem is structural rather than behavioural. In Project STAR (Tennessee, 1985–89), 3,743 kindergarten students across 79 schools were randomized to small or regular classes. A class shares a teacher, a room and a peer group, so the treatment is effectively assigned at the classroom level and reading scores are correlated within school — the intraclass correlation. Here the between-school share of variance is 0.225, which is substantial. Each additional child in an already-sampled school therefore contributes rather less than one child's worth of new information.

The point estimate is unaffected: the small-class effect is +5.82 reading points either way. What changes is the honesty of the interval. A heteroskedasticity-robust standard error gives 1.042 and a 95% interval of [3.78, 7.86]; clustering by school gives 1.850 and [2.19, 9.44], nearly 1.8× wider. The naive version is quietly pretending there are 3,700 independent students when the design says the real unit of information is closer to 79 schools.

Which standard error is right?

Real data can show the standard error changing, but it cannot show which one is right, because the true effect is unknown. Only a simulation can, and this is where the section earns its argument. Assign whole clusters at random with a true effect of exactly zero, 30 clusters of 40, intraclass correlation 0.15, and count how often each 95% interval contains the truth over 500 runs.

Naive intervals cover 54% of the time. A nominal 95% interval that is right barely half the time is not a slightly optimistic interval; it is a broken one, and every significance test built on it is inflated in the same proportion. Cluster-robust intervals cover 94%. The size of the penalty is not mysterious either — the design effect $1+(m-1)\,\text{ICC}$ is $1 + 39(0.15) \approx 6.8$ here, so honest standard errors should be about $\sqrt{6.8} \approx 2.62$ times larger, and the measured ratio is 2.55. Put in sample-size terms: 1,200 individuals carry the information of roughly 175.

Where this sits

Both fixes point outward. The CACE is the instrumental-variables estimator, with randomized assignment as the cleanest instrument that exists — which is why this notebook is the natural bridge from designed experiments to the instrument-based identification strategies later in the arc, where the instrument is found rather than assigned and every one of its assumptions has to be argued for. Cluster-robust inference recurs wherever observations arrive in correlated blocks, including purged and embargoed cross-validation, where the correlated block is a stretch of time rather than a school. And the selection that makes as-treated biased here is exactly the confounding that matching confronts in observational data, without the benefit of a randomized assignment to fall back on. The precision layer from covariate adjustment applies on top of all of this.

Notebooks

Downloads

The Sommer–Zeger vitamin A trial is entered directly from the published cell counts rather than shipped as a file — the whole trial is four numbers, which is part of why it has served as the canonical noncompliance example for thirty years.

References

• Angrist, J. D., Imbens, G. W. & Rubin, D. B. (1996). Identification of Causal Effects Using Instrumental Variables: Comment. Journal of the American Statistical Association 91(434), 465. — instrumental variables in the potential-outcomes framework
• Bloom, H. S. (1984). Accounting for No-Shows in Experimental Evaluation Designs. Evaluation Review 8(2), 225–246. — the no-show adjustment
• Frangakis, C. E. & Rubin, D. B. (2002). Principal Stratification in Causal Inference. Biometrics 58(1), 21–29. — principal stratification