Partial Pooling Across Subgroups

Four Effects, and Two Bad Options

The Electric Company experiment was randomized in matched pairs within grade, so the covariate-adjustment page reports it grade by grade. Read literally, those four numbers say the programme helps first-graders five times as much as fourth-graders. But the largest estimate is also the least precise — grade 1 at 8.79 carries a standard error of 2.61, against grade 4's 1.70 at 0.69. That combination, biggest effect and widest interval, is the signature of a number that owes something to luck and will regress on replication.

The frequentist analysis has two options and neither is satisfactory. Fit each grade separately — no pooling — and grade 1's noisy 8.79 stands as a real quantity. Fit one common effect — complete pooling, which gives 4.73 — and you have asserted the grades are identical, which nothing in the design guarantees. There is no principled middle, because choosing one would require knowing how much the grades actually differ, which is precisely what is unknown.

What the Shrinkage Actually Does

A hierarchical model makes that unknown a parameter. The grade effects are drawn from a common distribution whose spread $\sigma_\tau$ is estimated alongside them. If $\sigma_\tau \to 0$ the model becomes complete pooling; if $\sigma_\tau \to \infty$ it becomes no pooling; anything between produces shrinkage in an amount the data choose rather than the analyst.

How the shrinkage actually works is more interesting than the usual summary of it. It is often described as "the noisiest estimate moves most", and that is not right. Shrinkage is imprecision times distance from the common mean, and the two can pull against each other — visible here in a way four groups make unusually clear.

Grade 2 barely moves (+0.19) despite a middling standard error, because it sits almost exactly at the common mean already. Grade 4 is the most precise estimate in the set and still gets pulled nearly a full point, because it is far below the mean. And note the direction: grades 3 and 4 move up. Shrinkage is toward the common mean, not toward zero — a distinction that matters whenever subgroup estimates straddle the average, and one that "shrinkage" as a word actively obscures.

Is the Variation Across Grades Real?

The second deliverable is the question the frequentist version cannot pose. Comparing four point estimates establishes that they differ; it cannot say whether they differ by more than sampling noise would produce if the underlying effects were identical. The posterior for $\sigma_\tau$ answers that directly, and the answer here is honest uncertainty rather than a clean verdict: median 2.41 with a 95% interval of [0.13, 7.44], and $\Pr(\sigma_\tau < 1) = 0.19$. Four grades are simply not enough to pin down how much grades vary. That is worth reporting as a result in its own right, because the alternative — presenting four separate subgroup effects as though each were established — implies a precision the design never had.

A consequence worth being explicit about: the hierarchical estimate of the average effect, 3.86 with 95% interval [−0.24, 7.53], is wider than complete pooling's 4.73 (SE 1.16). That is not a loss. Complete pooling buys its precision by assuming the grades share one effect; the hierarchical model declines to assume it and pays by propagating $\sigma_\tau$'s uncertainty into the average. The pooled interval is narrower because it is answering an easier question that nobody established the right to ask.

Finally the subgroup comparison that started the page. The unpooled gap between grade 1 and grade 4 was +7.09 points and looked decisive. The posterior for that difference is +3.49 with a 95% interval of [−0.86, +10.02] and $\Pr(\text{grade 1} > \text{grade 4}) = 0.888$. Better than a coin flip, well short of established — and a fair description of what four dozen classrooms per grade can support. This is the mechanism behind a familiar disappointment: subgroup effects that look striking in one trial and shrink on replication were usually never as separated as the unpooled estimates implied.

Where this sits

The general lesson is that partial pooling is the right default whenever an experiment is blocked or stratified, which describes most field experiments. The frequentist page had to choose between two extremes; the hierarchical model treats that choice as a parameter and reports how confidently the data settle it. This is the hierarchical machinery of the Bayesian arc applied to a causal estimand rather than a descriptive one. The Project STAR example takes the same idea from four grades to seventy-nine schools, where the grouping is not a design feature but the thing that broke the standard errors in the first place.

Notebook

Downloads

References

• Efron, B. & Morris, C. (1975). Data Analysis Using Stein's Estimator and its Generalizations. Journal of the American Statistical Association 70(350), 311–319. — Stein estimation as partial pooling
• Rubin, D. B. (1981). Estimation in Parallel Randomized Experiments. Journal of Educational Statistics 6(4), 377. — estimation in parallel experiments
• Gelman, A. (2006). Conservative prior distributions for variance parameters in hierarchical models. Canadian Journal of Statistics 34(3), 377–390. — the variance prior that controls the pooling