The Large-Sample Limit

The Case Where the Prior Does Nothing

This group is arranged along one spine: how much the prior can actually move the answer. It opened on fifteen matched pairs, where the prior decides the verdict outright and Darwin’s result does not survive a second analysis. It ends here, on 305,866 voters, where the prior can be shown to be irrelevant rather than asserted to be.

The example is deliberate. A group about what Bayesian methods contribute is not honest unless it contains the case where they contribute nothing, and Gerber, Green and Larimer’s (2008) social-pressure mailing is large enough to settle it. The Neighbors treatment listed a household’s own turnout record and that of its neighbours, with a promise to send an updated list after the election, and it produced one of the largest turnout effects ever measured: 29.7% to 37.8%, an effect of 8.13 percentage points with a standard error of 0.27 on 229,444 voters. That is a t-statistic of 30.

The model is a logistic regression on the two cell counts — every variable here is discrete, so the experiment aggregates without losing anything. The treatment effect carries a $\mathcal{N}(0, s)$ prior, and the sweep drives $s$ from vague down to values nobody would defend. To keep each prior’s strength interpretable, $s$ is expressed as a multiple $k$ of the observed effect: $k=1$ is a sceptic who centres on no effect but allows one the size of the effect actually found; $k=0.01$ is a fanatic.

Effects in percentage points. The frequentist point estimate is 8.1310 for comparison.

The sceptic at $k=1$ moves the posterior by 0.0099 percentage points — about a tenth of one percent of the estimate. Scepticism at that level is simply not registered. The prior does win eventually, as it must: the shift first reaches a tenth of a point at $k=0.1$, and by $k=0.01$ the posterior has been dragged from 8.13 down to 0.68. But $k=0.01$ is a claim, made before seeing anything, that the effect is about one percent of what 229,444 voters went on to show. It takes exactly that to move the answer.

The Law Behind It, and a Correction

The reason is exact rather than empirical. For a normal likelihood with estimate $\hat\tau$ and standard error ${\rm SE}$, a $\mathcal{N}(0,s)$ prior drags the posterior mean toward zero by a fraction ${\rm SE}^2/(s^2+{\rm SE}^2)$. Setting $s = k|\hat\tau|$ and writing $t = \hat\tau/{\rm SE}$, everything cancels:

Sample size does not appear. What governs how much a prior can move an answer is the t-statistic. The notebook checks the law against the MCMC sweep rather than asserting it, and it tracks across four orders of magnitude in $k$; the largest discrepancy is 1.27%, at the dogmatic end, and it is not sampling noise. The law is exact on the scale the prior sits on (log-odds) while the shift is reported on the probability scale, and the map between them is not linear.

That slightly corrects this group’s own framing. These pages are ordered by sample size as a stand-in for information, and within this very arc the stand-in fails.

Fraction of the estimate a prior centred at zero removes, for three prior strengths.

The vitamin A CACE rests on 23,682 children at $t = 2.79$; the Electric Company grade-1 effect rests on 192 at $t = 3.37$. The experiment with 123 times the sample is the more prior-sensitive of the two. A ratio estimator spends much of its sample determining a denominator, while a blocked design with a strong baseline covariate extracts far more per observation than a raw comparison does. What a prior competes against is the precision actually achieved, which is what $t$ measures and $n$ does not. Project STAR happens to fall between the two on both counts — the coincidence that lets sample size pass for information most of the time.

Where the Prior Starts Working Again — on These Same Voters

The result above is a statement about the question, not the dataset. “229,444 voters” is the sample size of the headline. Ask something more specific and that number falls apart: turnout effects depend on household size, since the Neighbors mailing exposes a voter to more scrutiny in a larger household, and past voting is the strongest single predictor of turning out at all.

Splitting these same voters by household size and age band gives twenty cells whose treated arms run from 10,184 voters down to 14. The widest unpooled standard error is 13.55 points against the headline’s 0.27 — a factor of fifty. In the sparse cells the unpooled estimates are nonsense, and partial pooling moves them decisively.

Effects in percentage points; the four sparsest and three densest of twenty cells. Maximum r-hat across the model was 1.000.

The five smallest cells move by 10.96 points on average and have their standard errors cut from 7.43 to 1.31. The five largest move by 0.20. The machinery that could not touch the headline is decisive two rows down.

Two qualifications belong with that. The information here is borrowed from the other cells rather than supplied from outside, so this is a hierarchical prior rather than the kind swept above — the same mechanism, differently sourced. And the pooled standard errors are as small as they are because the model concludes the effect varies by only about 1 percentage point across cells and therefore pools hard. That is a conclusion drawn from the data rather than an assumption, but if it is wrong these intervals are overconfident, exactly as Project STAR’s varying-intercept model was.

The through-line for the whole group is that “how much does the prior matter” is not a question about Bayesian methods at all. It is a question about how much the data have already determined, and the t-statistic answers it exactly. At $t=30$ the choice of framework is a matter of interpretation and nothing else, since the posterior and the confidence interval agree to three decimal places. At $t=2$ it decides the verdict, which is where Darwin’s fifteen pairs came in. Everything between is a matter of degree, and the degree is calculable in advance.

Where this sits

The same experiment appears in covariate adjustment as the low-$\rho^2$ extreme, where a weak baseline covariate buys only 3% variance reduction. That is the mirror image of the point made here: both are statements about how much information the design has already extracted. The pooling in the last section is the machinery of the Electric Company example and Project STAR, applied where the cells are finally small enough for it to matter.

Notebook

Downloads

References

• Rubin, D. B. (1978). Bayesian Inference for Causal Effects: The Role of Randomization. The Annals of Statistics 6(1). — the Bayesian formulation
• Rubin, D. B. (1981). Estimation in Parallel Randomized Experiments. Journal of Educational Statistics 6(4), 377. — hierarchical estimation across experiments
• Efron, B. & Morris, C. (1975). Data Analysis Using Stein's Estimator and its Generalizations. Journal of the American Statistical Association 70(350), 311–319. — the frequentist reading of the same shrinkage