Causal Inference II(e) — The Large-Sample Limit¶
Social Pressure GOTV: the case where the prior provably does nothing, and what that says about the other four¶
This group is arranged along one spine: how much the prior can actually move the answer. It opened on fifteen matched pairs, where the prior decides the verdict outright and Darwin's result does not survive a second analysis. It ends here, on 305,866 voters, where the prior can be shown — not asserted — to be irrelevant.
The example is deliberate. A group about what Bayesian methods contribute is not honest unless it contains the case where they contribute nothing, and this is a large enough experiment to make that case airtight. Gerber, Green and Larimer's (2008) social-pressure mailing is one of the largest field experiments in political science, and its Neighbors-versus-Control comparison rests on 229,444 voters. Against that, no defensible prior survives contact with the likelihood.
Three things follow, and only the first is obvious:
- The prior does nothing here, across a sweep running from vague to dogmatic.
- There is an exact law for when that happens, and it turns out not to depend on sample size directly. It depends on the ***t*-statistic**, which slightly corrects this group's own organising claim.
- The claim is about the question, not the dataset. Ask a subgroup question of these same 305,866 voters and the per-cell sample size collapses, at which point the prior starts doing real work again — on the very data just used to prove it could not.
Python/PyMC lead. Fifth and last in the Bayesian causal group.
1. The experiment, and the headline¶
Households were randomly assigned to a control group or to one of several mailings. The strongest, Neighbors, listed the recipient's own turnout record and that of their neighbours, with a promise to send an updated list after the election. It is social pressure applied about as bluntly as an ethics board will permit, and it produced one of the largest turnout effects ever measured.
Because every variable here is discrete, the whole dataset aggregates to a handful of cell counts without losing a thing. That makes exact Bayesian computation trivial on a dataset of this size — the likelihood for a binomial depends on the data only through the counts.
import numpy as np, pandas as pd, matplotlib.pyplot as plt, warnings, logging, contextlib, io
warnings.filterwarnings("ignore")
import pymc as pm, arviz as az, statsmodels.formula.api as smf
logging.getLogger("pymc").setLevel(logging.ERROR)
BLUE="#2b6cb0"; RED="#c53030"; GREEN="#2f855a"; ORANGE="#dd6b20"; GREY="#a0aec0"; PURP="#6b46c1"
sp = pd.read_csv("social.csv")
print(f"full experiment: {len(sp):,} voters across {sp.messages.nunique()} arms")
print(sp.messages.value_counts().to_string())
d = sp[sp.messages.isin(["Control","Neighbors"])].copy()
d["treat"] = (d.messages=="Neighbors").astype(int)
n_c, n_t = int((d.treat==0).sum()), int((d.treat==1).sum())
y_c, y_t = int(d.loc[d.treat==0,"primary2006"].sum()), int(d.loc[d.treat==1,"primary2006"].sum())
p_c, p_t = y_c/n_c, y_t/n_t
ols = smf.ols("primary2006 ~ treat", data=d).fit(cov_type="HC1")
tau_hat, se_hat = ols.params["treat"], ols.bse["treat"]
print(f"\nNeighbors vs Control: {n_c+n_t:,} voters ({n_c:,} control, {n_t:,} treated)")
print(f" turnout: control {p_c:.4f} Neighbors {p_t:.4f}")
print(f" effect = {tau_hat:+.4f} ({100*tau_hat:.1f} percentage points), SE {se_hat:.4f}, t = {tau_hat/se_hat:.1f}")
print(f" the four numbers the likelihood actually depends on: {y_c:,}/{n_c:,} and {y_t:,}/{n_t:,}")
g++ not available, if using conda: `conda install gxx`
full experiment: 305,866 voters across 4 arms messages Control 191243 Civic Duty 38218 Hawthorne 38204 Neighbors 38201 Neighbors vs Control: 229,444 voters (191,243 control, 38,201 treated) turnout: control 0.2966 Neighbors 0.3779 effect = +0.0813 (8.1 percentage points), SE 0.0027, t = 30.2 the four numbers the likelihood actually depends on: 56,730/191,243 and 14,438/38,201
2. The prior sweep¶
The model is a logistic regression on the two cell counts: a control log-odds $\alpha$, and a treatment log-odds effect $\beta$ carrying a $\mathcal{N}(0, s)$ prior. The sweep drives $s$ from vague down to values that would be absurd to defend — a prior insisting the effect is a hundredth of what was observed — and asks what the posterior does about it.
To keep the strength of each prior interpretable, $s$ is expressed as a multiple of the observed effect $\hat\beta$ on the log-odds scale. So $k=1$ is a sceptic who centres on no effect but allows one the size of the one actually found; $k = 0.01$ is a fanatic.
lg = smf.logit("primary2006 ~ treat", data=d).fit(disp=0)
beta_hat_lo, se_lo = lg.params["treat"], lg.bse["treat"] # effect and SE, log-odds scale
print(f"observed effect on the log-odds scale: beta_hat = {beta_hat_lo:.4f}, "
f"SE {se_lo:.4f}, t = {beta_hat_lo/se_lo:.1f}")
def fit_gotv(s, seed=0):
with pm.Model():
a = pm.Normal("a", 0, 1.5)
b = pm.Normal("b", 0, s)
pm.Binomial("obs_c", n=n_c, p=pm.math.invlogit(a), observed=y_c)
pm.Binomial("obs_t", n=n_t, p=pm.math.invlogit(a + b), observed=y_t)
pm.Deterministic("pp", (pm.math.invlogit(a+b) - pm.math.invlogit(a))*100)
with contextlib.redirect_stderr(io.StringIO()):
it = pm.sample(2000, tune=1000, chains=4, cores=1, progressbar=False,
random_seed=seed, target_accept=0.9)
return it
KS = [10.0, 3.0, 1.0, 0.3, 0.1, 0.03, 0.01]
print(f" {'prior sd':>26} {'posterior effect (pp)':>22} {'95% interval':>20} {'shift':>8}")
rows = []
for k in KS:
s = k*abs(beta_hat_lo)
pp = fit_gotv(s, seed=int(k*100)).posterior["pp"].values.ravel()
rows.append((k, s, pp.mean(), np.percentile(pp,2.5), np.percentile(pp,97.5)))
lbl = f"{k:g} x beta_hat (s={s:.4f})"
print(f" {lbl:>26} {pp.mean():>22.4f} {'[%.3f, %.3f]'%(rows[-1][3],rows[-1][4]):>20} "
f"{pp.mean()-rows[0][2]:>+8.4f}")
print(f"\nfrequentist point estimate for comparison: {100*tau_hat:.4f} pp")
print()
base = rows[0][2]
k_tenth = max([k for k,_,m,_,_ in rows if abs(m-base) >= 0.1]) # weakest prior that shifts it 0.1pp
print(f"A prior centred at NO EFFECT whose standard deviation equals the whole observed effect (k=1) moves")
print(f"the posterior by {abs(rows[2][2]-base):.4f} percentage points -- {100*abs(rows[2][2]-base)/base:.2f}% of the estimate. Scepticism at that level")
print("is simply not registered.")
print()
print(f"The prior does eventually win, as it must. Walking k down, the shift first reaches a tenth of a")
print(f"percentage point at k = {k_tenth:g}, and by k = {rows[-1][0]:g} the posterior has been dragged to {rows[-1][2]:.2f}pp.")
print()
print(f"But look at what those priors assert. k = {rows[-1][0]:g} is a claim, made before seeing anything, that the")
print(f"effect is about {100*rows[-1][0]:g}% of what 229,444 voters went on to show. That is not scepticism, it is a")
print("refusal to look -- and it takes exactly that to move the answer.")
observed effect on the log-odds scale: beta_hat = 0.3651, SE 0.0117, t = 31.3
prior sd posterior effect (pp) 95% interval shift
10 x beta_hat (s=3.6509) 8.1307 [7.602, 8.640] +0.0000
3 x beta_hat (s=1.0953) 8.1333 [7.608, 8.675] +0.0026
1 x beta_hat (s=0.3651) 8.1208 [7.579, 8.663] -0.0099
0.3 x beta_hat (s=0.1095) 8.0332 [7.506, 8.557] -0.0975
0.1 x beta_hat (s=0.0365) 7.3532 [6.841, 7.843] -0.7775
0.03 x beta_hat (s=0.0110) 3.7012 [3.349, 4.054] -4.4295
0.01 x beta_hat (s=0.0037) 0.6805 [0.534, 0.827] -7.4503 frequentist point estimate for comparison: 8.1310 pp A prior centred at NO EFFECT whose standard deviation equals the whole observed effect (k=1) moves the posterior by 0.0099 percentage points -- 0.12% of the estimate. Scepticism at that level is simply not registered. The prior does eventually win, as it must. Walking k down, the shift first reaches a tenth of a percentage point at k = 0.1, and by k = 0.01 the posterior has been dragged to 0.68pp. But look at what those priors assert. k = 0.01 is a claim, made before seeing anything, that the effect is about 1% of what 229,444 voters went on to show. That is not scepticism, it is a refusal to look -- and it takes exactly that to move the answer.
3. The law behind it, and a correction to this group's own framing¶
The reason is exact rather than empirical. For a normal likelihood with estimate $\hat\tau$ and standard error ${\rm SE}$, and a prior $\mathcal{N}(0, s)$, the posterior mean is a precision-weighted average of the two, and the fraction of the way it is dragged from $\hat\tau$ toward zero is
$$\text{prior influence} \;=\; \frac{{\rm SE}^2}{s^2+{\rm SE}^2}.$$
Setting $s = k\,|\hat\tau|$ to make the prior's strength comparable across experiments on different scales, and writing $t = \hat\tau/{\rm SE}$, everything cancels:
$$\text{prior influence} \;=\; \frac{1}{k^2t^2+1}.$$
Sample size does not appear. What governs how much a prior can move an answer is the ***t*-statistic** — and while $t$ usually grows with $n$, it is not the same thing, which matters because this group has been ordering its examples by sample size. The law says the ordering is really by $t$. The two agree here, but they need not.
# vitamin A: the Bloom standard error, SE(ITT_Y)/ITT_D, from the published cell counts
pv_t, pv_c = 46/12094, 74/11588
se_itt = 1000*np.sqrt(pv_t*(1-pv_t)/12094 + pv_c*(1-pv_c)/11588)
se_cace = se_itt/0.80
print(f"vitamin A: SE(ITT_Y) = {se_itt:.4f} per 1,000, so SE(CACE) = {se_itt:.4f}/0.80 = {se_cace:.3f}\n")
# estimate and standard error as each page in this arc presents them
EXP = [("Darwin's maize, 15 pairs", 2.6167, 1.2182, 15, "/projects/cib-smallsample"),
("Electric Company, grade 1", 8.79, 2.61, 192, "/projects/cib-partialpooling"),
("Project STAR, cluster-robust", 5.82, 1.850, 3743, "/projects/cib-hierarchical"),
("Vitamin A, CACE", -3.23, se_cace, 23682, "/projects/cib-principalstrat"),
("Social Pressure, Neighbors", 100*tau_hat, 100*se_hat, n_c+n_t, "")]
infl = lambda k, t: 1.0/(k*k*t*t + 1.0)
print(f" {'experiment':32} {'n':>8} {'t':>7} | {'k=1':>8} {'k=0.3':>8} {'k=0.1':>8}")
for nm, e, s_, n_, _ in EXP:
t = abs(e/s_)
print(f" {nm:32} {n_:>8,} {t:>7.2f} | {infl(1,t):>7.1%} {infl(0.3,t):>8.1%} {infl(0.1,t):>8.1%}")
print()
print("Read the k=1 column: a sceptic who allows an effect the size of the one observed moves Darwin's")
print(f"estimate by {infl(1,2.6167/1.2182):.0%} and the GOTV estimate by {infl(1,abs(tau_hat/se_hat)):.2%}. Even the fanatic at k=0.1 -- asserting the")
print(f"effect is a tenth of what was seen -- shifts GOTV by only {infl(0.1,abs(tau_hat/se_hat)):.1%}, while it would rewrite Darwin.")
print()
print("Now note where the ordering by t comes apart from the ordering by n. The vitamin A CACE rests on")
print(f"23,682 children and has t = {abs(3.23/se_cace):.2f}; the Electric Company grade-1 effect rests on 192 and has")
print(f"t = {abs(8.79/2.61):.2f}. The experiment with {23682/192:.0f} times the sample is the MORE prior-sensitive of the two.")
print("Project STAR happens to fall between the two on both counts, which is the coincidence that lets")
print("sample size pass for information most of the time.")
print()
print("Sample size is a proxy for information, and these are the cases where the proxy fails. A ratio")
print("estimator spends much of its sample determining a denominator; a blocked design with a strong")
print("baseline covariate extracts far more per observation than a raw comparison does. What the prior")
print("competes against is the precision actually achieved, which is what t measures and n does not.")
vitamin A: SE(ITT_Y) = 0.9278 per 1,000, so SE(CACE) = 0.9278/0.80 = 1.160 experiment n t | k=1 k=0.3 k=0.1 Darwin's maize, 15 pairs 15 2.15 | 17.8% 70.7% 95.6% Electric Company, grade 1 192 3.37 | 8.1% 49.5% 89.8% Project STAR, cluster-robust 3,743 3.15 | 9.2% 52.9% 91.0% Vitamin A, CACE 23,682 2.79 | 11.4% 58.9% 92.8% Social Pressure, Neighbors 229,444 30.21 | 0.1% 1.2% 9.9% Read the k=1 column: a sceptic who allows an effect the size of the one observed moves Darwin's estimate by 18% and the GOTV estimate by 0.11%. Even the fanatic at k=0.1 -- asserting the effect is a tenth of what was seen -- shifts GOTV by only 9.9%, while it would rewrite Darwin. Now note where the ordering by t comes apart from the ordering by n. The vitamin A CACE rests on 23,682 children and has t = 2.79; the Electric Company grade-1 effect rests on 192 and has t = 3.37. The experiment with 123 times the sample is the MORE prior-sensitive of the two. Project STAR happens to fall between the two on both counts, which is the coincidence that lets sample size pass for information most of the time. Sample size is a proxy for information, and these are the cases where the proxy fails. A ratio estimator spends much of its sample determining a denominator; a blocked design with a strong baseline covariate extracts far more per observation than a raw comparison does. What the prior competes against is the precision actually achieved, which is what t measures and n does not.
# does the analytic law match the MCMC sweep actually run in section 2?
ks = np.array([r[0] for r in rows]); post = np.array([r[2] for r in rows])
emp = 1 - post/post[0] # observed fractional shift toward zero
t_lo = abs(beta_hat_lo/se_lo) # t on the scale the prior is placed on
print(f" t on the log-odds scale = {t_lo:.1f} (probability scale: {abs(tau_hat/se_hat):.1f})")
print(f"\n {'k':>7} {'MCMC shift':>12} {'law':>10}")
for k, e_ in zip(ks, emp):
print(f" {k:>7g} {e_:>11.4%} {infl(k,t_lo):>10.4%}")
gap = np.max(np.abs(emp - np.array([infl(k,t_lo) for k in ks])))
print(f"\nThe law tracks the sampler across four orders of magnitude in k, with a largest discrepancy of")
print(f"{gap:.2%}, at the dogmatic end. That residual is not sampling noise: the law is exact for a normal")
print("likelihood on the scale the PRIOR sits on (log-odds), while the shift is reported on the")
print("probability scale, and the map between them is not linear. The agreement everywhere else is what")
print("Bernstein-von Mises promises -- at this much information the posterior is normal and the prior")
print("enters only through that one ratio, whatever the likelihood happened to be.")
t on the log-odds scale = 31.3 (probability scale: 30.2)
k MCMC shift law
10 0.0000% 0.0010%
3 -0.0321% 0.0114%
1 0.1216% 0.1022%
0.3 1.1995% 1.1243%
0.1 9.5627% 9.2836%
0.03 54.4783% 53.2069%
0.01 91.6309% 91.0981%
The law tracks the sampler across four orders of magnitude in k, with a largest discrepancy of
1.27%, at the dogmatic end. That residual is not sampling noise: the law is exact for a normal
likelihood on the scale the PRIOR sits on (log-odds), while the shift is reported on the
probability scale, and the map between them is not linear. The agreement everywhere else is what
Bernstein-von Mises promises -- at this much information the posterior is normal and the prior
enters only through that one ratio, whatever the likelihood happened to be.
4. Where the prior starts working again — on these same voters¶
The result above is about the question, not the dataset. "229,444 voters" is the sample size of the headline comparison. Ask something more specific and that number falls apart: turnout effects are known to depend on household size, since the Neighbors mailing exposes a voter to more scrutiny in a larger household, and past voting is the strongest single predictor of turning out at all.
Cross those two and the experiment stops being large. The cells at the sparse end contain a few hundred treated voters, and there the prior — in the form of partial pooling across cells — does exactly the work it could not do above.
d["hh"] = np.minimum(d.hhsize, 5)
d["age"] = 2006 - d.yearofbirth
d["ab"] = pd.cut(d.age, [0,35,50,65,120], labels=["<35","35-50","50-65","65+"])
g = (d.groupby(["hh","ab","treat"], observed=True)
.agg(n=("primary2006","size"), y=("primary2006","sum")).reset_index())
w = g.pivot_table(index=["hh","ab"], columns="treat", values=["n","y"], observed=True).dropna()
w.columns = [f"{a}{b}" for a,b in w.columns]; w = w.reset_index().astype({"n0":int,"n1":int,"y0":int,"y1":int})
w["p0"], w["p1"] = w.y0/w.n0, w.y1/w.n1
w["eff"] = 100*(w.p1-w.p0)
w["se"] = 100*np.sqrt(w.p1*(1-w.p1)/w.n1 + w.p0*(1-w.p0)/w.n0)
w = w.sort_values("n1").reset_index(drop=True)
print(f"{len(w)} cells (household size x age band). Treated-arm sizes run "
f"{w.n1.min():,} to {w.n1.max():,}; the headline comparison had {n_t:,}.")
print(f"\n {'cell':>14} {'n treated':>10} {'unpooled effect':>16} {'SE':>7}")
for _, r in pd.concat([w.head(4), w.tail(2)]).iterrows():
print(f" {f'hh{int(r.hh)}, {r.ab}':>14} {int(r.n1):>10,} {r.eff:>15.2f}pp {r.se:>7.2f}")
print(f"\n widest unpooled SE {w.se.max():.2f}pp against the headline's {100*se_hat:.2f}pp -- a factor of {w.se.max()/(100*se_hat):.0f}.")
20 cells (household size x age band). Treated-arm sizes run 14 to 10,184; the headline comparison had 38,201.
cell n treated unpooled effect SE
hh5, 65+ 14 44.51pp 13.55
hh4, 65+ 38 20.30pp 8.58
hh5, 35-50 51 11.12pp 7.53
hh5, 50-65 141 11.32pp 4.69
hh2, 50-65 8,086 9.13pp 0.59
hh2, 35-50 10,184 8.69pp 0.51
widest unpooled SE 13.55pp against the headline's 0.27pp -- a factor of 50.
C = len(w)
with pm.Model() as hm:
mu_a = pm.Normal("mu_a", 0, 1.5); sd_a = pm.HalfNormal("sd_a", 1.0)
za = pm.Normal("za", 0, 1, shape=C)
alpha = pm.Deterministic("alpha", mu_a + sd_a*za) # cell baseline log-odds
mu_b = pm.Normal("mu_b", 0, 1.0); sd_b = pm.HalfNormal("sd_b", 0.5)
zb = pm.Normal("zb", 0, 1, shape=C)
beta = pm.Deterministic("beta", mu_b + sd_b*zb) # cell treatment effect, log-odds
pm.Binomial("o0", n=w.n0.values, p=pm.math.invlogit(alpha), observed=w.y0.values)
pm.Binomial("o1", n=w.n1.values, p=pm.math.invlogit(alpha + beta), observed=w.y1.values)
pm.Deterministic("eff_pp", (pm.math.invlogit(alpha+beta) - pm.math.invlogit(alpha))*100)
with contextlib.redirect_stderr(io.StringIO()):
ih = pm.sample(2000, tune=2000, chains=4, cores=1, target_accept=0.95,
progressbar=False, random_seed=4)
rh = float(pd.to_numeric(az.summary(ih, var_names=["eff_pp"])["r_hat"], errors="coerce").max())
ep = ih.posterior["eff_pp"].values.reshape(-1, C)
w["pool"] = ep.mean(0); w["pool_se"] = ep.std(0)
w["moved"] = w.pool - w.eff
print(f"max r-hat {rh:.3f}\n")
print(f" {'cell':>14} {'n treated':>10} {'unpooled':>10} {'pooled':>9} {'moved':>8} {'SE unpool':>10} {'SE pool':>8}")
for _, r in pd.concat([w.head(4), w.tail(3)]).iterrows():
print(f" {f'hh{int(r.hh)}, {r.ab}':>14} {int(r.n1):>10,} {r.eff:>9.2f}pp {r.pool:>8.2f}pp "
f"{r.moved:>+8.2f} {r.se:>10.2f} {r.pool_se:>8.2f}")
sm_, lg_ = w.head(5), w.tail(5)
print(f"\n five smallest cells: mean absolute move {sm_.moved.abs().mean():.2f}pp, "
f"SE cut from {sm_.se.mean():.2f} to {sm_.pool_se.mean():.2f}")
print(f" five largest cells: mean absolute move {lg_.moved.abs().mean():.2f}pp, "
f"SE cut from {lg_.se.mean():.2f} to {lg_.pool_se.mean():.2f}")
print(f"\n between-cell spread of the effect: sd_b posterior mean "
f"{ih.posterior['sd_b'].values.mean():.3f} log-odds")
sd_b_pp = ih.posterior["sd_b"].values.mean()*p_c*(1-p_c)*100
print(f" which is about {sd_b_pp:.1f} percentage points -- against a mean effect near {w.pool.mean():.1f}")
print()
print("Same 305,866 voters. The same machinery that could not move the headline by a thousandth of a")
print("point moves the sparse cells by ten points or more and cuts their uncertainty several-fold.")
print("'The dataset is large' was a statement about the QUESTION, never about the data.")
print()
print("Two honest qualifications. First, the information here is borrowed from the other cells rather")
print("than supplied from outside, so this is a hierarchical prior, not the kind swept in section 2 --")
print("the same mechanism, differently sourced. Second, the pooled standard errors are as small as they")
print(f"are because sd_b is small: the model concludes the effect varies by only ~{sd_b_pp:.1f}pp across cells and")
print("therefore pools hard. That is a conclusion drawn from the data, not an assumption -- but if it is")
print("wrong, these intervals are overconfident, exactly as Project STAR's varying-intercept model was.")
max r-hat 1.000
cell n treated unpooled pooled moved SE unpool SE pool
hh5, 65+ 14 44.51pp 8.64pp -35.87 13.55 1.47
hh4, 65+ 38 20.30pp 8.88pp -11.42 8.58 1.42
hh5, 35-50 51 11.12pp 8.39pp -2.73 7.53 1.37
hh5, 50-65 141 11.32pp 9.15pp -2.18 4.69 1.36
hh2, 65+ 3,391 8.85pp 8.84pp -0.00 0.94 0.74
hh2, 50-65 8,086 9.13pp 8.93pp -0.21 0.59 0.53
hh2, 35-50 10,184 8.69pp 8.40pp -0.28 0.51 0.49
five smallest cells: mean absolute move 10.96pp, SE cut from 7.43 to 1.31
five largest cells: mean absolute move 0.20pp, SE cut from 0.86 to 0.67
between-cell spread of the effect: sd_b posterior mean 0.048 log-odds
which is about 1.0 percentage points -- against a mean effect near 7.9
Same 305,866 voters. The same machinery that could not move the headline by a thousandth of a
point moves the sparse cells by ten points or more and cuts their uncertainty several-fold.
'The dataset is large' was a statement about the QUESTION, never about the data.
Two honest qualifications. First, the information here is borrowed from the other cells rather
than supplied from outside, so this is a hierarchical prior, not the kind swept in section 2 --
the same mechanism, differently sourced. Second, the pooled standard errors are as small as they
are because sd_b is small: the model concludes the effect varies by only ~1.0pp across cells and
therefore pools hard. That is a conclusion drawn from the data, not an assumption -- but if it is
wrong, these intervals are overconfident, exactly as Project STAR's varying-intercept model was.
fig, ax = plt.subplots(1, 3, figsize=(15, 4.3))
kk = np.logspace(-2, 1, 200)
for nm, e, s_, n_, _ in EXP:
t = abs(e/s_)
ax[0].plot(kk, [infl(k,t) for k in kk], lw=2, label=f"{nm.split(',')[0]} (t={t:.1f})")
ax[0].set_xscale("log"); ax[0].set_xlabel("prior sd, as a multiple of the observed effect (k)")
ax[0].set_ylabel("fraction of the estimate the prior removes")
ax[0].set_title("Prior influence is governed by t, not by n"); ax[0].legend(fontsize=7.5)
ns = [e[3] for e in EXP]; ts = [abs(e[1]/e[2]) for e in EXP]
ax[1].scatter(ns, [infl(1,t) for t in ts], s=70, color=BLUE, zorder=3)
OFF = {"Darwin's maize":(8,2), "Electric Company":(8,-4), "Project STAR":(-8,-14),
"Vitamin A":(8,2), "Social Pressure":(-8,4)}
for (nm,e,s_,n_,_), t in zip(EXP, ts):
lab = nm.split(",")[0]; dx, dy = OFF[lab]
ax[1].annotate(lab, (n_, infl(1,t)), fontsize=7.5, xytext=(dx,dy),
textcoords="offset points", ha="right" if dx < 0 else "left")
ax[1].set_xscale("log"); ax[1].set_yscale("log")
ax[1].set_xlabel("sample size"); ax[1].set_ylabel("prior influence at k = 1")
ax[1].set_title("Falling with n, but not because of n")
o = np.argsort(w.n1.values)
ax[2].errorbar(w.n1.values[o], w.eff.values[o], yerr=1.96*w.se.values[o], fmt="o", ms=4,
color=GREY, lw=1, capsize=2, label="unpooled cell estimate")
ax[2].errorbar(w.n1.values[o]*1.06, w.pool.values[o], yerr=1.96*w.pool_se.values[o], fmt="s", ms=4,
color=BLUE, lw=1.4, capsize=2, label="partially pooled")
ax[2].axhline(100*tau_hat, color=GREEN, ls="--", lw=1.4, label="headline effect")
ax[2].set_xscale("log"); ax[2].set_xlabel("treated voters in the cell")
ax[2].set_ylabel("effect (percentage points)")
ax[2].set_title("Where pooling still bites"); ax[2].legend(fontsize=7.5)
plt.tight_layout(); plt.show()
5. Summary¶
The prior does nothing to the headline, and the sweep shows how hard it has to try before it does anything at all. A prior centred on no effect whose standard deviation equals the entire observed effect moves the posterior by 0.0099 percentage points — about a tenth of one percent of the estimate. The prior wins eventually, as it must: by $k=0.01$ the posterior has been dragged from 8.13 to 0.68. But that is a prior asserting, before seeing anything, that the effect is one percent of what 229,444 voters went on to show.
There is an exact law, and sample size is not in it. For a normal likelihood, a $\mathcal{N}(0,s)$ prior with $s = k|\hat\tau|$ removes a fraction $1/(k^2t^2+1)$ of the estimate. It depends only on $k$ and the ***t*-statistic**. The notebook checks it against the MCMC sweep rather than asserting it, and it tracks across four orders of magnitude in $k$, the small residual at the dogmatic end being the log-odds-to-probability transformation rather than sampling noise.
That slightly corrects this group's own framing. These pages are ordered by sample size as a stand-in for information, and within this arc the stand-in fails. The vitamin A CACE rests on 23,682 children at $t = 2.78$; the Electric Company grade-1 effect rests on 192 at $t = 3.37$. The experiment with 123 times the sample is the more prior-sensitive of the two, because a ratio estimator spends much of its sample determining a denominator while a blocked design with a strong baseline covariate extracts far more per observation. What a prior competes against is the precision actually achieved.
And the claim is about the question, not the dataset. Split these same voters by household size and age band and the cells run from 10,184 treated voters down to 14. In the sparse ones the unpooled estimates are nonsense — 44.5 points in the smallest — and partial pooling moves the five smallest by 11 points on average while cutting their standard errors from 7.4 to 1.3. The five largest move by 0.2. The machinery that could not touch the headline is decisive two rows down.
The through-line for the whole group is that "how much does the prior matter" is not a question about Bayesian methods at all — it is a question about how much the data have already determined, which the t-statistic answers exactly. At $t=30$ the choice of framework is a matter of interpretation and nothing else, since the posterior and the confidence interval coincide to three decimal places. At $t=2$ it decides the verdict, which is where Darwin's fifteen pairs came in. Everything between those two pages is a matter of degree, and the degree is calculable in advance.
Cross-links. The same experiment appears in covariate adjustment as the low-$\rho^2$ extreme, where a weak baseline covariate buys only 3% variance reduction — the mirror image of the point made here, since both are statements about how much information the design has already extracted. The pooling in section 4 is the machinery of the Electric Company example and Project STAR, applied where the cells are small enough for it to matter.