Bayesian Analysis of a Small Experiment

A Result Four Thousandths of an Inch From the Boundary

The frequentist treatment of Darwin's maize settled it three ways, and read like for like the three do not quite agree. Fisher's exact permutation test over all $2^{15}$ sign patterns gives a two-sided $p$ of 0.0527; the paired $t$-test gives 0.0497; Neyman's interval is [0.004, 5.229]. On the conventional threshold the assumption-free test fails and the normal-theory one passes, separated by three thousandths. The interval is the reason for this page. The interval clears zero by four thousandths of an inch, and a result that marginal invites a question the frequentist machinery has no way to answer: how much of this conclusion is Darwin's plants, and how much is the analysis? There is no dial to turn. A Bayesian treatment has exactly that dial.

The model is deliberately plain, so nothing turns on cleverness: the 15 within-pair differences are normal around an effect $\tau$, with a prior on $\tau$ centred at zero and a scale $s$ that controls how firmly it insists. Centring at zero matters for how the output should be read — the prior is not helping the conclusion along, it is pulling against it.

The first result is that the verdict changes. With $s = 10$ — a prior standard deviation four times the observed effect, which nobody would describe as informative — the 95% credible interval (the range holding 95% of the posterior probability, which is what most people wrongly assume a confidence interval is) runs [−0.050, 5.256] and contains zero, where the $t$-interval excluded it. $\Pr(\tau>0) = 0.9721$, just under the 0.975 a one-sided test at 0.025 would require. Nothing about the plants changed between those two lines.

That is not a Bayesian conjuring trick, and it should not be read as one framework catching the other out. It is what a result sitting four thousandths of an inch from a threshold was always going to do on contact with a second method. The useful lesson is about the fragility of the original finding, not about the superiority of either approach.

How Much of the Verdict Is the Prior?

The sensitivity sweep is the deliverable the published page could not produce: refit across prior scales from 0.5 (a sceptic who thinks the effect is almost certainly under half an inch) to 100 (essentially flat), and watch what the data can and cannot defend.

Two different things happen to the two different claims. The direction is robust to ignorance but not to active scepticism: $\Pr(\tau>0)$ holds above 0.94 for any prior scale of 2 or more — that is, any prior that does not rule out an effect the size of the one observed — and then falls to 0.876 at $s=1$ and 0.745 at $s=0.5$. The magnitude is fragile throughout: the point estimate runs from 0.31 to 2.62 inches across the same sweep. Fifteen pots can say the effect is probably positive. They cannot say how large it is, and no amount of analysis will change that.

The Outliers, and the Assumption Fisher Refused to Make

The last section addresses the assumption Fisher was avoiding in the first place. Darwin's differences contain two large negatives, −8.375 and −6.000, against thirteen positives. The paired $t$-test must treat those as ordinary draws from the same normal, which inflates the estimated spread and widens its interval; Fisher's permutation test sidesteps the issue by assuming nothing about shape at all. A third route keeps the parametric structure but lets the tails be heavy, replacing the normal likelihood with a Student-$t$ whose degrees of freedom are estimated rather than assumed.

The degrees-of-freedom parameter $\nu$ is the dial: small values mean heavy tails, and above roughly 30 a Student-$t$ is indistinguishable from a normal. Its posterior median here is 15.2, with $\Pr(\nu < 10) = 0.31$ — mild but real tail-heaviness, exactly what two outliers in fifteen observations should produce. And the effect of taking it seriously runs against the intuition that robustness costs confidence: down-weighting the two negative pairs removes variance the $t$-test had to carry, so the estimate rises to 2.855 and the interval tightens to [0.183, 5.374], excluding zero more comfortably than the $t$-test did.

Which leaves three analyses of one fifteen-pot experiment disagreeing about whether the effect is established:

Reading those three lines together is the point of the page. The disagreement is not a defect in any of them — each is correct given its assumptions, and the assumptions are all defensible. It is a property of the evidence. A knife-edge result carries less information than its p-value suggests, and the only way to see that is to analyse it more than once.

Where this sits

The prior-sensitivity discipline recurs throughout the Bayesian arc, and the Student-$t$ likelihood is the same robustness device used in the heavy-tails distribution notebooks. What this page does not do is more interesting than what it does: it changes nothing about identification. Randomization already settled that, and every model here inherits it unexamined. The next example in this group moves from one effect with a sceptical prior to four effects shrinking toward each other, where the Bayesian machinery starts doing structural work rather than sensitivity analysis.

Notebook

References

• Gelman, A. (2006). Conservative prior distributions for variance parameters in hierarchical models. Canadian Journal of Statistics 34(3), 377–390. — priors for variance parameters
• Gelman, A., Jakulin, A., Pittau, M. G. & Su, Y.-S. (2008). A weakly informative default prior distribution for logistic and other regression models. The Annals of Applied Statistics 2(4). — weakly informative default priors
• Rubin, D. B. (1981). Estimation in Parallel Randomized Experiments. Journal of Educational Statistics 6(4), 377. — what pooling buys when each experiment is small