Randomized Experiments & Randomization Inference

Two Potential Outcomes, One Observed

This is where the whole arc is anchored, because it is the one setting in which a causal effect is identified by design rather than by assumption. The Rubin causal model gives each unit two potential outcomes — $Y_i(1)$, what it would show under treatment, and $Y_i(0)$, what it would show without — and defines the unit-level effect as their difference. The trouble is immediate: only one of the two is ever observed. That is the fundamental problem of causal inference, and it makes a causal effect a missing-data problem before it is anything else. Randomization is what rescues the average of those effects: if treatment is assigned independently of the potential outcomes, the treated and control groups match in expectation on every pre-treatment characteristic, observed and unobserved, so the plain difference in means is unbiased for the average treatment effect.

That claim is worth watching rather than accepting. Simulating a population whose true ATE is 4.0 and letting a confounder drive both who is treated and what they score, the naive difference in means returns 9.21 — biased by +5.21, because the treated happened to be the ones with more of the confounder. Randomizing the same units removes it: 3.94 on one draw, and averaged over 4,000 re-randomizations, exactly 4.00. Nothing about the estimator changed. What changed is who was assigned to which arm, which is the entire content of the claim.

Fisher — Testing a Sharp Null Exactly

Fisher asks the yes/no question, and answers it without a model, a distributional assumption, or a large sample. His sharp null is that treatment changes no unit's outcome at all — under which both potential outcomes are known for everybody, since they are equal to the one observed. That makes it possible to compute the test statistic under every assignment the randomization could have produced, and the $p$-value is simply the share of them at least as extreme as what happened. The reference distribution is not assumed; it is generated by the design.

The original illustration is the Lady Tasting Tea (Fisher, The Design of Experiments, 1935). Muriel Bristol claimed she could tell whether milk or tea had been poured first. Fisher gave her eight cups, four of each, told her the split, and asked her to name the four milk-first cups. She named all four. Under the null that she is guessing, the count of correct identifications is hypergeometric — drawing four from eight without replacement — so getting all four right has probability $1/\binom{8}{4} = 1/70$. The notebook enumerates that null distribution and gets 0.0143, matching scipy.stats.fisher_exact to four decimals. It is not an approximation to a p-value; it is the proportion of the seventy equally likely arrangements in which a guesser does this well.

Darwin’s maize, enumerated completely

The field version uses the data Fisher himself used: Darwin's maize, from The Effects of Cross and Self Fertilisation in the Vegetable Kingdom (1876). Darwin grew 15 pairs of Zea mays, one plant cross-fertilized and one self-fertilized in each pot, and measured final heights. The pairing is the design: two plants sharing a pot share soil, light and water, so the within-pair difference isolates fertilization from pot-to-pot variation. Under the sharp null the two labels within a pot are exchangeable — meaning the joint distribution is unchanged if you relabel which plant was which, so no ordering of the labels is more probable than another. That is the assumption the whole method rests on, and here the design supplies it rather than the analyst assuming it. Each observed difference was therefore equally likely to have come out with the opposite sign — and with 15 pairs there are only $2^{15} = 32{,}768$ sign patterns, few enough to enumerate the null distribution completely rather than sample it.

Cross-fertilized plants ran 2.617 inches taller on average, with 13 of 15 pairs favouring cross-fertilization. Exactly 863 of the 32,768 sign patterns produce a mean that large or larger, giving a one-sided $p$ of 0.02634 and two-sided 0.05267. The classical paired $t$-test gives t = 2.148 and $p = 0.0497$ — the same number Fisher reported. That agreement is the point of the exercise rather than a footnote to it: the permutation test assumes no normality and still lands within a few thousandths of the $t$-test, which is precisely the argument Fisher used the data to make.

Neyman — Estimating How Large

Neyman asks the other question — not did anything happen but how large is the average effect, and how well do we know it. His inference comes from the same re-randomization distribution drawn in the first section, and on Darwin's data it gives an ATE of 2.617 inches with $\text{SE} = s_d/\sqrt{n} = 4.718/\sqrt{15} = 1.218$, a normal interval of [0.229, 5.004] and a $t$-based one of [0.004, 5.229], wider as it should be at $n=15$. Worth noting the $t$-based interval's lower bound sits at 0.004 — it excludes zero by four thousandths of an inch, which is a fair description of how much evidence fifteen pots actually carry.

The two frameworks are complementary rather than rival, and the distinction survives the whole arc. Fisher tests a sharp null exactly, which is what small samples and the question "did anything happen?" call for. Neyman estimates an average and attaches an interval, which is what a decision needs. One more asymmetry is worth carrying forward: for unpaired two-arm trials Neyman's variance $s_1^2/n_1 + s_0^2/n_0$ is conservative, because the covariance between the two potential outcomes can never be estimated — nobody is ever observed under both. The missing counterfactual does not merely complicate estimation; it leaves a term in the variance permanently unidentified.

Where this sits

Everything after this subsection is an attempt to earn back what randomization gives away for free. The potential-outcomes framing is the causal face of Missing Data — an unobserved counterfactual is a missing value with a particular structure — and the hypergeometric null comes from the Statistical Distributions catalogue. The permutation logic recurs wherever a reference distribution is built by resampling rather than assumed, including the purged validation discipline in Purged & Embargoed Cross-Validation. What comes next in this group is what happens when randomization holds but is not enough on its own: covariate adjustment for precision, and noncompliance for when the assigned treatment is not the one received.

Notebooks

References

• Neyman, J. (1990). The Design of Experiments: Statistical Principles for Practical Application. Technometrics 32(4), 448. — the randomisation-based analysis
• Rubin, D. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology 66(5), 688–701. — potential outcomes
• Freedman, D. A. (2008). On regression adjustments to experimental data. Advances in Applied Mathematics 40(2), 180–193. — and what regression adjustment does to it