Covariate Adjustment in Experiments

Precision, Not Bias

Randomization already delivers an unbiased effect from a plain difference in means, so adjusting for covariates cannot be about removing bias — there is none to remove. It is about precision. A pre-treatment variable that predicts the outcome absorbs variation that has nothing to do with the treatment, shrinking the standard error without changing what the estimate targets. That is the everyday workhorse of modern experimentation, and it carries different names in different fields: ANCOVA (analysis of covariance) in clinical trials, CUPED (Controlled-experiment Using Pre-Existing Data, Deng et al. 2013) in online A/B testing. They are the same estimator, and it is worth knowing that, because the two literatures rediscovered it independently and still largely cite each other's results as novel.

Two real experiments bracket the phenomenon deliberately. The Electric Company (1971) randomized 192 classrooms across grades 1–4, 96 treated and 96 control, to watch an educational television programme or not, with reading measured before and after. Its pre-test correlates with the post-test at 0.883 — close to the best case adjustment can hope for. Social Pressure GOTV (Gerber, Green & Larimer 2008) mailed get-out-the-vote messages to 305,866 voters, and its only baseline covariate — whether the person voted in the 2004 primary — correlates with 2006 turnout at just 0.163. One near-ideal, one nearly useless, on purpose.

ANCOVA on a Strong Baseline

On the Electric Company the unadjusted difference is +5.66 points with a standard error of 2.537. Adding the pre-test moves the point estimate to +4.73 and the standard error to 1.169 — the estimate barely moves, as randomization guarantees, while the standard error is cut by 54% and the sampling variance by 79%. A bootstrap agrees independently: 2.47 unadjusted against 1.15 adjusted, about 2.1× more precise for the same target. The confidence interval narrows from a width of 9.94 to 4.58.

The small movement in the point estimate is worth naming rather than waving past, because it is the second thing adjustment does. Randomization balances covariates in expectation, not in any particular sample, and this sample has a slight pre-test imbalance across grades. Adjustment removes that chance imbalance too. So the shift from +5.66 to +4.73 is not evidence of bias being introduced; it is evidence of a lucky draw being corrected.

CUPED and the ρ² Law

CUPED is the same thing arrived at from a different direction. Rather than regressing the outcome on treatment and covariate, replace the outcome with an adjusted version and take a plain difference in means. Because $\theta$ is exactly the regression slope, the two estimators coincide — and the variance reduction has a closed form that makes the whole subject predictable in advance.

That law is what the two experiments were chosen to test, and it holds at both ends. The Electric Company, with $\rho = 0.883$ and $\rho^2 = 0.780$, gives a 79% variance reduction. Social Pressure, with $\rho = 0.163$ and $\rho^2 = 0.026$, gives 3% — and its effect moves only from +0.081 to +0.080. The practical consequence is that the value of adjustment is knowable before running it: square the correlation between the covariate and the outcome. A weak covariate is not harmful, it is simply not worth much, and no amount of methodological sophistication will change that.

Freedman’s Objection and Lin’s Fix

There is a real objection to plain ANCOVA, and it is not a technicality. Freedman (2008) showed it can be slightly biased in finite samples, and can even hurt precision, because a single common slope is imposed on both arms — if the covariate relates to the outcome differently under treatment and control, that constraint is wrong. Lin (2013) fixed it with a small change: centre the covariates and fully interact them with treatment, fitting a separate slope per arm. The result is guaranteed asymptotically no worse than the unadjusted difference, and it removes Freedman's bias, at the cost of a few parameters.

In practice the interaction rarely changes much at scale — on Social Pressure, Lin and ANCOVA agree to the fourth decimal — which is precisely why it is the right default: it costs almost nothing when it is unnecessary and protects you when it is not. The Electric Company is adjusted within grade, respecting its blocked design, and the per-grade view shows why the interacted form matters: adjustment tightens every grade, but it moves the estimates in different directions and by very different amounts, from +8.36 down to +4.27 in grade 2 while grade 3 rises from +0.33 to +1.91. Those are genuinely different slopes, not noise around a common one.

Where this sits

This is the precision layer sitting on top of the RCT foundations, where identification was settled and nothing was said about how sharp the answer would be. The same pre-period-covariate logic returns as the baseline period in difference-in-differences and as the residualization idea behind double machine learning, later in the arc. The blocked design here connects to stratified and matched-pair randomization — the Electric Company was randomized in matched pairs within grade, which is the same structure Darwin's pots have in the previous notebook. And CUPED is where this subject meets online experimentation, which the A/B-testing notebooks in this group take up in earnest.

Notebooks

Downloads

References

• Freedman, D. A. (2008). On regression adjustments to experimental data. Advances in Applied Mathematics 40(2), 180–193. — the critique: adjustment can hurt
• Lin, W. (2013). Critique Notes. Critique 41(2), 143–150. — the answer, with interactions
• Neyman, J. (1990). The Design of Experiments: Statistical Principles for Practical Application. Technometrics 32(4), 448. — the randomisation-based framework both argue within