A/B Testing — ML-Based Variance Reduction (CUPAC)

Python · NumPy · scikit-learn · R · gbm

Lifting the ρ² Ceiling

Covariate adjustment established the governing law: a pre-treatment control variate cuts the effect's variance by exactly $\rho^2$ , the squared correlation between the outcome and that covariate. It also established the ceiling, which is the reason this notebook exists. An outcome usually depends on many pre-experiment features, often nonlinearly, and any single covariate leaves most of that predictable structure untouched.

CUPAC (Control Using Predictions As Covariates; Tang et al., DoorDash 2020) removes the ceiling without changing the mathematics. Train a machine-learning model on all pre-experiment features to predict the outcome, and use its prediction $\hat Y$ as the control variate. The variance reduction is still exactly $\rho^2(Y, \hat Y)$ — but a flexible learner can track the outcome far better than any one feature, so that quantity is no longer capped by the best available covariate.

Y_i^{\text{cupac}} = Y_i - \beta\,(\hat Y_i - \bar{\hat Y}), \qquad \beta = \frac{\operatorname{Cov}(Y, \hat Y)}{\operatorname{Var}(\hat Y)}, \qquad \text{variance reduction} = \rho^2(Y, \hat Y)

The simulation builds an outcome that depends nonlinearly on six pre-experiment features. The best single covariate correlates moderately, so CUPED (Controlled-experiment Using Pre-Existing Data, the single-covariate form) delivers a respectable but bounded 34% variance reduction at $\rho^2 = 0.33$ . A cross-fitted gradient booster reaches $\rho^2 = 0.74$ , so CUPAC delivers 75% — more than double, and it cuts the standard error in half, from 0.0697 unadjusted to 0.0352.

true effect δ = 0.5	estimate	SE	ρ²	variance reduction
unadjusted	0.427	0.0697	—	—
CUPED (best single covariate)	0.512	0.0567	0.33	34%
CUPAC (cross-fitted ML prediction)	0.522	0.0352	0.74	75%

Halving a standard error is worth putting in the currency of the power calculation, because it is the same arithmetic running backwards. Sample size scales with $1/\delta^2$ for a given detectable effect, so cutting the standard error by half is equivalent to quadrupling the sample — obtained from data you already had, on an experiment you were running anyway. That is the cheapest large power gain available in experimentation, which is why it spread quickly through industry once DoorDash published it.

The One Rule That Keeps It Honest

The precision is free only if it introduces no bias, and that rests on a single discipline stated plainly: the control variate must be a function of pre-treatment data alone. If $\hat Y$ drew on anything measured after assignment — or on the treatment indicator itself — then subtracting it would remove part of the effect along with the noise, biasing the estimate toward zero. That is the mediator trap wearing a machine-learning costume, and it is easy to walk into when a feature pipeline is built for prediction rather than for causal work.

The second safeguard is cross-fitting. If the model that predicts a unit's outcome was trained on that unit, the prediction partly memorises the outcome's noise, and subtracting it removes variation that was never predictable in the first place. Out-of-fold predictions ensure each unit is scored by a model that never saw it. Both safeguards are verified rather than asserted: across repeated experiments all three estimators are unbiased with correct coverage, and CUPAC differs only in spread.

repeated experiments, true effect 0.5	mean estimate	bias	SD	95% coverage
unadjusted	0.504	+0.004	0.0677	0.94
CUPED	0.504	+0.004	0.0606	0.94
CUPAC	0.502	+0.002	0.0402	0.94

That table is the argument in its cleanest form. Bias is +0.002, coverage is 0.94, and the standard deviation falls from 0.0677 to 0.0402. Nothing was traded — no bias was accepted in exchange for precision, and no assumption beyond "these features were fixed before assignment" was added. It is worth contrasting that with almost everything else in this group, where every gain came with a cost: covariate adjustment risked Freedman's small-sample bias, clustering cost variance, bandits cost interpretability. Here the discipline is genuinely cheap, provided the one rule is respected.

Where this sits

The predictor can be any regressor from the ML arc — the notebook uses gradient boosting, but nothing depends on that choice, since only $\rho^2(Y,\hat Y)$ matters. The residualize-with-a-model idea is exactly what double machine learning does later in the arc, with one crucial difference in purpose: there it is used for identification, removing confounding in observational data, while here it is used purely for precision in an experiment where identification was never in doubt. The same estimator serving two entirely different jobs is worth noticing. It also stacks with everything else in this group — a tighter variance reaches any sequential boundary sooner, and applies equally to ratio metrics once their variance is computed at the right unit.

Notebooks

Python Control variates from first principles, CUPED with the best single covariate hitting its rho-squared ceiling, and CUPAC using a cross-fitted gradient booster over six nonlinear pre-experiment features, with a repeated-experiment check confirming all three estimators are unbiased at nominal coverage (Python) Open → R The same comparison in R, with a gradient-boosting control variate cross-fitted out of fold and the CUPED and CUPAC variance reductions reproduced (R) Open →

References

Deng, A., Xu, Y., Kohavi, R. & Walker, T. (2013). Improving the sensitivity of online controlled experiments by utilizing pre-experiment data. Proceedings of the sixth ACM international conference on Web search and data mining, 123–132. — CUPED, and variance reduction from pre-experiment data
Deng, A., Knoblich, U. & Lu, J. (2018). Applying the Delta Method in Metric Analytics. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 233–242. — the delta method for ratio metrics
Lin, W. (2013). Critique Notes. Critique 41(2), 143–150. — regression adjustment in experiments