A/B Testing — Ratio Metrics and the Delta Method

When the Randomization Unit Is Not the Analysis Unit

Most product metrics are ratios — clicks per session, revenue per session, conversions per visit — and that creates a mismatch which is easy to miss and expensive to keep. The experiment randomizes users, but the metric is computed over sessions. Sessions belonging to the same user are correlated: some people click constantly, others almost never. A standard error that treats every session as an independent trial is therefore too small, and the intervals built on it are too narrow. Nothing about the estimate is wrong; everything about its stated precision is.

The simulation randomizes by user, gives each user a random number of sessions and a user-level random effect that makes their sessions resemble one another, then measures click-through rate as total clicks over total sessions. One arm has 3,000 users generating 14,908 sessions at a CTR of 0.405. The naive formula treats that as 14,908 independent Bernoulli trials. The design says the independent unit is the user, so the effective sample size is far closer to 3,000.

The Delta Method

The correct treatment recognises that the arm metric is a ratio of means, $R = \bar c / \bar s$, where both averages are taken per user. The delta method linearizes that ratio around the means and returns its variance at the user level, with $n$ counting users rather than sessions. The covariance term matters: users with more sessions also accumulate more clicks, and ignoring that correlation would misstate the variance in either direction.

On this arm the naive standard error is 0.0041 and the delta method gives 0.0050 — about 1.24× larger. A cluster bootstrap that resamples whole users, which assumes nothing about linearization at all, returns 0.0050 as well. Two methods with different logic agreeing to four decimals is the strongest evidence available that this is the honest number.

What that costs in coverage

The consequence is measurable, and this is what makes the section more than a formula derivation. With a true lift of 0.020 and 1,500 simulated experiments, delta-method intervals cover 95.0% of the time, exactly as advertised. Naive session-level intervals cover 87.4%. A nominal 95% interval delivering 87% is not a rounding issue — it means roughly one experiment in eight is producing a confidence statement that is simply false, and since the failures are all in the direction of overconfidence, the surplus arrives as false positives. A team running hundreds of experiments a quarter on this footing is manufacturing winners.

The Sample Ratio Mismatch Guardrail

The last piece is a guardrail that costs one line and should precede everything else. A Sample Ratio Mismatch is a discrepancy between the intended assignment split and the observed one. It matters because the usual causes — biased assignment, a redirect dropping users, logging loss on one arm — do not merely add noise, they typically bias the effect estimate. The test is a chi-square goodness-of-fit on the arm counts against the intended split.

The convention is to alarm at a very small p-value, and the reason is visible in the third row below. A split of 50,600 / 49,400 is a deviation of six-tenths of a percentage point — invisible to the eye, and the kind of thing one would be tempted to wave through — yet it produces $\chi^2 = 14.40$ and $p = 0.00015$. At a hundred thousand users the test is sensitive enough that even a small genuine imbalance is unmistakable, which is precisely why a failure should be treated as disqualifying rather than debatable. No amount of variance reduction or sequential machinery rescues an experiment whose split is broken.

Where this sits

This is the design-effect lesson from cluster designs applied to a ratio estimand: the same principle that says analyse at the level you randomized, arriving in the form that dominates real product metrics. Pairing it with CUPED is complementary rather than redundant — the delta method corrects the variance, CUPED reduces it, and a metric can need both. The delta method itself recurs wherever a causal estimand is a nonlinear function of means, including restricted mean survival time later in the arc. And the guardrail generalises: sequential monitoring assumes a valid split just as much as a fixed-horizon test does.

Notebooks

References

• Deng, A., Knoblich, U. & Lu, J. (2018). Applying the Delta Method in Metric Analytics. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 233–242. — the delta method for ratio metrics
• Deng, A., Xu, Y., Kohavi, R. & Walker, T. (2013). Improving the sensitivity of online controlled experiments by utilizing pre-experiment data. Proceedings of the sixth ACM international conference on Web search and data mining, 123–132. — variance reduction on the same metrics