A/B Testing — Sequential Monitoring and Power

The Peeking Problem

An online A/B test is a randomized experiment, so identification is settled before it starts. What the online setting adds is a problem of a completely different kind: results arrive continuously, and everyone wants to look at them. A classical p-value is only valid at a sample size fixed in advance, and checking repeatedly turns each look into another opportunity to cross the threshold by luck. This is the most common inferential mistake in industry experimentation, and it is not a subtle one.

The way to measure it is an A/A test — both arms identical, true effect exactly zero, so every rejection is a false positive by construction. Testing once gives the nominal 4.8%. Checking after each batch and stopping the first time $|z| > 1.96$ gives this:

At twenty looks the false-positive rate is 24%. Roughly one A/A test in four declares a winner where nothing whatsoever is happening. And the curve is still climbing at fifty looks — there is no sample size at which peeking becomes safe, because the running z-statistic is a random walk that will eventually wander across any fixed boundary. Stopping at the first crossing is guaranteed to find one if you wait long enough.

The Classical Answer — Fix n in Advance

The classical discipline is to fix the sample size in advance from a power calculation and look exactly once. For a two-sample difference in means the requirement per arm follows from the effect worth detecting, and inverting it gives the minimum detectable effect at a sample size you already have.

Two consequences govern all experiment planning, and both are visible in the table. Sample size scales with $1/\delta^2$, so halving the effect you want to catch quadruples the users needed — 1,570 per arm at $\delta = 0.1$ becomes 6,279 at 0.05. And the MDE improves only as $1/\sqrt{n}$, so traffic buys sensitivity at a punishing rate. The design is honest, and it is also rigid enough that teams peek anyway, which is what makes the third approach necessary rather than merely elegant.

Always-Valid Inference — the Mixture SPRT

Always-valid inference resolves the tension instead of policing it. The mixture sequential probability ratio test (Johari, Pekelis & Walsh 2017 — the engine behind Optimizely's Stats Engine) places a mixing prior over the unknown effect and tracks the likelihood ratio of "some effect" against "none". Under the null that ratio is a non-negative martingale with mean 1, so Ville's inequality bounds the probability that it ever exceeds $1/\alpha$ — not at one sample size, but across all of them simultaneously.

That is the whole trick, and it is worth being clear about why it works. The fixed-sample test controls error at one pre-committed moment, so looking at other moments spends error it never budgeted for. The martingale bound is a statement about the entire path, so no amount of looking can exhaust it. The reciprocal $p_n = 1/\Lambda_n$ is an always-valid p-value and inverting the statistic gives a confidence sequence valid at all times.

The measured payoff is both halves at once. Under continuous peeking — checking after every single observation — the mSPRT holds Type-I error at 3.6%, against roughly 25% for naive peeking at the same cadence. And under a real effect of 0.1 it reaches 93% power with a median stop at 1,454 per arm, which is earlier than the fixed-horizon design's 1,570. Monitoring continuously is not merely permitted, it is faster.

The Pharmaceutical Answer — Group-Sequential Boundaries

The R companion supplies the pharmaceutical counterpart, which solved the same problem thirty years earlier under stricter regulatory constraints. Group-sequential designs pre-specify a small number of interim looks and spend the error budget across them with a boundary that starts stringent and relaxes. O'Brien–Fleming runs 4.56, 3.23, 2.63, 2.28, 2.04 across five looks — nearly impossible to cross early, essentially the nominal threshold at the end — while Pocock uses a flat 2.41 throughout. The cost is the interesting part: O'Brien–Fleming inflates the maximum sample size by only 2.8%, Pocock by 22.9%. Simulated on an A/A test at five looks, the O'Brien–Fleming boundary holds Type-I at 4.3% where naive five-look testing gives 14.1%.

Where this sits

This builds directly on the RCT foundations — the design is unchanged, only the monitoring is new — and pairs naturally with covariate adjustment, since CUPED cuts the variance and therefore reaches any given boundary sooner; the two compound rather than compete. The test-inversion logic behind a confidence sequence is the same move that produces conformal prediction's intervals in the ML arc, and the martingale argument is the same object that underpins the sequential methods there. The remaining notebooks in this group take up the other ways an online experiment fails without any identification problem at all: ratio metrics, correlated dashboards, and interference between users.

Notebooks

References

• Wald, A. (1945). Sequential Tests of Statistical Hypotheses. The Annals of Mathematical Statistics 16(2), 117–186. — the sequential probability ratio test
• Johari, R., Koomen, P., Pekelis, L. & Walsh, D. (2022). Always Valid Inference: Continuous Monitoring of A/B Tests. Operations Research 70(3), 1806–1821. — always-valid inference for A/B tests
• Howard, S. R., Ramdas, A., McAuliffe, J. & Sekhon, J. (2021). Time-uniform, nonparametric, nonasymptotic confidence sequences. The Annals of Statistics 49(2). — time-uniform confidence sequences