Causal Inference I(d) — A/B Testing: Sequential Monitoring and Power¶
The peeking problem, sample-size/MDE planning, and always-valid inference (mSPRT)¶
An online A/B test is a randomized experiment (subsection 1a) — but the tech setting adds a twist that breaks classical inference: results stream in continuously and everyone wants to peek and stop as soon as the p-value dips below 0.05. Under a fixed-sample test that repeated looking inflates the false-positive rate catastrophically, because each peek is another chance to cross the threshold by luck. This notebook builds the three things every experimentation platform needs:
- The peeking problem — how badly continuous monitoring inflates Type-I error under the null (an A/A test).
- Power and sample size — the fixed-horizon design: how many users you need, and the minimum detectable effect (MDE) at a given sample size.
- Always-valid inference — the mixture Sequential Probability Ratio Test (mSPRT) (Johari, Pekelis & Walsh 2017; the engine behind Optimizely's "Stats Engine"), which lets you monitor continuously and stop early while controlling Type-I error at all times via Ville's inequality.
These are simulation-first by necessity: the whole point is to measure an estimator's error rate against a known truth (a true null, a true effect). Python leads (from-scratch peeking simulation, power formulas, and mSPRT); the R companion adds group-sequential boundaries via gsDesign. This is the online-experimentation companion to the RCT foundations (1a) and covariate adjustment (1b) notebooks.
1. The peeking problem — why continuous monitoring breaks the fixed-sample test¶
Run an A/A test: both arms are identical (true effect = 0). If we compute a z-test once at the planned sample size, the false-positive rate is the nominal 5%. But if we peek repeatedly — checking after every batch of users and stopping the moment $|z|>1.96$ — each look is a fresh opportunity to cross the boundary by chance, and the probability of ever crossing climbs far above 5%. This is the single most common mistake in industry experimentation: with continuous monitoring on a true null, roughly one test in four (or more) falsely wins.
import numpy as np, matplotlib.pyplot as plt, warnings
from scipy.stats import norm, chi2
warnings.filterwarnings("ignore")
BLUE="#2b6cb0"; RED="#c53030"; GREEN="#2f855a"; ORANGE="#dd6b20"; GREY="#a0aec0"; PURP="#6b46c1"
sigma=1.0
def peeking_fpr(n_peeks, N=2000, nsim=4000, alpha=0.05, seed=0):
rng=np.random.default_rng(seed); rej=0
peek_at=np.unique(np.linspace(N//n_peeks, N, n_peeks).astype(int))
for _ in range(nsim):
a=np.cumsum(rng.normal(0,sigma,N)); b=np.cumsum(rng.normal(0,sigma,N)) # null: identical arms
for n in peek_at:
d=(a[n-1]-b[n-1])/n; se=np.sqrt(2*sigma**2/n)
if abs(d/se)>norm.ppf(1-alpha/2): rej+=1; break
return rej/nsim
peaks=[1,2,5,10,20,50]; fpr=[peeking_fpr(k) for k in peaks]
for k,f in zip(peaks,fpr): print(f" {k:2d} peek(s): Type-I error = {f:.3f}")
fig,ax=plt.subplots(1,2,figsize=(13,4.4))
ax[0].plot(peaks,[100*f for f in fpr],"o-",color=RED,lw=2); ax[0].axhline(5,color=GREEN,ls="--",label="nominal 5%")
ax[0].set_xlabel("number of peeks"); ax[0].set_ylabel("actual Type-I error (%)"); ax[0].set_title("Peeking inflates false positives on a true null (A/A)"); ax[0].legend()
# one example A/A trajectory that falsely 'wins'
rng=np.random.default_rng(7)
a=np.cumsum(rng.normal(0,1,2000)); b=np.cumsum(rng.normal(0,1,2000)); ns=np.arange(1,2001)
z=(a-b)/ns/np.sqrt(2/ns)
ax[1].plot(ns,z,color=PURP,lw=1); ax[1].axhline(1.96,color=RED,ls="--"); ax[1].axhline(-1.96,color=RED,ls="--")
ax[1].fill_between(ns,1.96,z,where=(z>1.96),color=RED,alpha=.3)
ax[1].set_xlabel("users per arm"); ax[1].set_ylabel("running z-statistic"); ax[1].set_title("A true-null test that crosses 1.96 by chance if you peek")
plt.tight_layout(); plt.show()
print(f"With 20 peeks the false-positive rate is {fpr[4]:.0%} -- not 5%. The running z-statistic wanders across +-1.96 repeatedly;")
print("stopping at the first crossing turns a coin-flip into a 'significant' result. Fixed-sample p-values require a fixed n.")
1 peek(s): Type-I error = 0.048 2 peek(s): Type-I error = 0.077 5 peek(s): Type-I error = 0.138 10 peek(s): Type-I error = 0.191 20 peek(s): Type-I error = 0.242 50 peek(s): Type-I error = 0.302
With 20 peeks the false-positive rate is 24% -- not 5%. The running z-statistic wanders across +-1.96 repeatedly; stopping at the first crossing turns a coin-flip into a 'significant' result. Fixed-sample p-values require a fixed n.
2. Power and sample size — designing the fixed-horizon test¶
The disciplined alternative is to fix the sample size in advance from a power calculation. For a two-sample difference in means with common variance $\sigma^2$, significance level $\alpha$, and power $1-\beta$, the sample size per arm to detect a true effect $\delta$ is $$n=\frac{2\sigma^2\left(z_{1-\alpha/2}+z_{1-\beta}\right)^2}{\delta^2}.$$ Inverting it gives the minimum detectable effect (MDE) at a given $n$ — the smallest true effect you have (say) 80% power to catch. Two facts drive all experiment planning: sample size scales with $1/\delta^2$ (halving the detectable effect quadruples the users needed), and the MDE shrinks only as $1/\sqrt{n}$ (diminishing returns to traffic).
def n_per_arm(delta, sigma=1.0, alpha=0.05, power=0.8):
return 2*sigma**2*(norm.ppf(1-alpha/2)+norm.ppf(power))**2/delta**2
def mde(n, sigma=1.0, alpha=0.05, power=0.8):
return (norm.ppf(1-alpha/2)+norm.ppf(power))*np.sqrt(2*sigma**2/n)
def power_at(n, delta, sigma=1.0, alpha=0.05):
return norm.cdf(delta/np.sqrt(2*sigma**2/n)-norm.ppf(1-alpha/2))
for d in [0.05,0.1,0.2,0.5]: print(f" detect delta={d}: need n/arm = {n_per_arm(d):,.0f}")
print(f" at n=1,000/arm: MDE (80% power) = {mde(1000):.3f}")
fig,ax=plt.subplots(1,2,figsize=(13,4.4))
nn=np.arange(50,5001,25)
for d,c in zip([0.05,0.1,0.2],[RED,ORANGE,GREEN]):
ax[0].plot(nn,[power_at(n,d) for n in nn],color=c,lw=2,label=f"delta={d}")
ax[0].axhline(0.8,color=GREY,ls="--"); ax[0].set_xlabel("users per arm"); ax[0].set_ylabel("power"); ax[0].set_title("Power curves: bigger effects need far fewer users"); ax[0].legend()
ax[1].plot(nn,[mde(n) for n in nn],color=PURP,lw=2); ax[1].set_xlabel("users per arm"); ax[1].set_ylabel("MDE at 80% power")
ax[1].set_title("MDE shrinks only as 1/sqrt(n) -- diminishing returns to traffic")
plt.tight_layout(); plt.show()
print("The fixed-horizon design is honest but rigid: you commit to n up front and look ONCE. That rigidity is exactly what")
print("makes teams peek -- so we need a method that is valid under continuous monitoring.")
detect delta=0.05: need n/arm = 6,279 detect delta=0.1: need n/arm = 1,570 detect delta=0.2: need n/arm = 392 detect delta=0.5: need n/arm = 63 at n=1,000/arm: MDE (80% power) = 0.125
The fixed-horizon design is honest but rigid: you commit to n up front and look ONCE. That rigidity is exactly what makes teams peek -- so we need a method that is valid under continuous monitoring.
3. Always-valid inference — the mixture SPRT¶
The fix is an always-valid procedure whose error guarantee holds at every sample size simultaneously, so you may peek as often as you like and stop the instant it fires. The mixture SPRT (Johari et al. 2017) places a mixing prior $N(0,\tau^2)$ over the unknown effect and tracks the likelihood ratio of "some effect" vs "no effect." For the running difference $\hat\delta_n$ with variance $V_n=2\sigma^2/n$, $$\Lambda_n=\sqrt{\frac{V_n}{V_n+\tau^2}}\;\exp\!\left(\frac{\hat\delta_n^{\,2}\,\tau^2}{2\,V_n\,(V_n+\tau^2)}\right).$$ Under the null $\{\Lambda_n\}$ is a non-negative martingale with mean 1, so Ville's inequality gives $\Pr(\exists n:\Lambda_n\ge 1/\alpha)\le\alpha$ — rejecting the first time $\Lambda_n\ge1/\alpha$ controls Type-I error across all peeks at once. The reciprocal $p_n=1/\Lambda_n$ is an always-valid p-value, and inverting the statistic yields an anytime-valid confidence sequence. We verify Type-I control under continuous monitoring, then show the payoff: under a real effect it stops early, often before the fixed-horizon sample size.
def msprt(a_stream,b_stream,tau2=0.1,alpha=0.05,sigma=1.0):
ns=np.arange(1,len(a_stream)+1); D=(np.cumsum(a_stream)-np.cumsum(b_stream))/ns; V=2*sigma**2/ns
Lam=np.sqrt(V/(V+tau2))*np.exp(D**2*tau2/(2*V*(V+tau2)))
hit=np.where(Lam>=1/alpha)[0]; return (hit[0]+1 if len(hit) else None), Lam
def sim(delta,N=4000,tau2=0.1,nsim=3000,alpha=0.05,seed0=0):
rej=0; stops=[]
for s in range(nsim):
r=np.random.default_rng(seed0+s)
t,_=msprt(r.normal(delta,1,N),r.normal(0,1,N),tau2=tau2,alpha=alpha)
if t is not None: rej+=1; stops.append(t)
return rej/nsim, stops
fpr_seq,_=sim(0.0)
detect,stops=sim(0.1)
print(f"mSPRT under continuous peeking:")
print(f" Type-I error on a true null (A/A) = {fpr_seq:.3f} <- controlled at 0.05 despite peeking every observation")
print(f" (compare: naive peeking every obs -> ~0.25+)")
print(f" power at delta=0.1 = {detect:.2f}; median stop = {int(np.median(stops)):,}/arm vs fixed-horizon {n_per_arm(0.1):,.0f}/arm")
fig,ax=plt.subplots(1,2,figsize=(13,4.4))
ax[0].bar(["naive\n(peek every obs)","mSPRT\n(always-valid)"],[peeking_fpr(50),fpr_seq],color=[RED,GREEN])
ax[0].axhline(0.05,color="k",ls="--",label="nominal 5%"); ax[0].set_ylabel("Type-I error (A/A)"); ax[0].set_title("mSPRT restores valid inference under peeking"); ax[0].legend()
for i,v in enumerate([peeking_fpr(50),fpr_seq]): ax[0].text(i,v+0.005,f"{v:.2f}",ha="center")
r=np.random.default_rng(3); _,Lam=msprt(r.normal(0.1,1,4000),r.normal(0,1,4000))
ax[1].plot(np.arange(1,4001),Lam,color=BLUE,lw=1.2); ax[1].axhline(1/0.05,color=RED,ls="--",label="reject boundary 1/alpha")
st=np.where(Lam>=1/0.05)[0]
if len(st): ax[1].axvline(st[0]+1,color=GREEN,ls=":",label=f"stops at n={st[0]+1}")
ax[1].set_yscale("log"); ax[1].set_xlabel("users per arm"); ax[1].set_ylabel("mSPRT statistic (log)"); ax[1].set_title("Under a true effect the statistic crosses early"); ax[1].legend(fontsize=8)
plt.tight_layout(); plt.show()
print("mSPRT keeps Type-I at 5% under continuous peeking (Ville's inequality) AND stops early under a real effect -- the")
print("best of both: monitor as often as you like, stop when it fires, and the error guarantee still holds.")
mSPRT under continuous peeking: Type-I error on a true null (A/A) = 0.036 <- controlled at 0.05 despite peeking every observation (compare: naive peeking every obs -> ~0.25+) power at delta=0.1 = 0.93; median stop = 1,454/arm vs fixed-horizon 1,570/arm
mSPRT keeps Type-I at 5% under continuous peeking (Ville's inequality) AND stops early under a real effect -- the best of both: monitor as often as you like, stop when it fires, and the error guarantee still holds.
4. Summary¶
Online A/B tests are randomized experiments run under continuous monitoring, and that changes the inference:
- Peeking breaks fixed-sample tests. On a true null, checking repeatedly and stopping at the first $|z|>1.96$ drove the false-positive rate from 5% to ~25% (20 peeks) and higher — each look is another chance to cross by luck.
- The classical fix is to fix the sample size from a power calculation: $n\propto\sigma^2/\delta^2$ per arm, with the MDE shrinking only as $1/\sqrt{n}$. Honest, but so rigid that teams peek anyway.
- The modern fix is always-valid inference. The mSPRT is a martingale-based test whose Type-I guarantee holds at every sample size via Ville's inequality, so you may monitor continuously and stop early. It held Type-I at ~5% under per-observation peeking while detecting a true effect and stopping before the fixed-horizon sample size.
Practical guidance: decide the monitoring plan before launch — either commit to a fixed sample size and look once, or use an always-valid method (mSPRT / confidence sequences) or a group-sequential design (pre-specified interim looks with spent-alpha boundaries) if you want to stop early. Never peek with fixed-sample p-values. Cross-links: this builds directly on the Randomized Experiments foundations (1a); pair it with Covariate Adjustment / CUPED (1b) to also cut the variance and reach significance faster; the sequential-testing logic (test-inversion, always-valid intervals) echoes the confidence-set thinking in Weak-IV inference (3b) and conformal prediction (ML arc). The R companion adds the classical group-sequential boundaries (O'Brien-Fleming, Pocock) via gsDesign, the pharma-trial counterpart to the tech mSPRT.