Causal Inference I(d) — A/B Testing: Sequential Monitoring and Power (R companion)¶
The peeking problem, power.t.test, and group-sequential boundaries with gsDesign¶
This companion reproduces the Python notebook's peeking simulation and power planning in base R, then brings in the classical sequential-testing framework from clinical trials — group-sequential designs via gsDesign (Anderson). Where the Python notebook used the tech-industry mSPRT (continuous monitoring), the pharma tradition pre-specifies a handful of interim analyses and spends the Type-I budget across them with O'Brien-Fleming or Pocock boundaries. Both solve the same problem — valid early stopping — and gsDesign is the reference tool for the group-sequential version.
1. The peeking problem¶
An A/A test (true effect = 0) monitored repeatedly and stopped at the first $|z|>1.96$ rejects far more than 5% of the time — each look is another chance to cross by luck. We reproduce the Python inflation curve in base R.
set.seed(0); sigma<-1
peeking_fpr<-function(n_peeks,N=2000,nsim=3000,alpha=0.05){
peek<-unique(round(seq(N/n_peeks,N,length.out=n_peeks))); rej<-0
for(i in 1:nsim){ a<-cumsum(rnorm(N,0,sigma)); b<-cumsum(rnorm(N,0,sigma))
for(n in peek){ z<-((a[n]-b[n])/n)/sqrt(2*sigma^2/n); if(abs(z)>qnorm(1-alpha/2)){rej<-rej+1; break} } }
rej/nsim }
peaks<-c(1,2,5,10,20,50); fpr<-sapply(peaks,peeking_fpr)
for(i in seq_along(peaks)) cat(sprintf(" %2d peek(s): Type-I error = %.3f\n", peaks[i], fpr[i]))
options(repr.plot.width=7.5, repr.plot.height=4.4)
plot(peaks, 100*fpr, type="b", pch=19, col="#c53030", lwd=2, xlab="number of peeks", ylab="actual Type-I error (%)",
main="Peeking inflates false positives on a true null"); abline(h=5, col="#2f855a", lty=2)
legend("topleft", c("naive repeated z-test","nominal 5%"), col=c("#c53030","#2f855a"), lty=c(1,2), pch=c(19,NA), bty="n")
1 peek(s): Type-I error = 0.047 2 peek(s): Type-I error = 0.082 5 peek(s): Type-I error = 0.150 10 peek(s): Type-I error = 0.185 20 peek(s): Type-I error = 0.241 50 peek(s): Type-I error = 0.313
2. Power and sample size — power.t.test¶
The fixed-horizon design fixes $n$ from a power calculation; base R's power.t.test inverts the same formula $n=2\sigma^2(z_{1-\alpha/2}+z_{1-\beta})^2/\delta^2$ used in Python. We list sample sizes for a range of effects and the MDE at a fixed $n$.
for(d in c(0.05,0.1,0.2,0.5)){
n<-power.t.test(delta=d, sd=1, sig.level=0.05, power=0.8)$n
cat(sprintf(" detect delta=%.2f: need n/arm = %.0f\n", d, n)) }
mde<-power.t.test(n=1000, sd=1, sig.level=0.05, power=0.8)$delta
cat(sprintf(" at n=1,000/arm: MDE (80%% power) = %.3f\n", mde))
options(repr.plot.width=7.5, repr.plot.height=4.4)
nn<-seq(50,5000,25)
pw<-function(n,d) power.t.test(n=n, delta=d, sd=1, sig.level=0.05)$power
plot(nn, sapply(nn,pw,d=0.2), type="l", col="#2f855a", lwd=2, ylim=c(0,1), xlab="users per arm", ylab="power", main="Power curves by true effect")
lines(nn, sapply(nn,pw,d=0.1), col="#dd6b20", lwd=2); lines(nn, sapply(nn,pw,d=0.05), col="#c53030", lwd=2); abline(h=0.8, lty=2, col="grey")
legend("bottomright", c("delta=0.2","delta=0.1","delta=0.05"), col=c("#2f855a","#dd6b20","#c53030"), lwd=2, bty="n")
detect delta=0.05: need n/arm = 6280 detect delta=0.10: need n/arm = 1571 detect delta=0.20: need n/arm = 393 detect delta=0.50: need n/arm = 64
at n=1,000/arm: MDE (80% power) = 0.125
3. Group-sequential boundaries — gsDesign¶
The clinical-trial answer to peeking is a group-sequential design: pre-specify $K$ interim analyses and use stopping boundaries that spend the 5% Type-I budget across the looks so the overall error stays at 5%. O'Brien-Fleming boundaries are very stringent early (hard to stop at the first look, boundary near the end ~1.96) and are the standard; Pocock boundaries are constant across looks (easier early stopping, but a larger final critical value). gsDesign computes the boundaries and the expected sample size under the alternative — the early-stopping payoff. A quick simulation confirms that testing at the 5 interim looks with the O'Brien-Fleming boundary holds Type-I at ~5%, unlike naive repeated testing.
suppressMessages(library(gsDesign))
K<-5
of<-gsDesign(k=K, test.type=2, sfu="OF", alpha=0.025, beta=0.2)
po<-gsDesign(k=K, test.type=2, sfu="Pocock", alpha=0.025, beta=0.2)
cat("Two-sided z-boundaries across", K, "interim looks:\n")
cat(" O'Brien-Fleming:", paste(sprintf("%.2f",of$upper$bound),collapse=" "), "\n")
cat(" Pocock :", paste(sprintf("%.2f",po$upper$bound),collapse=" "), "\n")
cat(sprintf("\nInflation factor over a fixed test (max sample size): OF %.3f, Pocock %.3f\n", of$n.I[K], po$n.I[K]))
# simulate Type-I with OF boundary at the 5 equally-spaced looks
set.seed(1); N<-2000; looks<-round(seq(N/K,N,length.out=K)); bnd<-of$upper$bound
rej<-mean(replicate(3000,{ a<-cumsum(rnorm(N)); b<-cumsum(rnorm(N))
any(sapply(seq_len(K), function(j){ n<-looks[j]; abs(((a[n]-b[n])/n)/sqrt(2/n))>bnd[j] })) }))
cat(sprintf("\nSimulated Type-I with OF boundary at 5 looks (A/A) = %.3f (target 0.05; naive 5-look = %.3f)\n", rej, peeking_fpr(5)))
options(repr.plot.width=8, repr.plot.height=4.4)
plot(1:K, of$upper$bound, type="b", pch=19, col="#2b6cb0", lwd=2, ylim=c(1.8,3.2), xlab="interim look", ylab="z-boundary",
main="Group-sequential stopping boundaries (gsDesign)")
lines(1:K, po$upper$bound, type="b", pch=17, col="#6b46c1", lwd=2); abline(h=1.96, lty=2, col="grey")
legend("topright", c("O'Brien-Fleming","Pocock","fixed 1.96"), col=c("#2b6cb0","#6b46c1","grey"), lwd=2, pch=c(19,17,NA), bty="n")
Warning message: "package 'gsDesign' was built under R version 4.6.1"
Two-sided z-boundaries across 5 interim looks:
O'Brien-Fleming: 4.56 3.23 2.63 2.28 2.04
Pocock : 2.41 2.41 2.41 2.41 2.41
Inflation factor over a fixed test (max sample size): OF 1.028, Pocock 1.229
Simulated Type-I with OF boundary at 5 looks (A/A) = 0.043 (target 0.05; naive 5-look = 0.141)
4. Summary¶
R reproduced the peeking inflation and the power.t.test sample-size/MDE planning of the Python notebook, then added the group-sequential tradition via gsDesign: pre-specified interim looks with O'Brien-Fleming or Pocock boundaries that spend the Type-I budget so the overall false-positive rate stays at 5%, with a modest inflation of the maximum sample size in exchange for the option to stop early. The simulation confirmed the O'Brien-Fleming boundary holds Type-I at ~5% across five looks where naive repeated testing does not.
Two traditions, one goal: valid early stopping. The tech-industry mSPRT / confidence sequences (Python notebook) allow continuous monitoring; the clinical-trial group-sequential designs (here) allow a pre-specified number of interim looks — choose continuous monitoring when data stream in and you want to stop anytime, group-sequential when analyses are scheduled (as in trials). Cross-links: the foundations are the Randomized Experiments notebook (1a); combine with CUPED (1b) to hit significance with fewer users; the always-valid/boundary logic parallels the test-inversion intervals of Weak-IV inference (3b).