Causal Inference I(f) — A/B Testing: Interference and Switchback Designs (R companion)¶
SUTVA violations, cluster randomization, and switchbacks in base R¶
This companion reproduces the Python notebook: a linear-in-means interference model in which the user-level A/B measures only the direct effect and misses the spillover (biased for the total policy effect), the cluster-randomization fix that recovers the total effect at the cost of variance, and the switchback design for temporal carryover whose validity depends on the window length. All base R — the lessons are about design, not packages.
1. The SUTVA violation¶
Outcome depends on own treatment and the treated fraction in the market: $Y_i=\alpha+\beta T_i+\gamma f_m+\varepsilon_i$. The total (policy) effect of shipping to everyone ($f_m:0\to1$) is $\beta+\gamma$. A within-market 50/50 A/B holds $f_m\approx0.5$ in both arms, so $\gamma f_m$ cancels and the estimate captures only the direct effect $\beta$.
set.seed(0); alpha<-2; beta<-1; gamma<--0.6 # total effect = beta+gamma = 0.4
M<-200; per<-50
user_AB<-function(seed){ set.seed(seed); Y<-c(); Tr<-c()
for(m in 1:M){ t<-rbinom(per,1,0.5); f<-mean(t); Y<-c(Y,alpha+beta*t+gamma*f+rnorm(per)); Tr<-c(Tr,t) }
mean(Y[Tr==1])-mean(Y[Tr==0]) }
naive<-sapply(1:400,user_AB)
cat(sprintf("TRUE effects: direct beta=%.2f spillover gamma=%.2f TOTAL (policy)=beta+gamma=%.2f\n", beta, gamma, beta+gamma))
cat(sprintf(" naive user-level A/B = %.3f (SD %.3f) -> recovers DIRECT effect, %.1fx too large for the policy question\n",
mean(naive), sd(naive), mean(naive)/(beta+gamma)))
options(repr.plot.width=13, repr.plot.height=4.2); par(mfrow=c(1,2))
hist(naive, breaks=30, col="grey", main="Naive A/B centers on the DIRECT effect", xlab="user-level A/B estimate")
abline(v=beta+gamma, col="#2f855a", lwd=2, lty=2); abline(v=beta, col="#c53030", lwd=2, lty=3)
legend("topright", c("TRUE total 0.4","direct 1.0"), col=c("#2f855a","#c53030"), lty=c(2,3), lwd=2, bty="n")
fr<-seq(0,1,0.02); plot(fr, alpha+beta+gamma*fr, type="l", col="#2f855a", lwd=2, ylim=c(1,3.2), xlab="fraction treated", ylab="expected outcome", main="Spillover lowers everyone's outcome")
lines(fr, alpha+gamma*fr, col="#2b6cb0", lwd=2); legend("topright", c("treated","control"), col=c("#2f855a","#2b6cb0"), lwd=2, bty="n")
par(mfrow=c(1,1))
TRUE effects: direct beta=1.00 spillover gamma=-0.60 TOTAL (policy)=beta+gamma=0.40
naive user-level A/B = 0.986 (SD 0.021) -> recovers DIRECT effect, 2.5x too large for the policy question
2. Cluster randomization recovers the total effect¶
Randomize whole markets: treated markets have $f_m=1$, control markets $f_m=0$, so the comparison moves the treated fraction the full 0→1 and captures $\beta+\gamma$. The cost is variance — the effective sample size is the number of markets.
cluster_rand<-function(seed){ set.seed(seed); tm<-rbinom(M,1,0.5); mb<-numeric(M)
for(m in 1:M){ f<-tm[m]; mb[m]<-mean(alpha+beta*tm[m]+gamma*f+rnorm(per)) }
mean(mb[tm==1])-mean(mb[tm==0]) }
clust<-sapply(1:400,cluster_rand)
cat(sprintf("TRUE total effect = %.2f\n", beta+gamma))
cat(sprintf(" naive user-level A/B = %.3f (bias %+.3f, SD %.3f)\n", mean(naive), mean(naive)-(beta+gamma), sd(naive)))
cat(sprintf(" cluster (market) rand. = %.3f (bias %+.3f, SD %.3f) -> unbiased, %.1fx the SD\n",
mean(clust), mean(clust)-(beta+gamma), sd(clust), sd(clust)/sd(naive)))
options(repr.plot.width=8.5, repr.plot.height=4.4)
hist(naive, breaks=30, col=rgb(197,48,48,140,max=255), xlim=range(c(naive,clust)), main="Cluster randomization trades bias for variance", xlab="estimated total effect")
hist(clust, breaks=30, col=rgb(47,133,90,150,max=255), add=TRUE); abline(v=beta+gamma, col="black", lwd=2, lty=2)
legend("topright", c("user-level (biased)","cluster (unbiased)","TRUE total"), fill=c(rgb(197,48,48,140,max=255),rgb(47,133,90,150,max=255),NA), border=NA, lty=c(NA,NA,2), col=c(NA,NA,"black"), bty="n")
TRUE total effect = 0.40
naive user-level A/B = 0.986 (bias +0.586, SD 0.021)
cluster (market) rand. = 0.399 (bias -0.001, SD 0.019) -> unbiased, 0.9x the SD
3. Switchback designs for temporal interference¶
When treatment carries over in time, randomize whole-system on/off time windows. The estimate recovers the total effect only when the window exceeds the carryover horizon and the first period(s) of each window are dropped as burn-in; too-short windows measure only the direct effect (like the naive A/B).
Tn<-6000
switchback<-function(window, burn, seed){ set.seed(seed); n_win<-Tn%/%window; w<-rbinom(n_win,1,0.5)
on<-rep(w, each=window)[1:Tn]; carry<-c(0, on[-Tn])
y<-alpha+beta*on+gamma*carry+rnorm(Tn)
pos<-rep(0:(window-1), n_win)[1:Tn]; keep<-pos>=burn
mean(y[keep & on==1])-mean(y[keep & on==0]) }
configs<-list(c(1,0),c(2,1),c(5,1),c(10,1),c(20,2))
cat(sprintf("TRUE total (steady-state) effect = %.2f; direct = %.2f; carryover = 1 period\n", beta+gamma, beta))
ws<-c(); es<-c()
for(cf in configs){ e<-mean(sapply(1:200, function(s) switchback(cf[1],cf[2],s))); ws<-c(ws,cf[1]); es<-c(es,e)
tag<-if(abs(e-(beta+gamma))<abs(e-beta)) "recovers TOTAL" else "biased toward DIRECT"
cat(sprintf(" window=%2d, burn-in=%d: estimate %.3f (%s)\n", cf[1], cf[2], e, tag)) }
options(repr.plot.width=8.5, repr.plot.height=4.4)
plot(ws, es, type="b", pch=19, col="#6b46c1", lwd=2, ylim=c(0.2,1.1), xlab="switchback window length", ylab="estimated effect",
main="Windows must exceed the carryover horizon (with burn-in)")
abline(h=beta+gamma, col="#2f855a", lty=2, lwd=2); abline(h=beta, col="#c53030", lty=3, lwd=2)
legend("right", c("TRUE total 0.4","direct 1.0"), col=c("#2f855a","#c53030"), lty=c(2,3), lwd=2, bty="n")
TRUE total (steady-state) effect = 0.40; direct = 1.00; carryover = 1 period
window= 1, burn-in=0: estimate 0.996 (biased toward DIRECT) window= 2, burn-in=1: estimate 0.402 (recovers TOTAL) window= 5, burn-in=1: estimate 0.398 (recovers TOTAL) window=10, burn-in=1: estimate 0.404 (recovers TOTAL) window=20, burn-in=2: estimate 0.403 (recovers TOTAL)
4. Summary¶
R reproduced the interference story: the user-level A/B centered on the direct effect (1.0) and missed the spillover, overstating the total policy effect (0.4) by 2.5x; cluster randomization recovered the total effect unbiasedly at higher variance; and the switchback recovered it only with windows longer than the carryover horizon plus burn-in.
The discipline: name the estimand, then pick a design that contains the interference — user-level A/B for the direct effect, cluster randomization for spatial spillover, switchbacks for temporal carryover — and size for the reduced effective sample with cluster-robust inference. Cross-links: the identification-level counterpart to the standard-error clustering in Noncompliance & Cluster Designs (1c); the direct/total split parallels mediation (subsection 8); the estimand discipline echoes ITT vs CACE (1c).