Causal Inference I(e) — A/B Testing: Ratio Metrics and the Delta Method (R companion)¶
Delta-method variance, the cluster bootstrap, and the Sample Ratio Mismatch check in base R¶
This companion reproduces the Python notebook in base R: it shows the naive session-level standard error under-covering a ratio metric (CTR = clicks / sessions) when the experiment randomizes users, fixes it with the delta method at the user level (cross-checked by a cluster bootstrap), and runs the Sample Ratio Mismatch (SRM) guardrail with chisq.test. No special packages — the point is that the correct analysis is a few lines once you analyze at the randomization unit.
1. The analysis-unit problem¶
Randomize by user, measure CTR over sessions. Sessions within a user are correlated (a user random effect), so the session-level SE $\sqrt{p(1-p)/N_{\text{sessions}}}$ understates the true variance — the effective sample size is closer to the number of users.
set.seed(1)
gen_arm<-function(n_users, lift=0){
s<-rpois(n_users,4)+1; u<-rnorm(n_users,0,0.8)
p<-pmin(pmax(1/(1+exp(-(-0.4+u)))+lift,0),1); c<-rbinom(n_users,s,p); list(c=c,s=s) }
a<-gen_arm(3000); p<-sum(a$c)/sum(a$s); naive_se<-sqrt(p*(1-p)/sum(a$s))
cat(sprintf("Example arm: %d users, %d sessions, CTR = %.3f\n", length(a$c), sum(a$s), p))
cat(sprintf(" naive session-level SE = %.4f (pretends %d independent sessions)\n", naive_se, sum(a$s)))
options(repr.plot.width=13, repr.plot.height=4.2); par(mfrow=c(1,2))
hist(a$s, breaks=0:20, col="#2b6cb0", main="Sessions per user", xlab="sessions")
hist(a$c/a$s, breaks=25, col="#6b46c1", main="Per-user click rates vary widely", xlab="per-user click rate"); abline(v=p, col="#c53030", lwd=2)
par(mfrow=c(1,1))
Example arm: 3000 users, 14901 sessions, CTR = 0.419
naive session-level SE = 0.0040 (pretends 14901 independent sessions)
2. The delta method and cluster bootstrap¶
The arm metric $R=\bar c/\bar s$ is a ratio of per-user means; the delta method gives its user-level variance $$\widehat{\operatorname{Var}}(R)=\frac{1}{n\,\bar s^2}\left[\operatorname{Var}(c)-2R\operatorname{Cov}(c,s)+R^2\operatorname{Var}(s)\right].$$ We validate it against a cluster bootstrap (resample whole users) and confirm 95% CI coverage against the naive SE.
ratio_se<-function(c,s){ n<-length(c); R<-sum(c)/sum(s); sbar<-mean(s)
v<-(var(c)-2*R*cov(c,s)+R^2*var(s))/(sbar^2*n); c(R=R, se=sqrt(v)) }
cluster_boot<-function(c,s,B=500){ n<-length(c); rs<-replicate(B,{i<-sample(n,n,replace=TRUE); sum(c[i])/sum(s[i])}); sd(rs) }
rs<-ratio_se(a$c,a$s); seb<-cluster_boot(a$c,a$s)
cat(sprintf("SE of arm CTR: naive %.4f | delta method %.4f (%.2fx) | cluster bootstrap %.4f\n",
naive_se, rs["se"], rs["se"]/naive_se, seb))
# coverage over sims (true lift 0.02) -- name it covsim, NOT cov (base cov() is used inside ratio_se)
covsim<-function(lift=0.02,n=3000,nsim=1200){ cd<-cn<-0
for(k in 1:nsim){ t<-gen_arm(n,lift); ctl<-gen_arm(n,0)
rt<-ratio_se(t$c,t$s); rc<-ratio_se(ctl$c,ctl$s); d<-unname(rt["R"]-rc["R"]); sed<-unname(sqrt(rt["se"]^2+rc["se"]^2))
pt<-sum(t$c)/sum(t$s); pc<-sum(ctl$c)/sum(ctl$s); sen<-sqrt(pt*(1-pt)/sum(t$s)+pc*(1-pc)/sum(ctl$s))
cd<-cd+(abs(d-lift)<1.96*sed); cn<-cn+(abs((pt-pc)-lift)<1.96*sen) }
c(delta=cd/nsim, naive=cn/nsim) }
cvg<-covsim()
cat(sprintf("95%% CI coverage (true lift 0.02): delta method %.3f (calibrated) | naive %.3f (under-covers)\n", cvg["delta"], cvg["naive"]))
options(repr.plot.width=13, repr.plot.height=4.2); par(mfrow=c(1,2))
barplot(c(naive=naive_se, delta=rs["se"], bootstrap=seb), col=c("#c53030","#2f855a","#2b6cb0"), ylab="SE of arm CTR", main="Naive SE too small; delta = bootstrap")
bp<-barplot(100*c(delta=cvg["delta"], naive=cvg["naive"]), col=c("#2f855a","#c53030"), ylab="95% CI coverage (%)", main="Only the delta method covers", ylim=c(0,100)); abline(h=95,lty=2)
text(bp, 100*c(cvg["delta"],cvg["naive"])-5, sprintf("%.0f%%",100*c(cvg["delta"],cvg["naive"])), col="white", font=2)
par(mfrow=c(1,1))
SE of arm CTR: naive 0.0040 | delta method 0.0051 (1.27x) | cluster bootstrap 0.0054
95% CI coverage (true lift 0.02): delta method 0.949 (calibrated) | naive 0.864 (under-covers)
3. The Sample Ratio Mismatch guardrail — chisq.test¶
Check the arm counts against the intended split with a chi-square goodness-of-fit test; alarm at a very small p-value ($p<0.001$). A failing SRM signals broken assignment/logging and a likely-biased effect.
srm<-function(nA,nB,pA=0.5){ ch<-chisq.test(c(nA,nB), p=c(pA,1-pA)); c(chi2=unname(ch$statistic), p=ch$p.value) }
cases<-list(c(50120,49880), c(50250,49750), c(50600,49400)); nm<-c("healthy 50/50","mild skew","corrupted")
cat("SRM chi-square test (intended 50/50; alarm if p < 0.001):\n")
ps<-numeric(3)
for(i in 1:3){ r<-srm(cases[[i]][1],cases[[i]][2]); ps[i]<-r["p"]
cat(sprintf(" %-14s %d/%d: chi2=%6.2f, p=%.4g %s\n", nm[i], cases[[i]][1], cases[[i]][2], r["chi2"], r["p"],
if(r["p"]>=0.001) "PASS" else "FAIL -- investigate")) }
options(repr.plot.width=8, repr.plot.height=4)
cols<-ifelse(ps>=0.001,"#2f855a","#c53030")
barplot(-log10(pmax(ps,1e-12)), names.arg=nm, col=cols, ylab="-log10(p-value)", main="SRM guardrail: a broken split fails the check")
abline(h=-log10(0.001), lty=2)
SRM chi-square test (intended 50/50; alarm if p < 0.001):
healthy 50/50 50120/49880: chi2= 0.58, p=0.4479 PASS mild skew 50250/49750: chi2= 2.50, p=0.1138 PASS corrupted 50600/49400: chi2= 14.40, p=0.0001478 FAIL -- investigate
4. Summary¶
R reproduced the ratio-metric inference story: the naive session-level SE was ~1.25x too small and its 95% intervals under-covered, while the delta method at the user level (matching a cluster bootstrap) restored nominal coverage; and the SRM chisq.test flagged a corrupted 50/50 split. The rule is simple and universal: analyze ratio metrics at the randomization unit (delta method / cluster bootstrap / cluster-robust SE), and SRM-check every experiment first.
Cross-links: the clustering is the design-effect lesson of Noncompliance & Cluster Designs (1c) applied to a ratio estimand; combine with CUPED (1b) to also shrink the variance; the delta method recurs for nonlinear estimands like RMST in Causal Survival (subsection 10).