Causal Inference I(e) — A/B Testing: Ratio Metrics and the Delta Method

When the randomization unit isn't the analysis unit — plus the Sample Ratio Mismatch guardrail

Most product metrics are ratios: click-through rate (clicks / sessions), revenue per session, conversion per visit. The catch is that experiments randomize users but these metrics are computed over sessions — so the randomization unit (user) differs from the analysis unit (session). Sessions from the same user are correlated, and a standard error that treats sessions as independent is too small, producing confidence intervals that under-cover and a stream of false "significant" results. This is one of the most common — and most invisible — inference bugs in industry experimentation.

This notebook builds the fix:

1. The analysis-unit problem — why naive session-level standard errors under-cover a ratio metric.
2. The delta method — linearize the ratio and compute its variance at the user level (validated against a cluster bootstrap), restoring correct coverage.
3. Sample Ratio Mismatch (SRM) — a one-line chi-square guardrail that catches a broken experiment (biased assignment or logging) before you trust any result.

Simulation-first, because seeing an SE be wrong requires a known truth. Python leads (from-scratch delta method, cluster bootstrap, SRM); the R companion mirrors it. This extends the Randomized Experiments (1a) and Noncompliance & Clusters (1c) notebooks — the clustering intuition of 1c, now applied to the ratio estimand that dominates real metrics.

1. The analysis-unit problem

We simulate an experiment randomized by user. Each user has a random number of sessions, and a user-level random effect makes their sessions correlated (some users click a lot, some rarely). The metric is CTR = total clicks / total sessions per arm — a ratio. The treatment lifts the per-session click probability by a true amount we control.

The tempting analysis treats every session as an independent Bernoulli trial and reports $\text{SE}=\sqrt{p(1-p)/N_{\text{sessions}}}$. Because sessions within a user are correlated, that formula understates the variance: the effective number of independent observations is closer to the number of users than the number of sessions.

import numpy as np, matplotlib.pyplot as plt, warnings
from scipy.stats import norm, chi2
warnings.filterwarnings("ignore")
BLUE="#2b6cb0"; RED="#c53030"; GREEN="#2f855a"; ORANGE="#dd6b20"; GREY="#a0aec0"; PURP="#6b46c1"
def gen_arm(n_users, lift=0.0, seed=0):
    r=np.random.default_rng(seed)
    s=r.poisson(4,n_users)+1                            # sessions per user (varies)
    u=r.normal(0,0.8,n_users)                           # user random effect -> within-user correlation
    p=np.clip(1/(1+np.exp(-(-0.4+u)))+lift, 0, 1)       # per-session click prob, lifted by treatment
    c=r.binomial(s,p)                                   # clicks per user
    return c,s
c,s=gen_arm(3000, lift=0.0, seed=1)
p=c.sum()/s.sum(); naive_se=np.sqrt(p*(1-p)/s.sum())
print(f"Example arm: {len(c):,} users, {s.sum():,} sessions, CTR = {p:.3f}")
print(f"  naive session-level SE = {naive_se:.4f}  (pretends {s.sum():,} independent sessions)")
print(f"  but sessions cluster within users -> effective n is closer to {len(c):,} users")
fig,ax=plt.subplots(1,2,figsize=(13,4.2))
ax[0].hist(s,bins=range(1,20),color=BLUE,alpha=.7); ax[0].set_xlabel("sessions per user"); ax[0].set_ylabel("users"); ax[0].set_title("Users contribute unequal numbers of sessions")
ctr_u=c/s
ax[1].hist(ctr_u,bins=25,color=PURP,alpha=.7); ax[1].axvline(p,color=RED,lw=2,label=f"pooled CTR {p:.2f}")
ax[1].set_xlabel("per-user click rate"); ax[1].set_ylabel("users"); ax[1].set_title("Per-user rates vary widely (over-dispersion)"); ax[1].legend()
plt.tight_layout(); plt.show()
print("Unequal sessions + heterogeneous per-user rates are exactly the conditions under which the session-level SE fails.")

Example arm: 3,000 users, 14,908 sessions, CTR = 0.405
  naive session-level SE = 0.0040  (pretends 14,908 independent sessions)
  but sessions cluster within users -> effective n is closer to 3,000 users

Unequal sessions + heterogeneous per-user rates are exactly the conditions under which the session-level SE fails.

2. The delta method for a ratio metric

The arm metric is a ratio of means, $R=\bar c/\bar s$, where $\bar c,\bar s$ are the per-user averages of clicks and sessions. The delta method linearizes $R$ around the means and gives its variance at the user level: $$\widehat{\operatorname{Var}}(R)=\frac{1}{n\,\bar s^{2}}\left[\operatorname{Var}(c)-2R\,\operatorname{Cov}(c,s)+R^{2}\operatorname{Var}(s)\right],$$ with $n$ the number of users. This correctly accounts for the fact that a user is the independent unit. The treatment-effect SE combines the two arms, $\sqrt{\widehat{\operatorname{Var}}(R_T)+\widehat{\operatorname{Var}}(R_C)}$. We validate it two ways: its 95% intervals cover at the nominal rate (where the naive ones under-cover), and it agrees with a cluster bootstrap that resamples whole users.

def ratio_se(c,s):
    n=len(c); R=c.sum()/s.sum(); sbar=s.mean()
    v=(np.var(c,ddof=1)-2*R*np.cov(c,s,ddof=1)[0,1]+R**2*np.var(s,ddof=1))/(sbar**2*n)
    return R, np.sqrt(v)
def cluster_boot_se(c,s,B=500,seed=0):
    r=np.random.default_rng(seed); n=len(c); rs=[]
    for _ in range(B):
        idx=r.integers(0,n,n); rs.append(c[idx].sum()/s[idx].sum())
    return np.std(rs)
# coverage check against known truth
def coverage(lift=0.02, n_users=3000, nsim=1500):
    cd=cn=0; ests=[]
    for k in range(nsim):
        ct,st=gen_arm(n_users,lift,2*k); cc,sc=gen_arm(n_users,0.0,2*k+1)
        Rt,et=ratio_se(ct,st); Rc,ec=ratio_se(cc,sc); d=Rt-Rc; sed=np.hypot(et,ec); ests.append(d)
        pt=ct.sum()/st.sum(); pc=cc.sum()/sc.sum()
        sen=np.hypot(np.sqrt(pt*(1-pt)/st.sum()), np.sqrt(pc*(1-pc)/sc.sum()))
        cd+= abs(d-lift)<1.96*sed; cn+= abs((pt-pc)-lift)<1.96*sen
    return cd/nsim, cn/nsim, np.mean(ests)
covd,covn,truth=coverage()
ct,st=gen_arm(3000,0.02,11); _,se_delta=ratio_se(ct,st); se_boot=cluster_boot_se(ct,st); pt=ct.sum()/st.sum(); se_naive=np.sqrt(pt*(1-pt)/st.sum())
print(f"Standard error of one arm's CTR:")
print(f"  naive (session-level) = {se_naive:.4f}")
print(f"  delta method (user)   = {se_delta:.4f}   ({se_delta/se_naive:.2f}x larger -- the honest SE)")
print(f"  cluster bootstrap     = {se_boot:.4f}   (agrees with the delta method)")
print(f"\n95% CI coverage of the treatment effect (true lift {truth:.3f}), 1500 sims:")
print(f"  delta method = {covd:.3f}  (calibrated)")
print(f"  naive        = {covn:.3f}  (UNDER-covers -> false positives)")
fig,ax=plt.subplots(1,2,figsize=(13,4.2))
ax[0].bar(["naive\nsession","delta\nmethod","cluster\nbootstrap"],[se_naive,se_delta,se_boot],color=[RED,GREEN,BLUE])
ax[0].set_ylabel("SE of arm CTR"); ax[0].set_title("Naive SE is too small; delta method = bootstrap")
for i,v in enumerate([se_naive,se_delta,se_boot]): ax[0].text(i,v+0.0001,f"{v:.4f}",ha="center",fontsize=8)
ax[1].bar(["delta method","naive"],[100*covd,100*covn],color=[GREEN,RED]); ax[1].axhline(95,color="k",ls="--")
ax[1].set_ylabel("95% CI coverage (%)"); ax[1].set_title("Only the delta method covers at the nominal rate")
for i,v in enumerate([100*covd,100*covn]): ax[1].text(i,v-4,f"{v:.0f}%",ha="center",color="white",fontweight="bold")
plt.tight_layout(); plt.show()
print("The delta method (or an equivalent cluster/bootstrap SE at the user level) is mandatory for ratio metrics -- the naive")
print("session-level SE turned a 95% interval into ~88% coverage, silently inflating the false-positive rate.")

Standard error of one arm's CTR:
  naive (session-level) = 0.0041
  delta method (user)   = 0.0050   (1.24x larger -- the honest SE)
  cluster bootstrap     = 0.0050   (agrees with the delta method)

95% CI coverage of the treatment effect (true lift 0.020), 1500 sims:
  delta method = 0.950  (calibrated)
  naive        = 0.874  (UNDER-covers -> false positives)

The delta method (or an equivalent cluster/bootstrap SE at the user level) is mandatory for ratio metrics -- the naive
session-level SE turned a 95% interval into ~88% coverage, silently inflating the false-positive rate.

3. The Sample Ratio Mismatch guardrail

Before trusting any result, check that users actually landed in the arms in the intended proportion. A Sample Ratio Mismatch (SRM) — say a 50/50 design that comes back 51.5/48.5 — is a red flag that something upstream is broken (biased assignment, a redirect that drops users, logging loss), and it usually biases the effect estimate. The test is a one-line chi-square goodness-of-fit on the arm counts against the intended split; the convention is to alarm at a very small p-value (e.g., $p<0.001$), because with large samples even a tiny genuine imbalance is detectable and worth investigating. A failing SRM invalidates the experiment regardless of how good the headline number looks.

def srm_p(nA, nB, pA=0.5):
    tot=nA+nB; exp=np.array([tot*pA, tot*(1-pA)]); obs=np.array([nA,nB])
    stat=((obs-exp)**2/exp).sum(); return stat, 1-chi2.cdf(stat,1)
cases=[("healthy 50/50",50120,49880),("mild skew",50250,49750),("corrupted",50600,49400)]
print("SRM chi-square test (intended 50/50; alarm if p < 0.001):")
for nm,a,b in cases:
    st,pv=srm_p(a,b); flag="PASS" if pv>=0.001 else "FAIL -- investigate before trusting results"
    print(f"  {nm:14s} {a}/{b}: chi2={st:6.2f}, p={pv:.4g}  {flag}")
fig,ax=plt.subplots(figsize=(8,4))
labels=[c[0] for c in cases]; ps=[srm_p(c[1],c[2])[1] for c in cases]
cols=[GREEN if p>=0.001 else RED for p in ps]
ax.bar(labels,[-np.log10(max(p,1e-12)) for p in ps],color=cols)
ax.axhline(-np.log10(0.001),color="k",ls="--",label="alarm threshold (p=0.001)")
ax.set_ylabel("-log10(p-value)"); ax.set_title("SRM guardrail: a broken split fails the chi-square check"); ax.legend()
plt.tight_layout(); plt.show()
print("SRM is cheap and non-negotiable: run it on every experiment first. A mismatch means the randomization or logging is")
print("broken and the treatment effect is likely biased -- no variance-reduction or sequential method can rescue a broken split.")

SRM chi-square test (intended 50/50; alarm if p < 0.001):
  healthy 50/50  50120/49880: chi2=  0.58, p=0.4479  PASS
  mild skew      50250/49750: chi2=  2.50, p=0.1138  PASS
  corrupted      50600/49400: chi2= 14.40, p=0.0001478  FAIL -- investigate before trusting results

SRM is cheap and non-negotiable: run it on every experiment first. A mismatch means the randomization or logging is
broken and the treatment effect is likely biased -- no variance-reduction or sequential method can rescue a broken split.

4. Summary

Real product metrics are ratios measured over a finer unit than the one you randomized, and that mismatch breaks naive inference:

• The analysis-unit problem: randomizing users but computing CTR over sessions makes session-level standard errors too small (here ~1.25x), so 95% intervals covered only ~88% — a hidden false-positive engine.
• The delta method linearizes the ratio and computes its variance at the user level, restoring nominal coverage; it agreed with a cluster bootstrap, and either is mandatory for ratio metrics (equivalently, cluster-robust SEs at the randomization unit — the design-effect lesson of notebook 1c applied to a ratio).
• Sample Ratio Mismatch is a one-line chi-square guardrail that catches broken assignment/logging before you trust a result; a failing SRM invalidates the experiment.

Practical guidance: always analyze at the randomization unit (delta method, cluster-robust SE, or cluster bootstrap for ratio metrics), and run an SRM check on every experiment first. Cross-links: the clustering intuition is the design effect of Noncompliance & Cluster Designs (1c); pairing this with CUPED (1b) both corrects and shrinks the variance; the delta method reappears whenever a causal estimand is a nonlinear function of means (e.g., RMST / survival-probability contrasts in subsection 10). The R companion reproduces the delta method, cluster bootstrap, and SRM test.