Causal Inference I(f) — A/B Testing: Interference and Switchback Designs

When treating one unit affects another — SUTVA violations, cluster randomization, and switchbacks

Every method so far has leaned on SUTVA — the assumption that one unit's treatment does not affect another's outcome. In marketplaces, social networks, and any system with shared resources, that assumption fails: a discount that lures one rider takes a driver away from another; a feature that boosts one seller's ranking demotes competitors; a post shown to one user is reshared to their friends. This interference quietly biases the standard user-level A/B test — often making a feature look far better (or worse) than it is for the business as a whole.

This notebook builds the interference problem and its two standard fixes:

1. The SUTVA violation — a linear-in-means marketplace where the user-level A/B measures only the direct effect and misses the spillover, so it is biased for the total (policy) effect of shipping the feature to everyone.
2. Cluster randomization — randomize whole markets, so treated and control units don't interfere across arms; this recovers the total effect (at the cost of variance).
3. Switchback designs — for temporal interference (carryover), randomize time windows of the whole system on/off; recovers the total effect if the window exceeds the carryover horizon.

Simulation-first with a known total effect, because the whole point is to see the naive estimate miss it. Python leads; the R companion mirrors it. This is the SUTVA-failure companion to Noncompliance & Cluster Designs (1c) — there clustering was about standard errors; here it is about identification under interference.

1. The SUTVA violation — interference biases the user-level A/B

We use the standard linear-in-means interference model (Manski; Hudgens & Halloran): a user's outcome depends on their own treatment and on the fraction treated in their market $f_m$, $$Y_i=\alpha+\beta\,T_i+\gamma\,f_m+\varepsilon_i.$$ Here $\beta$ is the direct effect (your own treatment) and $\gamma$ is the spillover — negative when treatment cannibalizes a shared resource (a treated user's booking is a control user's lost booking). The business question is the total / policy effect of turning the feature on for everyone versus no one: $f_m$ goes 0→1, so the total effect is $\beta+\gamma$.

The usual A/B test randomizes users within each market at ~50/50. But then $f_m\approx0.5$ in both arms, so the $\gamma f_m$ term is identical for treated and control and cancels — the estimate captures $\beta$ (direct) alone and is blind to the spillover. With cannibalization ($\gamma<0$) the A/B badly overstates the value of shipping.

import numpy as np, matplotlib.pyplot as plt, warnings
warnings.filterwarnings("ignore")
BLUE="#2b6cb0"; RED="#c53030"; GREEN="#2f855a"; ORANGE="#dd6b20"; GREY="#a0aec0"; PURP="#6b46c1"
alpha,beta,gamma = 2.0, 1.0, -0.6      # direct beta=1.0, spillover gamma=-0.6 -> TRUE total effect = 0.4
M, per = 200, 50
def user_AB(seed=0):
    r=np.random.default_rng(seed); Y=[]; T=[]
    for m in range(M):
        t=r.binomial(1,0.5,per); f=t.mean(); Y.append(alpha+beta*t+gamma*f+r.normal(0,1,per)); T.append(t)
    Y=np.concatenate(Y); T=np.concatenate(T); return Y[T==1].mean()-Y[T==0].mean()
naive=[user_AB(s) for s in range(400)]
print(f"TRUE effects:  direct beta = {beta:.2f}   spillover gamma = {gamma:.2f}   TOTAL (policy) = beta+gamma = {beta+gamma:.2f}")
print(f"  naive user-level A/B = {np.mean(naive):.3f} (SD {np.std(naive):.3f})")
print(f"  -> it recovers the DIRECT effect {beta:.2f}, not the total {beta+gamma:.2f}: {np.mean(naive)/(beta+gamma):.1f}x too large for the business question")
fig,ax=plt.subplots(1,2,figsize=(13,4.2))
ax[0].axhline(beta+gamma,color=GREEN,lw=2,ls="--",label=f"TRUE total effect {beta+gamma:.1f}")
ax[0].axhline(beta,color=RED,lw=2,ls=":",label=f"direct effect {beta:.1f}")
ax[0].hist(naive,bins=30,color=GREY,alpha=.8,orientation="vertical")
ax[0].set_xlabel("user-level A/B estimate"); ax[0].set_ylabel("sims"); ax[0].set_title("Naive A/B centers on the DIRECT effect, missing spillover"); ax[0].legend(fontsize=8)
fr=np.linspace(0,1,50); ax[1].plot(fr,alpha+beta+gamma*fr,color=GREEN,lw=2,label="treated unit")
ax[1].plot(fr,alpha+gamma*fr,color=BLUE,lw=2,label="control unit")
ax[1].set_xlabel("fraction treated in market"); ax[1].set_ylabel("expected outcome"); ax[1].set_title("Spillover: everyone's outcome falls as more are treated"); ax[1].legend(fontsize=8)
plt.tight_layout(); plt.show()
print("At a 50/50 within-market split, the spillover term shifts BOTH arms equally and cancels -- so the A/B sees only the")
print("direct effect. Ship-to-everyone moves the treated fraction 0->1, and the missed spillover is exactly that gap.")

TRUE effects:  direct beta = 1.00   spillover gamma = -0.60   TOTAL (policy) = beta+gamma = 0.40
  naive user-level A/B = 0.988 (SD 0.019)
  -> it recovers the DIRECT effect 1.00, not the total 0.40: 2.5x too large for the business question

At a 50/50 within-market split, the spillover term shifts BOTH arms equally and cancels -- so the A/B sees only the
direct effect. Ship-to-everyone moves the treated fraction 0->1, and the missed spillover is exactly that gap.

2. Cluster randomization recovers the total effect

The fix for cross-sectional interference is to randomize at a level coarse enough to contain the spillover — whole markets (cities, regions) instead of users. Now a treated market has $f_m=1$ and a control market has $f_m=0$, so comparing market-level outcomes moves the treated fraction the full 0→1 and captures both the direct effect and the spillover: $$(\alpha+\beta+\gamma)-(\alpha)=\beta+\gamma,$$ the total effect. The price is variance: the effective sample size is the number of markets, not users, so cluster-randomized experiments need many clusters and are less precise. This is the classic interference bias-variance trade-off — the user-level test is precise but biased, the cluster test is unbiased but noisier.

def cluster_rand(seed=0):
    r=np.random.default_rng(seed); tm=r.binomial(1,0.5,M); rows=[]
    for m in range(M):
        f=tm[m]; y=alpha+beta*tm[m]+gamma*f+r.normal(0,1,per); rows.append((tm[m],y.mean()))
    a=np.array(rows); return a[a[:,0]==1,1].mean()-a[a[:,0]==0,1].mean()
clust=[cluster_rand(s) for s in range(400)]
print(f"TRUE total effect = {beta+gamma:.2f}")
print(f"  naive user-level A/B   = {np.mean(naive):.3f}  (bias {np.mean(naive)-(beta+gamma):+.3f}, SD {np.std(naive):.3f})")
print(f"  cluster (market) rand. = {np.mean(clust):.3f}  (bias {np.mean(clust)-(beta+gamma):+.3f}, SD {np.std(clust):.3f})")
print(f"  -> clustering removes the interference bias; variance rises ({np.std(clust)/np.std(naive):.1f}x SD) because n = markets, not users")
fig,ax=plt.subplots(figsize=(8.5,4.4))
ax.hist(naive,bins=30,color=RED,alpha=.55,label=f"user-level A/B (biased, mean {np.mean(naive):.2f})")
ax.hist(clust,bins=30,color=GREEN,alpha=.6,label=f"cluster rand. (unbiased, mean {np.mean(clust):.2f})")
ax.axvline(beta+gamma,color="k",lw=2,ls="--",label=f"TRUE total effect {beta+gamma:.1f}")
ax.set_xlabel("estimated total effect"); ax.set_ylabel("sims"); ax.set_title("Cluster randomization trades bias for variance"); ax.legend(fontsize=8)
plt.tight_layout(); plt.show()
print("If the policy question is 'what happens when we ship to everyone?', the cluster estimate answers it and the user-level")
print("estimate does not. Use enough clusters (and cluster-robust SEs, notebook 1c) to keep the variance manageable.")

TRUE total effect = 0.40
  naive user-level A/B   = 0.988  (bias +0.588, SD 0.019)
  cluster (market) rand. = 0.402  (bias +0.002, SD 0.021)
  -> clustering removes the interference bias; variance rises (1.1x SD) because n = markets, not users

If the policy question is 'what happens when we ship to everyone?', the cluster estimate answers it and the user-level
estimate does not. Use enough clusters (and cluster-robust SEs, notebook 1c) to keep the variance manageable.

3. Switchback designs for temporal interference

Interference is often over time, not space: a pricing or dispatch change alters the system state (available supply, queue length) that persists into the next period, so today's treatment spills into tomorrow's outcome — a single market can't be cleanly split into treated and control users. The standard tool is a switchback: turn the feature on and off for the whole system across time windows, randomizing the windows. Comparing on-windows to off-windows moves the system fully between states and, in steady state, captures the total effect including carryover.

The crucial design parameter is the window length relative to the carryover horizon. With windows too short (e.g., alternate every period), each on-period is preceded by a random state and the carryover averages out — the switchback then measures only the direct effect, exactly like the naive A/B. With windows long enough to reach steady state — and discarding the first period(s) of each window as burn-in — the estimate recovers the total effect. We simulate one-period carryover and sweep the window length.

T=6000
def switchback(window, burn, seed=0):
    r=np.random.default_rng(seed); n_win=T//window; w=r.binomial(1,0.5,n_win)
    on=np.repeat(w,window)[:T]; carry=np.r_[0,on[:-1]]
    y=alpha+beta*on+gamma*carry+r.normal(0,1,T)
    pos=np.tile(np.arange(window),n_win)[:T]; keep=pos>=burn
    yk,ok=y[keep],on[keep]; return yk[ok==1].mean()-yk[ok==0].mean()
configs=[(1,0),(2,1),(5,1),(10,1),(20,2)]
res=[(w,b,np.mean([switchback(w,b,s) for s in range(200)])) for w,b in configs]
print(f"TRUE total (steady-state) effect = {beta+gamma:.2f}; direct = {beta:.2f}; carryover = 1 period")
for w,b_,e in res:
    tag="recovers TOTAL" if abs(e-(beta+gamma))<abs(e-beta) else "biased toward DIRECT (carryover contaminates)"
    print(f"  window={w:2d}, burn-in={b_}: estimate {e:.3f}  ({tag})")
fig,ax=plt.subplots(figsize=(8.5,4.4))
ws=[w for w,_,_ in res]; es=[e for _,_,e in res]
ax.plot(ws,es,"o-",color=PURP,lw=2)
ax.axhline(beta+gamma,color=GREEN,ls="--",label=f"TRUE total {beta+gamma:.1f}"); ax.axhline(beta,color=RED,ls=":",label=f"direct {beta:.1f}")
ax.set_xlabel("switchback window length"); ax.set_ylabel("estimated effect"); ax.set_title("Windows must exceed the carryover horizon (with burn-in)"); ax.legend(fontsize=8)
plt.tight_layout(); plt.show()
print("Window=1 measures only the direct effect (same blind spot as the naive A/B); windows >= the carryover horizon, with")
print("burn-in periods discarded, recover the total effect. Switchbacks are the temporal analogue of cluster randomization.")

TRUE total (steady-state) effect = 0.40; direct = 1.00; carryover = 1 period
  window= 1, burn-in=0: estimate 0.998  (biased toward DIRECT (carryover contaminates))
  window= 2, burn-in=1: estimate 0.397  (recovers TOTAL)
  window= 5, burn-in=1: estimate 0.399  (recovers TOTAL)
  window=10, burn-in=1: estimate 0.399  (recovers TOTAL)
  window=20, burn-in=2: estimate 0.400  (recovers TOTAL)

Window=1 measures only the direct effect (same blind spot as the naive A/B); windows >= the carryover horizon, with
burn-in periods discarded, recover the total effect. Switchbacks are the temporal analogue of cluster randomization.

4. Summary

Interference breaks SUTVA, and with it the standard A/B test — because a user-level experiment holds the treated fraction roughly equal across arms, it estimates only the direct effect and is blind to spillover:

• In the linear-in-means marketplace, the naive A/B centered on the direct effect (1.0) while the true total (policy) effect of shipping to everyone was 0.4 — a 2.5x overstatement driven by cannibalization ($\gamma<0$).
• Cluster randomization (whole markets) moves the treated fraction the full 0→1, recovering the total effect unbiasedly at the cost of higher variance (effective $n$ = clusters).
• Switchback designs handle temporal interference by randomizing whole-system on/off time windows; they recover the total effect only when the window exceeds the carryover horizon and burn-in periods are discarded — otherwise they inherit the naive test's direct-effect-only blind spot.

Practical guidance: decide which estimand you need — the direct effect (user-level A/B is fine) or the total policy effect (you must contain the interference). For marketplace/network features, randomize at a level coarse enough to absorb the spillover (cluster for spatial, switchback for temporal), size for the reduced effective sample, and use cluster-robust inference. Cross-links: this is the identification-level counterpart to the standard-error clustering in Noncompliance & Cluster Designs (1c); the "total vs direct" split parallels direct/indirect effects in mediation (subsection 8); and the estimand discipline echoes ITT vs CACE (1c) — different valid questions, different designs. The R companion reproduces the interference bias, the cluster fix, and the switchback window sweep.