A/B Testing — Interference and Switchback Designs

When SUTVA Fails

Every method up to this point has rested on SUTVA — the stable unit treatment value assumption, which requires that one unit's treatment does not affect another unit's outcome, and that there is only one version of the treatment. In marketplaces, social networks and anything with a shared resource, that assumption simply fails. A discount that attracts one rider takes a driver away from another. A ranking boost for one seller demotes a competitor. A post shown to one user is reshared to their friends. When treatment leaks between units, the standard A/B test does not become noisy — it becomes precisely wrong, which is considerably more dangerous.

The workhorse is the linear-in-means model: a user's outcome depends on their own treatment and on the fraction treated in their market. Here $\beta$ is the direct effect of your own treatment and $\gamma$ the spillover, negative when treatment cannibalizes something shared — one user's booking is another's lost booking. The question a business actually faces is the total or policy effect: what happens if we ship this to everyone rather than nobody. That moves the treated fraction from 0 to 1, so the answer is $\beta + \gamma$.

Now the failure. A conventional A/B randomizes users within each market at roughly 50/50 — which means the treated fraction is about 0.5 in both arms. The spillover term is therefore identical for treated and control users and cancels in the difference. The experiment cannot see it, not because it is small but because the design has subtracted it away.

With a true direct effect of 1.00, a spillover of −0.60 and therefore a true total effect of 0.40, the naive user-level A/B returns 0.988. It has recovered the direct effect almost exactly — and overstated the thing the business cares about by 2.5×. The detail that makes this genuinely hazardous is the standard deviation: 0.019. The estimate is not merely biased, it is tight around the wrong number. Every diagnostic an experimenter would normally consult — narrow interval, strong significance, stable across reruns — looks excellent. Nothing internal to the experiment reveals the problem.

Containing the Spillover

The fix for interference across space is to randomize at a level coarse enough to contain the spillover: whole markets rather than users. A treated market then has a treated fraction of 1 and a control market 0, so the comparison moves the full distance and picks up both terms.

Cluster randomization returns 0.402 against a truth of 0.400 — a bias of +0.002. The stated cost is variance, since the effective sample size becomes the number of markets rather than users, and it is real: the standard deviation rises from 0.019 to 0.021. But it is worth putting those two numbers side by side, because the trade is extremely lopsided. Clustering removes a bias of +0.588 at a cost of 0.002 in standard deviation — the bias eliminated is roughly thirty times the noise added. The bias-variance framing is correct in principle and can make this sound like a genuine dilemma; here it is not close.

Switchbacks, and the Window That Decides Everything

Interference is often temporal rather than spatial. A pricing or dispatch change alters system state — available supply, queue length — that persists into the next period, so today's treatment contaminates tomorrow's outcome and no split of users can separate them. The tool is a switchback: turn the feature on and off for the entire system across randomized time windows.

The design parameter that decides everything is the window length relative to the carryover horizon, and the sweep shows it sharply. With a one-period carryover, alternating every period returns 0.998 — the direct effect again, the naive A/B's blind spot reproduced exactly in the time domain. Each on-period is preceded by a randomly-chosen state, so the carryover averages out instead of accumulating. Windows of two or more, with the first period discarded as burn-in, recover the total effect: 0.397, 0.399, 0.399, 0.400.

The lesson generalises past the specific numbers. A switchback is not automatically the right answer to temporal interference — it is the right answer only if the window exceeds the carryover horizon and burn-in is discarded. Run with too short a window and it inherits precisely the failure it was adopted to fix, while looking like a sophisticated design. That is the same trap as the user-level A/B: a method that appears to address interference while quietly measuring the direct effect.

Where this sits

This is the identification-level counterpart to cluster designs, where clustering was about honest standard errors; here it is about which estimand you recover at all. The discipline is the same one that separates intention-to-treat from the complier average causal effect in that notebook: decide which effect the decision needs before choosing the design. If the direct effect is genuinely what you want, a user-level A/B is fine and far cheaper. If you need the policy effect, no amount of analysis rescues a design that cancelled the spillover — you must contain it, with clusters for spatial interference and switchbacks for temporal. The direct-versus-total split also anticipates the decomposition into direct and indirect effects in mediation later in the arc.

Notebooks

References

• Hudgens, M. G. & Halloran, M. E. (2008). Toward Causal Inference With Interference. Journal of the American Statistical Association 103(482), 832–842. — causal inference when units interfere
• Aronow, P. M. & Samii, C. (2017). Estimating average causal effects under general interference, with application to a social network experiment. The Annals of Applied Statistics 11(4). — exposure mappings on a network
• Rubin, D. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology 66(5), 688–701. — the SUTVA this relaxes