A/B Testing — Multiple Comparisons and the False Discovery Rate

Python · NumPy · SciPy · R · p.adjust

Looking Many Times in Parallel

Peeking inflates the error rate by looking many times in sequence. The identical disaster arrives when you look many times in parallel — scoring one treatment against a dashboard of dozens of metrics, or comparing many variants at once. Each test at $\alpha = 0.05$ carries its own 5% false-positive chance, so the two problems are the same problem wearing different clothes: multiplicity, across time in one case and across metrics in the other.

The simulation scores an experiment on 200 metrics, of which 40 genuinely moved and 160 are null. Null p-values are uniform on [0,1] by construction, so roughly 5% of those 160 land under 0.05 in every single run — about 8 spurious winners per experiment, arriving reliably rather than occasionally.

The quantity that actually matters for a dashboard is the family-wise error rate: the probability of at least one false positive anywhere in the set. It climbs fast, and with 160 null metrics it is not approaching certainty — it has arrived at it.

number of null metrics tested at 0.05	P(at least one false positive)
1	0.050
5	0.226
10	0.401
50	0.923
160 (this dashboard)	1.000

At 1.000, every experiment produces at least one spurious winner. Not most; every one. Any team scanning a large metric dashboard without correction and celebrating whatever turns green is, with probability one, celebrating noise at least once per experiment.

Two Corrections, Two Different Goals

Two families of correction exist and they target genuinely different things, which is the substance of the notebook rather than a detail of implementation.

Bounding any false positive — Bonferroni and Holm

Bonferroni bounds the FWER by testing each metric at $\alpha/K$ . It works, and the price is severe: with $K = 200$ the per-test threshold is 0.00025, so only overwhelming effects survive. Holm's step-down achieves the same guarantee with uniformly more power by comparing the $i$ -th smallest p-value to $\alpha/(K-i+1)$ and stopping at the first failure. Holm dominates Bonferroni — it is never worse and sometimes better — so there is no reason to prefer plain Bonferroni when FWER control is the goal. The measured gap here is small (power 0.260 against 0.255) but it is free.

Bounding the proportion — Benjamini–Hochberg

\text{reject } (1),\dots,(i^\ast) \quad \text{where} \quad i^\ast = \max\Big\{\, i : p_{(i)} \le \tfrac{i}{K}\,q \,\Big\}

The deeper question is whether bounding the probability of any false positive is the right objective at all. For a screen across hundreds of metrics it usually is not. You do not need zero false positives; you need the proportion of your reported winners that are spurious to be small. That quantity is the false discovery rate, and Benjamini–Hochberg controls it: sort the p-values, find the largest $i$ with $p_{(i)} \le \frac{i}{K}q$ , and reject everything up to it.

The payoff is large. BH holds the FDR at 0.040 against a 0.05 target while recovering 61% of the true movers — roughly 2.4× the power of the FWER methods, which find only about one true mover in four. What BH is buying with that power is worth stating precisely, because it is a genuine trade rather than a free lunch: it accepts a small, controlled fraction of false discoveries in exchange for finding far more real effects. Bonferroni's 0.6% false-discovery rate is not a superior result, it is the symptom of a method that rejects almost nothing.

200 metrics · 40 true movers · 160 nulls	FWER	FDR	power
naive, no correction	1.000	0.189	0.857
Bonferroni	0.062	0.006	0.255
Holm (dominates Bonferroni)	0.062	0.006	0.260
Benjamini–Hochberg	—	0.040	0.613

Which is right depends entirely on what the answer feeds. A regulatory submission or a single pre-registered primary endpoint wants FWER control, because one false claim is the thing being guarded against. A metric dashboard used to decide what to investigate next wants FDR control, because the cost of a false lead is bounded and the cost of missing three real effects out of four is not. The practical resolution used by experimentation platforms is to do both: pre-register a small set of primary metrics and hold them to a strict standard, apply BH to the broader guardrail dashboard, and treat any uncorrected scan as hypothesis-generating rather than confirmatory.

Where this sits

This is the parallel-testing twin of sequential monitoring — the same multiplicity arithmetic, across metrics instead of across looks, and it is worth noticing that a team can be scrupulous about one while entirely ignoring the other. The pre-registration discipline it points to is the same one that keeps a randomized experiment honest, since randomization protects identification but nothing protects an analyst who tries enough outcomes. The threshold-selection logic also connects to the calibration and selection machinery in the ML arc, where choosing among many models raises a structurally identical problem.

Notebooks

Python A 200-metric dashboard with 40 true movers and 160 nulls, showing the family-wise error rate reaching 1.000 under naive testing, then Bonferroni, Holm and Benjamini-Hochberg compared on FWER, FDR and power from the same simulated p-values (Python) Open → R The same comparison in base R via p.adjust with the bonferroni, holm and BH methods, reproducing the multiplicity story and the power gap between family-wise and false-discovery control (R) Open →

References

Benjamini, Y. & Hochberg, Y. (1995). Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. Journal of the Royal Statistical Society Series B: Statistical Methodology 57(1), 289–300. — the false discovery rate
Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics 6, 65–70. — the step-down alternative to Bonferroni