Bayesian A/B Testing and Multi-Armed Bandits

Experimentation as a Decision

The frequentist notebooks in this group all ask one question: can we reject the null. The Bayesian view reframes an experiment as a decision under uncertainty — given what we have seen, how probable is it that B beats A, and how much do we expect to lose if we ship the wrong one. For a conversion rate the machinery is the Beta–Binomial conjugate model, which is the same object as the shrinkage estimators in the Bayesian arc, doing a different job.

Two quantities come out of the posteriors and both are directly actionable in a way a p-value is not. $\Pr(p_B > p_A)$ is the probability B is better. The expected loss of shipping B, $\mathbb{E}[\max(p_A - p_B, 0)]$, is how much conversion rate you expect to forgo if B turns out to be worse — the probability of being wrong weighted by how badly wrong. The decision rule is to ship when that expected loss falls below a small caliper — a threshold set in the units of the metric itself, so “I will accept at most half a hundredth of a percentage point of expected lost conversion” rather than “I want p < 0.05”. That bounds the downside of being wrong instead of controlling a hypothetical long-run error rate.

The worked case is deliberately borderline, and it is worth reading carefully because the Bayesian and frequentist habits disagree here by design.

Notice that the 95% credible interval on the uplift is [−0.0070, +0.0378] — it contains zero. A reader trained to check whether an interval excludes zero would stop there and call the result inconclusive. The decision rule ships B anyway, and it is not being reckless. There is an 8.9% chance B is worse, but conditional on being worse it is likely only slightly worse, so the expected cost of shipping it is 0.00047 — under five hundredths of a percentage point of conversion. Shipping A instead carries an expected loss of 0.01583, thirty-four times larger. The question the decision rule asks is not "am I sure?" but "what does being wrong cost?", and on those numbers the answer is clear even though significance is not.

Bayesian Posteriors Do Not License Free Peeking

Now the part that gets oversold everywhere, and the notebook is refreshingly blunt about it. A widely repeated claim is that Bayesian A/B testing solves the peeking problem — that because a posterior is not a long-run error rate, you may monitor continuously and stop whenever $\Pr(p_B > p_A)$ crosses 0.95. That is false in the frequentist sense, and the simulation shows how badly.

On an A/A test where the arms are identical, that stopping rule declares a winner 58% of the time. A single fixed-sample look at the same threshold fires about 6%, as designed. The posterior wanders exactly as the running z-statistic does, and stopping at the first favourable crossing exploits that wandering just as effectively. It is worth putting this next to the frequentist peeking result: naive peeking there reached 24% at twenty looks, and this reaches 58%. Adopting Bayesian language and then peeking freely is not a fix for the problem — on these numbers it is substantially worse than the frequentist error it was supposed to avoid.

What Bayesian decision theory genuinely provides is a different contract, not a stronger version of the same one. Stopping when the expected loss of shipping the leader is below a caliper bounds how much you expect to lose, whenever you stop. That is a statement about decision quality, and it is often the more useful guarantee — but it is not a false-positive rate, and it should not be described as one. If a guaranteed error rate is what you need, the always-valid methods in the sequential notebook are the tool.

Bandits — Earning While Learning

The last section changes the objective. A/B testing fixes the split for the duration, so half the traffic keeps seeing the worse arm until the test ends; those lost conversions are regret. A multi-armed bandit adapts, shifting traffic toward arms that look better as evidence accumulates. Thompson sampling is the natural Bayesian rule: draw one sample from each arm's posterior and play whichever draw is highest. Arms that are probably best receive most of the traffic, while genuinely uncertain arms keep getting explored in proportion to their posterior chance of winning — exploration and exploitation balanced without a tuning parameter.

Across four arms with true rates from 0.10 to 0.13 over 20,000 pulls, Thompson cuts regret to 94.6 against uniform A/B's 300.0 — a factor of about 3.2. Its final allocation tells the story better than the total: 68% of traffic to the best arm and 3% to the worst, reached without ever being told which was which.

The trade-off is real and worth stating plainly, because bandits are often presented as strictly superior. They optimise earnings during the test, not the cleanliness of the causal readout at the end. Adaptive allocation means the arms are no longer compared on equal footing or at equal sample size, so the end-of-test estimate is harder to interpret and the machinery of the rest of this group — fixed-horizon power, unbiased differences in means — no longer applies straightforwardly. Bandits suit ongoing optimisation across many variants; a one-off causal question is better served by a fixed design.

Where this sits

The Beta–Binomial conjugacy here is the same machinery as the baseball shrinkage and binomial GLMM notebooks in the Bayesian arc, applied to a decision rather than an estimate. The peeking caveat is the direct counterpart of sequential monitoring, and reading the two together is the point: the inflation is a property of stopping on a wandering statistic, not of the statistical philosophy attached to it. And the expected-loss framing — probability of being wrong weighted by the cost of being wrong — is the same decision-theoretic move that separates a calibrated probability from a ranking in Calibration.

Notebooks

References

• Rubin, D. B. (1978). Bayesian Inference for Causal Effects: The Role of Randomization. The Annals of Statistics 6(1). — the Bayesian view of randomisation
• Rubin, D. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology 66(5), 688–701. — potential outcomes
• Hill, J. L. (2011). Bayesian Nonparametric Modeling for Causal Inference. Journal of Computational and Graphical Statistics 20(1), 217–240. — a nonparametric Bayesian response surface