Why the Propensity Score Resists Bayes

Python · PyMC · ArviZ  ·  LaLonde NSW/CPS, and a known-design simulation

A Factorisation, Not an Opinion

Everything in the selection-on-observables group is organised around the propensity score. It is the balancing score that collapses eight covariates to one, the thing matching matches on, the thing weighting weights by, and the object whose estimation occupies most of the applied literature.

A likelihood-based Bayesian analysis throws it away. The reason is a factorisation, not an opinion — writing α\alpha for the parameters governing treatment assignment and β\beta for those governing the outcome:

p(Y,W∣X,α,β)  =  p(W∣X,α)⏟assignment⋅p(Y∣W,X,β)⏟outcomep(Y, W \mid X, \alpha, \beta) \;=\; \underbrace{p(W \mid X, \alpha)}_{\text{assignment}} \cdot \underbrace{p(Y \mid W, X, \beta)}_{\text{outcome}}

If α\alpha and β\beta carry independent priors, the posterior for β\beta — and therefore for the treatment effect — does not involve α\alpha at all. The assignment model is ancillary — it carries no information about the effect, so a posterior that handles it correctly is one that ignores it. A Bayesian who writes down the full likelihood and turns the handle gets the same answer whether or not the propensity model is in the room.

Two models on the same observational sample: an outcome model alone, and that same model plus a complete logistic model of treatment assignment, fitted jointly in the same PyMC model. If the factorisation is right the effect posterior should be unmoved — not similar, unmoved to within Monte Carlo error, the residual jitter left by the sampler itself, which sets the floor below which two posteriors cannot be told apart.

modeleffectsd95% interval
outcome model only1,542775[25, 3,070]
+ full propensity model, fitted jointly1,559783[42, 3,083]

Difference in posterior mean: USD 18. Monte Carlo error of each mean alone: USD 8 and USD 7. Randomized benchmark USD 1,794.

It is. The difference is USD 18, against a Monte Carlo error of USD 8 in each mean and a posterior standard deviation of about USD 780. The propensity model is not approximately irrelevant here; it is exactly irrelevant, and the gap between the two answers is smaller than the noise in either one.

Why That Should Worry You

It is tempting to read that as a convenience — one fewer model to fit. It is better read as a warning, because the propensity score was never decoration. It buys robustness to getting the outcome model wrong. Inverse-probability weighting is consistent whenever the propensity model is right, however badly the outcome is modelled; a pure outcome-model analysis has no such protection. A coherent Bayesian, having established that the assignment model cannot enter, has discarded exactly the information that would have provided the insurance.

Robins & Ritov (1997) turned this into a formal problem: in high dimensions there are settings where any procedure ignoring the known assignment mechanism performs badly while one using it does fine. The notebook builds a small concrete version — the assignment mechanism is known by design, so there is no excuse for an estimator that cannot use it, and the outcome model is misspecified in a way correlated with treatment.

estimatormeanbiassdRMSE
outcome model, linear misspecified2.23+0.230.320.39
weighting by the true propensity score1.97−0.031.021.02

300 replicates, n = 600, 10 covariates, true effect 2.0. The assignment mechanism is known by construction.

The bias appears exactly as the theory says. But read the whole row before declaring a winner, because the honest result is a trade rather than a victory. Weighting pays for its unbiasedness with 3.2 times the spread, and on root-mean-square error the misspecified regression wins — 0.39 against 1.02. An analyst minimising expected squared error would take the biased estimator here, and would be right to.

So the design information is worth something, and what it is worth is unbiasedness rather than accuracy. That is the precise form of the loss. The information is real; the likelihood principle says it is not in the likelihood.

The Obvious Repair, and Why It Fails

The resolution is not to bolt the propensity score onto a Bayesian analysis. Two-step recipes that draw α\alpha from its posterior and then match or weight are incoherent as a single model — pragmatic devices rather than posteriors, usually justified by frequentist properties. The resolution is to notice what actually failed: not the absence of a propensity model, but a misspecified outcome model. The regression could not represent a function that bends, and the bending was correlated with treatment.

That suggests an obvious repair — make the outcome model flexible enough that misspecification stops being the binding constraint. Applied naively, the obvious repair fails, and it is worth seeing fail before seeing it done properly.

estimatormeanbiassd
outcome model, linear misspecified2.18+0.180.31
outcome model, flexible no propensity score0.48−1.520.13

Same construction, true effect 2.0. The flexible model is tighter and far more wrong.

A flexible outcome model, with no propensity score anywhere, does not recover the estimate. It attenuates the effect from 2.0 to 0.48 — a bias of −1.52, roughly eight times the linear model's error and in the opposite direction.

The mechanism is worth being precise about. The prognostic signal here swings across roughly ten units while the treatment effect is 2. A regularized learner spends its splits where the variance is, which is the covariates, and the treatment indicator is left to soak up what little remains. The shrinkage that makes a model good at predicting is what destroys it as an estimator of a causal contrast.

That has a name — regularization-induced confounding — and it is not an argument against flexible models but the argument for a particular kind of them. A model that separates the prognostic part from the treatment part, and admits the propensity score back in as a covariate to absorb the selection, repairs it. That is Bayesian Causal Forests, and it is the third example in this group.

The shape of the argument is worth stating once, plainly. A coherent Bayesian cannot use the assignment model, so robustness has to come from the outcome model instead. Making the outcome model flexible introduces a new bias. Fixing that bias requires letting the propensity score back in through the only door still open to it — as a predictor rather than as a weight. The propensity score returns, but not in the role the frequentist group gave it.

Where this sits

This is the opening move of the Bayesian selection-on-observables group, and it is what motivates the rest of it. The frequentist treatment of the same data is in Potential Outcomes & Matching; the estimate whose fragility is priced in Confounding as a Parameter is the same regression-adjusted number that appears here.

Notebook

Python The ancillarity of the propensity score demonstrated in PyMC on LaLonde — an outcome model fitted alone and then jointly with a complete logistic assignment model — followed by a constructed setting with a known assignment mechanism where the design information is worth real unbiasedness, and a flexible outcome model that fails in the opposite direction (Python, PyMC) Open →

Downloads

lalonde_obs.csv 185 NSW treated against 429 CPS controls — the sample both models are fitted to lalonde_exp.csv The randomized NSW sample supplying the USD 1,794 benchmark

Everything in the last two sections is simulated from seeded generators, since grading an estimator against a known assignment mechanism requires constructing one.

References