← All examples

Causal Inference: Bayesian Selection on Observables

The Bayesian arc over randomized experiments was ordered by how much the prior can move the answer, and its punchline was that this reduces to the t-statistic: priors competing against data, and losing as information accumulates.

This group has a different subject, because Selection on Observables leaves a different gap. There, identification rests on an assumption nothing can test, and every diagnostic in the group turned out to measure something other than what you want to know — aggregate balance picked the wrong estimator, exact balance cost 77% of the effective sample without saying so, and a p-value stayed silent on fragility. What is missing is not precision. It is a way to reason about what was never recorded.

So the priors here do a job they could not do in the previous group: they stand in for quantities the data cannot identify. That is the honest case for a Bayesian treatment of observational causal inference, and it is also the reason a careless version would be worthless — re-fitting inverse-probability weighting in PyMC and recovering the same number with wider intervals adds nothing. Each example has to produce something the frequentist version structurally cannot.

What a Bayesian treatment adds here, and what it costs

Every value is committed notebook output, on the same LaLonde sample the frequentist group used, so the two are directly comparable and the randomized benchmark grades both.

A · The test each example has to pass: return something the frequentist version structurally cannot Re-fitting an existing estimator in PyMC and recovering the same number with wider intervals would add nothing. Each of the three returns a different object. Why the propensity score resists Bayes a proof, not a preference — and a bill attached to it The likelihood factorises, so with independent priors the effect posterior cannot involve the assignment model. Fitting one jointly moves the answer USD 18 against a Monte Carlo error of USD 8. What it costs is unbiasedness: with a known assignment mechanism and a misspecified outcome model, weighting is unbiased at −0.03 and the regression carries +0.23. Response surfaces and the bias they create a per-unit posterior that reports its own inadequacy Every unit gets its own effect with uncertainty attached, which no frequentist estimator in the group produces at all. And the uncertainty is the finding: the per-unit effects are systematic (R² = 0.67) and swamped by their own noise (signal-to-noise 0.12). The object exists and does not identify anything. Confounding as a parameter a posterior over the thing that was never measured Rosenbaum returns a breakdown threshold, Γ* = 1.21, and says nothing about how likely you are to be past it. A prior on the unobserved confounder, calibrated against the confounders actually measured, returns Pr(effect > 0) = 0.827 instead — a probability rather than a boundary. B · And the surprise: the flexible Bayesian learner is the worst adjusted estimate in the collection Estimated effect of job training on 1978 earnings, LaLonde NSW/CPS. Dashed line = the randomized experiment’s USD 1,794. naive difference in means no adjustment at all −$635 −135% BART, single response surface one flexible outcome model, counterfactual imputed $283 −84% Mahalanobis matching the frequentist group’s tidiest balance table $719 −60% TMLE doubly robust with a targeting step $1,199 −33% AIPW doubly robust $1,225 −32% entropy balancing exact mean balance by construction $1,273 −29% IPW inverse-propensity weighting $1,316 −27% Bayesian causal forests separate treatment surface; propensity as a predictor $1,328 −26% NN-propensity matching nearest neighbour on the propensity score $1,792 essentially exact randomized benchmark $1,794 C · One diagnosis, reached three independent ways: regularization-induced confounding The shrinkage that makes a model good at predicting destroys it as an estimator of a causal contrast. It is not a tuning problem, and more capacity does not fix it. It appears unbidden in a controlled simulation True effect 2.0, known design, the flexible model brought in specifically to repair the linear one’s bias. linear outcome model +0.23 • flexible learner attenuates to 0.48 — eight times the bias, in the opposite direction It survives every attempt to tune it away On LaLonde the single BART surface returns USD 283 against a USD 1,794 benchmark, the worst adjusted estimate above. changing only the random seed moves it more than quadrupling the trees • across every configuration it lands between USD 154 and USD 716 It is the parameterisation, not the capacity The decisive test holds capacity fixed and changes only how the model is written. at a matched 75 trees: Bayesian causal forests 1,232 against a single surface’s 489 — a gap of 743 with identical capacity

A is the group’s entry condition. A Bayesian rerun of an existing estimator that returns the same number with wider intervals is not worth a page, so each example is held to producing an object the frequentist analysis does not have. The first returns a proof: the likelihood factorises, so a coherent posterior for the effect cannot involve the assignment model — which is an uncomfortable result, because the entire frequentist group is organised around that model. The second returns a per-unit posterior, something no estimator in the previous group produces. The third returns a probability where sensitivity analysis returns a threshold.

The first of those comes with a bill, and it is stated rather than buried. Where the assignment mechanism is genuinely known and the outcome model is wrong, weighting by the true propensity score is unbiased at −0.03 while the regression carries +0.23. The Bayesian is not free to use the correction that would have fixed that, because the factorisation forbids it. Unbiasedness is the price, and the defence is not that it does not matter — it is that weighting pays 3.2× the spread for it, so the biased estimator still wins on RMSE.

B is the result that reorganised the group. If the propensity score is unavailable then robustness has to come from the outcome model, and the obvious way to make an outcome model robust is to make it flexible. Doing exactly that produces USD 283 — the worst adjusted estimate in the entire collection, frequentist entries included, and the only one that fails to beat Mahalanobis matching. A method chosen for its flexibility finishes below every method chosen for anything else.

C is the reason, and it is not overfitting. The prognostic signal in this data moves 1978 earnings by thousands while the treatment effect is under two thousand dollars, so an ensemble that allocates its splits by predictive gain spends them almost entirely on the part of the surface that is not the causal contrast. The shrinkage that makes the model good at predicting is what destroys it as an estimator. The three rows are three independent routes to that conclusion, and the third is the decisive one: holding capacity fixed at 75 trees and changing only how the model is parameterised moves the estimate from 489 to 1,232. Capacity was never the variable.

Which is what makes the repair more than a patch. Bayesian causal forests give the treatment effect its own surface, so shrinkage applies to it separately rather than as a by-product of fitting the outcome — and the propensity score reappears as a predictor inside the prognostic part, where the factorisation permits it, rather than as a weight, where it does not. The number that comes back, USD 1,328, is not remarkable on its own; it sits mid-table beside AIPW and entropy balancing. What is remarkable is that it was reached without using the object the whole frequentist group depends on.

The honest summary of the group is therefore mixed, and worth stating as such. On this dataset the Bayesian route does not win: nearest-neighbour matching still lands closest to the benchmark and does so with far less machinery. What the Bayesian route supplies is the thing the previous group could not supply at all — a way to reason about the confounder nobody recorded, and to report Pr(effect > 0) = 0.827 under a confounder as strong as the worst one measured, rather than a threshold and a shrug.

How the three examples relate

The order is forced rather than chosen: the first example rules out the standard toolkit, which creates the problem the second runs into, which is what the third and the repair exist to address.

What is ruled out

Can a coherent Bayesian use a propensity score at all?

1 · Why the propensity score resists Bayesthe likelihood factorises, so the answer is no — and unbiasedness is what that costs
What is left, and why it fails

If robustness must come from the outcome model, make the outcome model flexible.

2 · Response surfaces and the bias they createUSD 283, the worst adjusted estimate in the collection — then Bayesian causal forests repair it
What only a prior can reach

The assumption itself is unidentified. Put a distribution on it.

3 · Confounding as a parametera posterior over the unmeasured confounder, calibrated against the measured ones
Where it comes from

The frequentist pass over the same data.

Selection on Observablesthe diagnostics that measure something other than what you want to know

The thread joining them is that every prior in this group stands in for something the data cannot identify, which is what separates it from the Bayesian treatment of randomized experiments. There, priors competed against data and lost as information accumulated, and the whole arc reduced to the t-statistic. Here there is no amount of data that settles how strong the unmeasured confounder is, or what the response surface does where treated and control units do not overlap. The prior is not competing with anything — which is the strongest case for using one, and exactly why a careless one would go unchallenged.

Why the Propensity Score Resists Bayes

The opening move, and an uncomfortable one. The likelihood factorises into an assignment part and an outcome part, so with independent priors the treatment-effect posterior cannot involve the propensity model — the object the entire frequentist group is built around. Fitting a complete logistic assignment model jointly with the outcome model moves the answer by USD 18, against a Monte Carlo error — the sampler’s own jitter — of USD 8. What that costs is unbiasedness: where the assignment mechanism is known and the outcome model is misspecified, weighting by the true propensity score is unbiased at −0.03 while the regression carries +0.23 — though weighting pays 3.2× the spread, so on RMSE the biased estimator wins. Then the obvious repair fails instructively: a flexible outcome model attenuates the effect from 2.0 to 0.48, eight times the linear model’s bias and in the opposite direction. That is regularization-induced confounding, and it is why this group needs a third example.

View example →

Response Surfaces and the Bias They Create

If a coherent Bayesian cannot use the propensity score, robustness has to come from the outcome model — so model the outcome and impute the counterfactual. It does not work. A single BART surface returns USD 283 against a benchmark of USD 1,794, the worst adjusted estimate in the collection, and it is unstable: changing only the random seed moves it more than quadrupling the trees does. Its per-unit effects are systematic (R² 0.67) yet swamped by their own uncertainty (signal-to-noise 0.12). The diagnosis is regularization-induced confounding: the shrinkage that makes a model good at predicting destroys it as an estimator of a causal contrast. Bayesian Causal Forests repairs it — a separate treatment surface, with the propensity score admitted to the prognostic part — reaching USD 1,328, and a matched 75-tree test (1,232 against 489) shows the parameterisation rather than the capacity is doing the work. The propensity score returns, but as a predictor rather than a weight.

View example →

Confounding as a Parameter — a Posterior, Not a Threshold

Rosenbaum’s Γ = 1.21 — the factor by which a hidden confounder would have to shift treatment odds within a matched pair — says where the LaLonde result breaks and nothing about how likely you are to be past it. Put a prior on the unobserved confounder instead — calibrated against the confounders actually measured — and integrate it out. The bias formula is verified rather than assumed: dropping each covariate reproduces γ × δ to the dollar. That benchmarking then overturns the usual instinct about which variable to worry about: re74 has the largest outcome coefficient at USD 1,918 per SD yet causes only USD 327 of bias, while black — half as predictive but 1.12 SD out of balance — causes USD 678. Confounding is a product, so ranking suspects by how well they predict the outcome gets this dataset wrong. Against a confounder as strong as the worst one measured, Pr(effect > 0) falls from 0.976 to 0.827, and erasing the effect entirely would take a confounder 2.3× worse than anything in the model.

View example →