Confounding as a Parameter

A Threshold Is Not an Answer

The sensitivity notebook ended on a number that is hard to act on. Nearest-neighbour propensity matching recovered the experimental benchmark almost exactly — USD 1,792 against a randomized truth of USD 1,794 — and Rosenbaum bounds then showed the result is destroyed by a confounder shifting within-pair treatment odds by a factor of only $\Gamma = 1.21$.

That is a tipping point. It says where the conclusion breaks and nothing about whether we are likely to be past it. A referee asking "is 1.21 a lot?" gets no answer from the number itself, because $\Gamma$ is a worst case over all confounders of that strength, and worst cases are rarely where reality sits.

This page asks it the other way round. Instead of finding the confounder strength at which the result dies, put a prior on how strong the unobserved confounder plausibly is and integrate it out. What comes back is a posterior on the treatment effect that already accounts for hidden confounding — a distribution rather than a breakpoint.

The move that makes this honest rather than decorative is benchmarking. The prior on the unobserved confounder is calibrated against the confounders that were measured, so "plausible" means no stronger than the eight covariates already in the model. That is a defensible claim rather than a guess, and Imbens (2003) made exactly this argument on exactly this dataset.

What a Confounder Would Have to Do

If a confounder $U$ is omitted, the estimate is biased by a quantity with a clean structure — $\gamma_U$ is how much $U$ moves the outcome (dollars per standard deviation, covariates held fixed), and $\delta_U$ is how imbalanced $U$ is between arms after adjustment. A confounder must do both to matter.

Both quantities are measurable for the covariates we do observe, which is what makes calibration possible — and the formula is checked rather than assumed. Dropping each covariate in turn and refitting reproduces $\gamma_j\delta_j$ to the dollar, with a largest discrepancy of zero. That is what licenses applying the same arithmetic to a confounder nobody can measure.

Regression-adjusted estimate USD 1,548 (SE 781) against a randomized benchmark of USD 1,794. Largest gap between the formula and the refit: USD 0.

The worst confounder here is not the variable that predicts earnings best. re74 has by far the largest outcome coefficient — USD 1,918 per standard deviation — and contributes only USD 327 of bias, because after adjustment it is barely 0.17 standard deviations out of balance. black is about half as predictive of earnings but sits 1.12 standard deviations apart between the arms (84% of NSW trainees against 20% of CPS respondents) and contributes USD 678, the largest of the eight.

Confounding is a product, so a variable can be strongly prognostic and harmless, or weakly prognostic and devastating. Ranking candidate confounders by how well they predict the outcome — the usual instinct, and the one behind most "but did you control for…" objections — gets this dataset wrong.

A Prior for Something Nobody Measured

Three calibrations are run, from generous to severe, with half-normal priors on the magnitudes and the imbalance free to take either sign, since a confounder is as free to hide a real effect as to manufacture a false one. Sampling uncertainty in $\hat\tau$ enters at the same time, so the posterior mixes both sources of doubt rather than reporting them separately.

Worst observed bias for comparison: USD 678. Sampler health: maximum r-hat 1.000 across all three fits — 1.00 is what a converged chain looks like — and one divergence in the first.

The implied bias column is the calibration made auditable rather than trusted: it is what each prior actually asserts about the damage $U$ could do, against a worst observed bias of USD 678. Note that a half-normal whose scale is set to the largest observed value still places plenty of mass above it, so these labels are generous rather than strict. The third row is frankly beyond anything the data support, and is included to show how quickly the analysis goes uninformative once that is allowed.

Now the part that is easy to misread. The posterior mean barely moves across the three rows — 1,559, then 1,555, then 1,699 — while the interval widens from about USD 3,000 to USD 38,000. Because the confounder is as free to hide a real effect as to invent one, symmetric confounding uncertainty widens the posterior without shifting it.

So sensitivity analysis is not a bias correction. It does not say the answer was really smaller. It says how much less is known than the unadjusted interval claimed — which is a different and less comfortable kind of result, and the reason it is so often reported as a single threshold instead.

What It Buys Over the Threshold

The question can also be asked in reverse, on the only scale anyone has intuition for. Wiping out the estimate entirely requires USD 1,548 of bias. The worst confounder actually in the model supplies USD 678. So the unobserved confounder would have to be about 2.3 times as damaging as the worst one we measured — a claim a subject-matter referee can actually argue with.

That is the reporting a threshold cannot produce. $\Gamma^\star = 1.21$ states that a confounder of that strength exists which would overturn the finding; it attaches no probability to that confounder being real, and says nothing about confounders of any other strength. The posterior prices the whole range, so the reportable claim becomes "positive with probability 0.83 even against a confounder as strong as the worst one we measured" rather than "it breaks at 1.21".

The limit is worth stating plainly. None of this is a test. If the true confounder is genuinely unlike anything observed — a selection process the CPS never asked about — the calibration has nothing to say, and the third row shows how fast the intervals become useless once that possibility is admitted. Benchmarking makes the prior defensible; it does not make it true.

Where this sits

This is the Bayesian counterpart to Rosenbaum bounds and the E-value, and it inherits the estimate produced by matching and regression adjustment. It is also the one place in the causal collection where a prior is doing something no amount of data could: standing in for a variable that was never recorded. The Bayesian analysis of randomized experiments asks the opposite question, where identification is settled and the prior competes against data rather than substituting for it.

Notebook

Downloads

References

• Rubin, D. B. (1978). Bayesian Inference for Causal Effects: The Role of Randomization. The Annals of Statistics 6(1). — causal effects as a missing-data problem
• Rosenbaum, P. R. & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika 70(1), 41–55. — adjustment by the propensity score
• Wang, C., Parmigiani, G. & Dominici, F. (2012). Bayesian Effect Estimation Accounting for Adjustment Uncertainty. Biometrics 68(3), 661–671. — carrying adjustment uncertainty through to the estimate
• Zigler, C. M. & Dominici, F. (2014). Uncertainty in Propensity Score Estimation: Bayesian Methods for Variable Selection and Model-Averaged Causal Effects. Journal of the American Statistical Association 109(505), 95–107. — model averaging over which confounders to include