Sensitivity Analysis for Unobserved Confounding

The Question a Sceptic Actually Asks

Every method in this group rests on the same untestable assumption. The matching notebook was blunt about it: balance on the covariates in hand is checkable, but "no hidden confounder" is an article of faith. Sensitivity analysis is the honest response. It does not test the assumption, which is impossible — it asks the quantitative question a sceptic actually cares about.

How strong would an unmeasured confounder have to be, in its association with both treatment and outcome, to explain away the estimated effect? If the answer is "impossibly strong", the finding is robust. If a weak one would suffice, it is fragile. Reporting that number is what separates a credible observational claim from a naive one.

Rosenbaum bounds work on matched pairs. Under the null of no effect and no hidden bias, the treated-minus-control difference within a pair is equally likely to be positive or negative. A hidden confounder breaks that symmetry, and $\Gamma \ge 1$ bounds how unequal the within-pair assignment odds could be. $\Gamma = 1$ is a randomized experiment; $\Gamma = 2$ means a confounder could make one member of a pair twice as likely to be treated. For each $\Gamma$ we compute the worst-case $p$-value of the Wilcoxon signed-rank test, and report $\Gamma^\star$, the point where significance is lost.

Significance and Robustness Are Different Things

The instructive contrast is not robust against fragile, which a $p$-value would also catch. It is two studies with the same $p$-value and different robustness. Significance tracks the standardized effect multiplied by $\sqrt{n}$; $\Gamma^\star$ tracks the standardized effect alone. So a large effect on 30 pairs and a small effect on 480 can arrive at the same $p$-value while tolerating completely different amounts of hidden bias.

The more significant study is the less robust one.

Both studies are overwhelmingly significant, and the small effect is the more significant of the two — $p = 9.3\times10^{-7}$ on 480 pairs against $2.4\times10^{-6}$ on 30. A referee reading $p$-values alone would rank the 480-pair study as the stronger evidence. Rosenbaum bounds reverse that. The large effect holds to $\Gamma^\star = 4.2$; the small one breaks at $\Gamma^\star = 1.4$, where a confounder shifting within-pair odds by barely 43% would erase it.

The reason is that $\Gamma$ measures how far the observed differences could be reshuffled before the sign pattern stops looking systematic, which depends on the effect relative to the noise — not on the sample size that turned it into a small $p$-value. Significance can always be bought with $n$; robustness to hidden bias cannot.

The Previous Page’s Success Story, Reassessed

Then the same machinery is turned on the previous page’s success story. Nearest-neighbour propensity matching recovered the experimental benchmark almost exactly, at USD 1,792 against a randomized truth of USD 1,794. Taking those 185 matched pairs and running Rosenbaum bounds on the within-pair earnings differences gives $\Gamma^\star = 1.21$.

An unmeasured confounder making one member of a matched pair merely 1.21 times more likely to have enrolled would be enough to destroy the significance. The estimate landed on the truth and is fragile — and those are separate properties, which is the whole lesson. It is significant to begin with ($p = 0.003$ at $\Gamma = 1$), so the number means what it appears to mean. This is LaLonde’s 1986 warning, quantified: "we matched and got a plausible number" is not the end of an observational analysis.

The E-value — a Scale-Free Threshold

Rosenbaum bounds need matched pairs. The E-value (VanderWeele & Ding 2017) applies to any estimate expressed as a risk ratio, needs no matching, and has become the most widely reported sensitivity measure for that reason. It gives the minimum strength of association a confounder would need with both treatment and outcome to explain the effect away:

Reported for the point estimate and for the confidence limit nearest the null, it says how much unmeasured confounding the effect — and separately its significance — could tolerate. Calibrated against VanderWeele’s own benchmark, a smoking-and-lung-cancer-scale $RR = 3.9$ yields an E-value of 7.26: a confounder would need roughly sevenfold associations with both smoking and cancer, and no such thing is known, so the finding stands. A weak $RR = 1.3$ yields 1.92, which a modest confounder could supply.

The practical rule is to accompany every observational effect with a sensitivity number — $\Gamma^\star$ for matched designs, an E-value for ratio estimates, both point and interval — and to read it against what confounders could plausibly exist in the application. The number is part of the result, not an appendix to it.

Where this sits

This is the honest companion to Potential Outcomes & Matching and to everything built on unconfoundedness: IPW, doubly-robust AIPW, Double ML and causal survival analysis all inherit the same untestable assumption and the same duty to probe it. Sensitivity analysis is what a DAG’s unmeasured-confounder arrow looks like when you decline to assume it away.

Notebooks

Downloads

References

• Cinelli, C. & Hazlett, C. (2019). Making Sense of Sensitivity: Extending Omitted Variable Bias. Journal of the Royal Statistical Society Series B: Statistical Methodology 82(1), 39–67. — omitted-variable bias, made interpretable
• VanderWeele, T. J. & Ding, P. (2017). Sensitivity Analysis in Observational Research: Introducing the E-Value. Annals of Internal Medicine 167(4), 268–274. — the E-value
• Imbens, G. W. (2003). Sensitivity to Exogeneity Assumptions in Program Evaluation. American Economic Review 93(2), 126–132. — sensitivity to the exogeneity assumption