Brooks Chapman and Schroeder 2018 — Understanding Treatment Effect Estimates When Treatment Effects Are Heterogeneous for More Than One Outcome

Summary

Brooks, Chapman, and Schroeder (2018) extend the essential heterogeneity / local average treatment effect (LATE) framework from single-outcome to multi-outcome treatment choice. When patient–provider dyads make decisions by weighing expected benefits and detriments together — "sorting on the mix" — the true values of average treatment effect on the treated (ATT) and LATE for each outcome depend on the correlation structure of treatment effects across outcomes in the study population, even when the full distribution of benefit effects is identical across populations. Using five simulation scenarios ( $1000$ runs $\times$ $5000$ patients each), they show that (1) estimator identification still holds (regression → ATT, instrumental variables (IV) → LATE, per outcome), but (2) these estimands vary across populations as the benefit/detriment correlation structure shifts, making external validity contingent on matching that structure.

Key Claims

Regression estimators yield ATT and IV estimators yield LATE for each outcome separately — the single-outcome identification results generalize to multi-outcome settings.
True ATT and LATE for the benefit outcome vary substantially across simulation scenarios despite an identical benefit treatment effect distribution across populations — the variation is entirely driven by changes in the benefit/detriment correlation structure.
Treatment-risk paradox (positive correlation between benefit and detriment effects): patients with highest benefit probability also have highest detriment risk; rational sorting on the mix leads them to avoid treatment. Attempting to match "high-risk" patient treatment rates to lower-risk patient rates would cause intolerable detriment increases.
When decision dyads use incorrect detriment expectations (scenarios IV and V), treatment is overused. LATE estimates signal this: the estimated detriment value for marginal patients substantially exceeds the estimated benefit value.
ATT $>$ LATE for benefit when effects are negatively correlated (standard essential heterogeneity); ATT $<$ LATE for benefit is possible under positive correlation (treatment-risk paradox).
External validity requires that the correlation structure of benefit/detriment treatment effects is consistent across populations — a much stronger condition than assumed in most observational studies.
Of $56$ IV studies reviewed in the literature, few discussed limits to generalizing LATE beyond marginal patients in their study population.

Concepts Introduced or Extended

Heterogeneous Treatment Effects

Entities Mentioned

Quotes

"If treatment choices reflect expected effects over more than one outcome, our simulation results showed that treatment effect estimates can provide evidence as to whether treatments were over or underused in the study population. We also showed that these estimates are very sensitive to the distributions of treatment effects across outcomes in each study population."

"Researchers and policy makers should be extremely cautious of generalizing estimates from a single study population to other patient populations."

My Take

The paper makes a genuine extension to the essential heterogeneity framework: treatment-risk paradox is reframed not as a clinical puzzle but as a predictable rational-choice consequence of multi-outcome sorting. The diagnostic logic — compare ATT and LATE jointly across outcomes to infer whether treatment is over/underused — is practical. The main limitation is that the paper uses binary outcomes with linear probability models; the extent to which the simulation results carry over to continuous outcomes or nonlinear models is not assessed. The paper is also primarily a methods contribution; it does not provide empirical evidence of any specific case where single-outcome interpretation led to a policy mistake, making the applied stakes somewhat abstract.