Natural Experiments

Definition

A natural experiment is a research design that exploits exogenous, quasi-random variation in treatment generated by policy changes, government rules, institutional events, or other forces outside the researcher's control. Unlike a true randomized controlled trial, the researcher does not assign treatment; an identifiable external event does so in a way that is plausibly independent of potential outcomes. The goal is to find variation in a key explanatory variable whose determinants are transparent — clearly understood and scrutinizable for exogeneity. "Quasi-experiment" (the term preferred in psychology) is more precise: such studies are not quite experiments. The term "natural experiment" is the dominant usage in economics but misleadingly implies spontaneity and experimental design.

Key Ideas

Core thesis (Meyer 1995): Understanding the source of variation is the primary methodological lesson. If you cannot experimentally control variation, you must understand its source — otherwise the estimate is uninterpretable. Calling a source of variation a natural experiment does not make it exogenous.
Sources of exogenous variation: Cross-state policy variation (U.S. federal system allows state-by-state policy changes); government lotteries (Vietnam draft, Netherlands conscription rates); sharp sudden policy changes affecting some groups but not others; institutional rules (age cutoffs, birthdate thresholds, administrative district boundaries).
Political economy threat: Not every law change is a good natural experiment. Policies driven by past or expected future outcomes create spurious correlations (e.g., high crime → crackdown → lower crime). Remedy: understand the political determinants of the change; apply Granger and Sims exogeneity tests (Cook and Tauchen 1982; Besley and Case 1994).
Antecedents: Rose (1952) on compulsory mediation laws; Simon (1966) on liquor taxes — both used comparison states without the law change. Formal methodology developed by Campbell (1957, 1969), Campbell and Stanley (1966), Cook and Campbell (1979) in psychology.
Instrumental variables (IV) for imprecise assignment: When a natural event shifts treatment probabilities rather than deterministically assigning treatment, it serves as an instrument. Examples: Vietnam draft lottery $\to$ veteran status; quarter of birth $\to$ schooling; policy dummy $\to$ continuous benefit level in two-stage least squares (2SLS). See Instrumental Variables and Local Average Treatment Effect.
External validity limitation: Natural experiments identify effects for specific groups over specific ranges of the explanatory variable — a LATE-like parameter for the complier population defined by the instrument. Conventional studies also assume homogeneous treatment effects but rarely identify which source of variation is influential. The advantage of natural experiments is transparency about the source of variation, not automatically better external validity.
Rosenbaum's control by systematic variation: A method for probing comparability when groups are known to differ on an observable variable — examine whether groups that differ on that variable respond similarly to other changes. If so, parallel trends is more credible. Multiple comparison groups formalize this as a test of overidentifying restrictions.

Area Fixed-Effects Model (Moffitt 2005)

The area fixed-effects model formalizes ecological IV for panel or repeated cross-section data. Write the structural system in first differences:

$\Delta Y_i = \alpha + \beta_i \Delta T_i + \gamma \Delta X_i + \varepsilon_i$ $\Delta T_i = \delta + \theta \Delta X_i + \phi \Delta Z_i + \upsilon_i$

where $\Delta$ denotes change from Time 1 to Time 2, and $\Delta Z_i$ is the change in the area-level environmental variable (a new law, a policy expansion, a price change). Area fixed effects that would be correlated with $T$ in levels cancel out in differencing. $\Delta Z$ then instruments for $\Delta T$ — individual changes in $T$ are still presumed endogenous, so differencing alone is not sufficient.

This is not the individual fixed-effects (FE) model. The individual FE model estimates only $\Delta Y = \alpha + \beta_i \Delta T + \gamma \Delta X + \varepsilon$ (without an instrument for $\Delta T$ ), assuming that differencing eliminates all bias. Moffitt (2005) argues this assumption "leaves unspecified why individual changes in $T$ occur" — those changes may be endogenous. The area FE model ties changes in $T$ to a specific, measurable $\Delta Z$ .

The model can be run on repeated cross-sections (not just panels): aggregate up to area means, or use individual data with area and time dummies and the area $\times$ time interaction of $Z_i$ .

Objections: (1) Residential sorting — location may not be exogenous. (2) Time-varying unobservables correlated with $\Delta Z$ and $\Delta Y$ simultaneously. (3) Lagged adjustment — the model assumes immediate and permanent response, but norms and behavior may adjust slowly. (4) Mechanism distance — the further $Z$ is from individual $T$ , the more spurious mechanisms can generate a spurious $Z \to Y$ association.

Population-Segment Fixed-Effects (Moffitt 2005)

A related design in which nationwide policies affect demographic groups differently (e.g., low-income single mothers vs. married women). Any difference in outcome trends between the affected and unaffected groups is attributed to differential policy exposure. Mathematically equivalent to the area FE model with demographic groups replacing geographic areas.

Moffitt argues this design has weaker support: the assumption that different demographic groups would evolve in parallel absent the policy is "much more suspect" than the comparable geographic assumption. Groups defined by marital status, income, or family structure differ on many time-varying observables and unobservables. See Difference-in-Differences.

Two Types of Extrapolation Failure (Moffitt 2005)

Any instrumental variable, however valid, faces two distinct limits on generalizability:

Mechanism specificity: Each $Z$ represents one specific cause of variation in $T$ . Whether postponing childbearing via abortion policy access has the same effect as postponing it via a labor market shock is an empirical question the framework cannot answer from within. "The effect of $T$ " is an ill-posed question without specifying the mechanism inducing $T$ to change. The structural model assumes $\beta_i$ is independent of the particular $Z_i$ that moved $T$ — but this is an assertion, not a testable property of the data.
Range restriction: $Z$ induces variation in $T$ only across a particular range (e.g., from $30\%$ to $40\%$ smokers). Extrapolation to the rest of the population requires additional assumptions. Instruments inducing larger first-stage variation are preferred for extrapolation but are often harder to defend for internal validity. This is the sharpest formulation of why LATE estimates from examiner IV (Maestas et al. 2013) — which move allowance rates by $\approx 10$ percentage points among marginal applicants — cannot be directly applied to the full applicant pool or to policy interventions of different magnitudes.

Campbell's Threats to Validity

Adapted from Donald Campbell's experimental design literature for economics (Meyer 1995).

Internal Validity (9 threats)

Internal validity: whether the estimated difference was caused by the treatment within the study context.

Omitted variables — events concurrent with the treatment providing alternative explanations for the outcome change
Trends in outcomes — time trends independent of treatment (inflation, secular aging, wage growth)
Misspecified variances — understated standard errors from omitted group error terms; outcomes for units within a group are correlated (clustering)
Mismeasurement — changes in survey methods, question wording, or definitions over the study period
Political economy — endogenous policy changes driven by past or expected future outcomes
Simultaneity — joint determination of treatment and outcome
Selection — nonrandom assignment correlated with potential outcomes; Ashenfelter's dip (earnings dip precedes training program entry)
Attrition — differential loss of respondents from treatment and comparison groups
Omitted interactions — differential trends or omitted variables that affect treatment and comparison groups differently; this is precisely what the parallel trends assumption rules out in Difference-in-Differences

External Validity (3 threats)

External validity: whether effects generalize to other individuals, settings, and time periods.

Interaction of selection and treatment — the treatment group is unrepresentative; the estimated effect applies only to that group at that margin
Interaction of setting and treatment — effects vary across geographic or institutional contexts
Interaction of history and treatment — effects vary across time periods; temporary vs. permanent policy changes may have different effects as institutions adapt

Research Design Hierarchy

One-group before/after (simple differences): Weak. Requires all variation in outcomes attributable to the treatment. Useful as preliminary analysis or when a strong single-group design is available.
Before/after with untreated comparison group (Difference-in-Differences): The dominant design. Differences out common time trends and time-invariant group differences. Key assumption: no omitted interaction between group membership and time period.
Triple-differences (higher-order interactions): Treatment defined as a three-way interaction (state × demographic group × post-period). Removes state×time and state×group trends. Examples: Gruber (1994) maternity mandates; Yelowitz (1994) Medicaid expansions.

Open Questions

How should applied researchers communicate the LATE-like nature of natural experiment estimates, including which specific group the estimate applies to?
When the political economy of a policy change is complex or unknown, how should researchers bound the potential endogeneity bias?
Can LATE estimates from natural experiments with different complier populations be combined into population-level parameters?