Difference-in-Differences

The Canonical 2×2

The panel page ended by noting that twoway fixed effects is difference-in-differences generalized to many groups and periods. This page starts from the two-group, two-period case where that identity is transparent, and then shows what breaks when treatment timing varies — which is where the specification-dependence seen earlier acquires a name.

Card & Krueger (1994) is the canonical example and remains among the most argued-over results in applied economics. New Jersey raised its minimum wage in 1992; Pennsylvania did not. Comparing 410 fast-food stores — 331 in NJ, 79 in PA — before and after gives the design its cleanest form.

The first thing to check is whether the policy bit at all. Starting wages in NJ went from $4.61 to $5.08; in PA they were flat, $4.63 to $4.62. The treatment is real.

Then the outcome. Full-time-equivalent employment in NJ rose slightly, from 20.44 to 21.03. In PA it fell, from 23.33 to 21.17 — the 1992 downturn. The second difference nets that common shock out, and what remains is the estimate: +2.75. Everything rests on one assumption, called parallel trends: that absent the wage rise, New Jersey would have moved the same way Pennsylvania did. Employment did not fall in New Jersey relative to Pennsylvania; it rose.

Run as a regression, the interaction reproduces the arithmetic exactly at +2.754, now with a clustered standard error of 1.302. That equivalence is the bridge: DiD is twoway fixed effects with a single treatment dummy — group effect, time effect, and their interaction.

What Happens When Timing Varies

Which is where the trouble starts. Real policies rarely switch on for everyone at once; states adopt in different years, firms at different times. The natural move is to keep that twoway fixed-effects (TWFE) regression and let a treatment indicator turn on when each unit adopts. That regression is not estimating what it appears to.

Goodman-Bacon's decomposition shows why. TWFE on staggered data is a weighted average of every possible two-group, two-period comparison — and some of those comparisons use an already-treated group as the control. On a simulation with a known ATT of 3.580, the naive TWFE returns 2.677.

The decomposition uses cell-count weights rather than Goodman-Bacon’s variance weights, so it approximates the pooled estimate (2.536 against 2.677) rather than reproducing it exactly. The direction and source of the bias are what it establishes.

The decomposition locates the loss precisely. Comparisons against never-treated units average 3.407 and against later-treated units 2.756 — both in the right neighbourhood. The comparisons against earlier-treated units average 0.843, and they carry 27% of the weight.

The mechanism is worth stating plainly, because it is not an estimation error. Those comparisons use a control group whose own treatment effect is still growing during the window. A rising control makes the treated group look flat by comparison, so the 2×2 comes back near zero. Every ingredient is a legitimate difference-in-differences; the pooled average of them is biased — which is why the problem went unnoticed for so long.

Using Only Clean Controls

Callaway & Sant'Anna's estimator refuses the contaminated comparisons outright. It computes a separate group-time effect for each adoption cohort at each period, using only clean controls — never-treated or not-yet-treated units — then aggregates. Built from scratch it returns 3.553 against the true 3.580.

The event-study coefficients explain what TWFE was mangling. Pre-treatment coefficients sit near zero, so parallel trends holds; post-treatment effects grow with time since adoption. That dynamic heterogeneity is precisely what makes an already-treated control group unsuitable, and precisely what a single pooled coefficient cannot represent.

The general lesson generalises past DiD. A weighted average of unbiased estimates need not be unbiased when the weights are chosen by the estimator rather than by you. TWFE picks its own comparisons, some of them indefensible, and reports the blend as though it were an average treatment effect.

Where this sits

The 2×2 is twoway fixed effects with one dummy, which is where the sensitivity noticed on that page comes from. The event-study check is the same logic as the placebo cutoffs in regression discontinuity — look where the effect should be absent and confirm that it is. The next page asks what to do when the pre-trends used to license all of this are themselves estimated with error.

Notebooks

Downloads

References

• Card, D. & Krueger, A. B. (1994). Minimum wages and employment: a case study of the fast-food industry in New Jersey and Pennsylvania. American Economic Review. — the New Jersey minimum-wage study
• Ashenfelter, O. & Card, D. (1985). Using the longitudinal structure of earnings to estimate the effect of training programs. Review of Economics and Statistics. — the dip that made pre-trends a standard check
• Bertrand, M., Duflo, E. & Mullainathan, S. (2004). How much should we trust differences-in-differences estimates?. Quarterly Journal of Economics. — serial correlation, and the standard errors it breaks
• Goodman-Bacon, A. (2021). Difference-in-differences with variation in treatment timing. Journal of Econometrics. — what two-way fixed effects is actually averaging when timing varies
• Callaway, B. & Sant'Anna, P. H. C. (2021). Difference-in-differences with multiple time periods. Journal of Econometrics. — the group-time estimator that avoids the bad comparisons
• Sun, L. & Abraham, S. (2021). Estimating dynamic treatment effects in event studies with heterogeneous treatment effects. Journal of Econometrics. — the same problem, in event-study form