Bertrand Duflo and Mullainathan 2003 — How Much Should We Trust Differences-in-Differences Estimates

Summary

Bertrand, Duflo, and Mullainathan (2003) demonstrate that conventional ordinary least squares (OLS) standard errors in Differences-in-Differences (DiD) regressions are severely biased downward due to serial correlation in outcomes, leading to rejection rates of $44$–$67\%$ on randomly assigned placebo laws instead of the nominal $5\%$. A survey of $92$ DiD papers (1990–2000) confirms almost none correct for this. The paper evaluates several fixes: parametric autoregressive (AR($k$)) corrections fail; clustering by group (state), pre/post aggregation, and block bootstrap all work, with clustering being the most widely adopted solution. Published in the Quarterly Journal of Economics (QJE) 119(1): 249–275 (2004).

Key Claims

• $45\%$ false-rejection rate under naive OLS: Using Current Population Survey (CPS) female wage data ($50$ states $\times$ $21$ years), randomly assigned placebo laws yield $67.5\%$ rejection without grouped-error correction and $44\%$ after clustering at state-year level. Under a Monte Carlo with state-level empirical distribution, the rate is $49\%$. Since the laws have no effect, all rejections are false positives; a well-calibrated test should reject $\approx 5\%$.
• Three interacting factors drive over-rejection: (1) Long time series — average $16.5$ periods across surveyed papers; (2) high serial correlation in typical DiD outcomes — CPS wage residuals: $\hat{\rho}_1 = 0.51$, $\hat{\rho}_2 = 0.44$, $\hat{\rho}_3 = 0.33$; (3) the treatment variable itself is serially correlated (switches on and stays on). When the treatment variable is randomized repeatedly (no serial correlation in $I_{st}$), OLS rejection rate drops to $5\%$.
• Survey of $92$ DiD papers (six journals, 1990–2000): Only $5/92$ attempt to correct for serial correlation. $80/92$ have the grouped-error (Moulton) problem; only $36/80$ correct it. Most papers use employment or wages — both highly positively serially correlated — making the problem near-universal.
• Parametric corrections (AR(1)) fail: AR(1) correction reduces false-rejection from $49\%$ to $24\%$ in CPS Monte Carlo — better but far above $5\%$. Two problems: (a) OLS estimation of the autocorrelation parameter is biased downward in short panels (Nickell 1981); (b) the true data generating process is not AR(1) — mis-specification of the AR structure produces inconsistent standard errors. AR(2) or AR(1)+noise data corrected with AR(1) yields $30$–$39\%$ rejection.
• Clustering by group works for large $N$: The "arbitrary variance-covariance matrix" — Stata's cluster option applied at the state (not state$\times$year) level — produces $6.3\%$ rejection rate at $N = 50$ in CPS Monte Carlo with good power. Degrades with small $N$: $8\%$ for $N = 10$, $11.5\%$ for $N = 6$. Consistent as $N \to \infty$ for fixed panel length (Kezdi 2002).
• Pre/post aggregation works even for small $N$: Collapsing all years into one pre-period and one post-period observation per group, then running DiD on a 2-period panel with a $t$-test adjusted for small sample size, yields $5.3\%$ rejection for $N = 50$ and $5.3\%$ for $N = 10$. But power drops fast: only $16\%$ power against a $2\%$ effect with $N = 50$.
• Block bootstrap works for large $N$: State-level block bootstrap (sampling entire state time series) achieves $6.5\%$ rejection for $N = 50$; degrades faster than clustering as $N$ falls ($13\%$ for $N = 20$, $23\%$ for $N = 10$). Rarely used in practice due to implementation complexity.
• Over-rejection is not a small-$N$ problem: Holding the number of states constant at $50$, over-rejection remains roughly constant as more states are added. The problem scales with time periods, not cross-sectional units — with $7$ time periods, rejection rate is still $15\%$.

Concepts Introduced or Extended

• Difference-in-Differences

Entities Mentioned

• Marianne Bertrand
• Esther Duflo
• Sendhil Mullainathan

Quotes

"Most papers that employ Differences-in-Differences estimation (DD) use many years of data and focus on serially correlated outcomes but ignore that the resulting standard errors are inconsistent."

"For each [placebo] law, we use OLS to compute the DD estimate of its 'effect' as well as the standard error of this estimate. These conventional DD standard errors severely understate the standard deviation of the estimators: we find an 'effect' significant at the 5 percent level for up to 45 percent of the placebo interventions."

"Collapsing the time series information into a 'pre' and 'post' period and explicitly taking into account the effective sample size works well even for small numbers of states."

My Take

This is one of the most influential applied econometrics papers of the 2000s — it changed how practitioners handle inference in DiD settings. The empirical evidence is striking because it is directly actionable: the paper not only identifies a near-universal flaw in the published DiD literature but ranks competing fixes. The recommended solution (clustering at the group level) became standard in virtually every DiD application since, and is now a default in most econometrics software. The paper's survey of 92 published papers showing $87/92$ either ignoring or inadequately correcting serial correlation is the most honest audit of a methodology in the literature. The one limitation: the paper focuses on standard error estimation assuming the parallel trends assumption holds; it does not address threats from differential pre-trends or treatment effect heterogeneity across groups — both of which became major topics in the subsequent DiD literature (Callaway and Sant'Anna 2021, de Chaisemartin and D'Haultfoeuille 2020).