Honest DiD

Why a Flat-Looking Pre-Trend Is Not Enough

Difference-in-differences rests on parallel trends — that absent treatment, the groups would have moved in lockstep. It is untestable by construction: only the pre-treatment trends are ever observed, never the counterfactual post-treatment ones. The universal practice is to plot an event study, judge the pre-treatment coefficients flat enough, and proceed.

Two things are wrong with that. Pre-trend tests are underpowered, so a violation can be present and statistically invisible. And a flat pre-trend does not imply a flat post-trend — the assumption being licensed is about a period the data never cover.

Because both scenarios here are simulated, the truth is available, and it makes the failure exact rather than rhetorical. The event-study coefficient at $k=0$ does not estimate $\tau$. It estimates $\tau + g$, since it is measured against $k=-1$ where the treated group already sits one period below its own drift. The naive estimate is overstated by precisely the per-period trend.

Pre-trend max |first difference|: 0.30 robust, 0.39 fragile — against sampling noise of about 0.14.

And the diagnostic in universal use cannot tell these apart. The largest first difference among the pre-treatment coefficients — the number an analyst actually inspects — comes out at 0.30 and 0.39. Nearly identical, for biases differing tenfold, and both comparable to the 0.14 of sampling noise in a first difference. In the robust scenario that statistic is mostly noise resting on a small real drift; in the fragile one it is mostly signal. Nothing in the number separates them.

Pricing the Assumption Instead of Asserting It

Rambachan & Roth (2023) replace the binary ritual with a sensitivity analysis, and the move is the same one Rosenbaum bounds make for unconfoundedness: stop asserting the assumption, and ask how large a violation the conclusion could absorb. Rather than assume exact parallel trends, bound the unseen post-treatment violation by the seen pre-treatment ones.

The relative-magnitudes restriction $\Delta^{RM}(\bar M)$ allows the post-treatment differential trend to deviate by at most $\bar M$ times the largest deviation visible before treatment. $\bar M = 1$ means "no worse than the worst thing already in the pre-period". For each $\bar M$ a robust confidence interval is computed that holds for any trend satisfying the restriction; as $\bar M$ grows the interval widens, and the breakdown value $\bar M^\star$ is where it first touches zero.

That number is directly interpretable. Above 1, the effect survives violations larger than anything the pre-period displays. Below 1, it cannot withstand even a violation the size of those already visible.

The two scenarios diverge sharply, on pre-trends an analyst would have waved through in both cases. The robust finding tolerates a violation 2.5× the pre-period worst; the fragile one breaks at 1.25× — barely more than what is already on the plot.

The threshold at $\bar M \approx 1$ is not arbitrary, and the simulation shows why. A post-treatment violation "as large as the pre-period's" is exactly what a constant differential trend delivers, since the drift does not stop when treatment starts. An estimate that breaks below $\bar M = 1$ is one that requires the trend to politely halt at the treatment date — which nothing in the design guarantees and nothing in the data can check.

One note on provenance. The event studies are built from scratch here, but the robust confidence sets are computed by the HonestDiD package — a moment-inequality construction — in the R companion, and loaded into the Python notebook as data. The breakdown values are read off those sets.

Where this sits

This is the difference-in-differences analogue of Rosenbaum bounds and the E-value: both price an untestable assumption instead of asserting it, and both report a threshold rather than a verdict. It operates on the event-study estimates from the previous page, whose staggered-adoption problem is a separate failure that Honest DiD does not address — a Callaway–Sant'Anna estimate, which fixes the comparison groups rather than the trend assumption, can still rest on a parallel-trends assumption that will not bear weight. And the partial-identification machinery is kin to the Anderson–Rubin confidence sets, which are also built by collecting the parameter values a test cannot reject.

Notebooks

Downloads

Both scenarios are simulated, because separating a robust finding from a fragile one requires knowing which is which.

References

• Rambachan, A. & Roth, J. (2023). A More Credible Approach to Parallel Trends. Review of Economic Studies 90(5), 2555–2591. — bounding violations of parallel trends
• Roth, J. (2022). Pretest with Caution: Event-Study Estimates after Testing for Parallel Trends. American Economic Review: Insights 4(3), 305–322. — why pre-trend tests mislead
• Callaway, B. & Sant'Anna, P. H. C. (2021). Difference-in-Differences with multiple time periods. Journal of Econometrics 225(2), 200–230. — the estimator being made honest