Causal Inference VI(b) — Honest DiD: sensitivity to parallel-trends violations

Rambachan & Roth (2023) — how big a violation would overturn the result?

Difference-in-differences and event studies rest on parallel trends — the assumption that, absent treatment, the treated and control groups would have moved in lockstep. It is fundamentally untestable: we only ever see the pre-treatment trends, never the counterfactual post-treatment ones. The universal practice is to eyeball an event study, declare the pre-treatment coefficients "flat enough," and proceed — but pre-trend tests are underpowered (a violation can be present yet statistically invisible), and a flat pre-trend does not guarantee a flat post-trend.

Rambachan & Roth (2023) replace this all-or-nothing ritual with a sensitivity analysis — the DiD analogue of the Rosenbaum-bounds / E-value approach from the matching subsection. Rather than assume exact parallel trends, they ask: how large would a post-treatment violation have to be to overturn the conclusion, relative to the violations we can see in the pre-period? Two restriction families:

• Relative magnitudes $\Delta^{RM}(\bar M)$ — the post-treatment differential trend may deviate by at most $\bar M$ times the largest deviation observed in the pre-period. $\bar M=1$ means "post-treatment violations no bigger than the worst pre-treatment one."
• Smoothness $\Delta^{SD}(M)$ — the differential trend may curve (change slope) by at most $M$ per period, allowing a linear extrapolation of the pre-trend.

For each restriction they compute a robust confidence set for the effect, and the key deliverable is the breakdown value $\bar M^\star$: the point at which the robust CI first includes zero — i.e., how much of a violation the finding can tolerate before losing significance. We build the event study from scratch and, on a robust and a fragile scenario, report the breakdown value computed with the HonestDiD methodology. Python-lead (from-scratch event study; HonestDiD sensitivity via the R companion, loaded here); R companion runs HonestDiD directly.

1. The event study — and why a flat-looking pre-trend is not enough

We simulate two staggered-treatment datasets with a differential trend contaminating the estimate: the treated group drifts relative to the control at rate $g$ per period, so the true treatment effect $\tau$ is entangled with the trend. The event study regresses the outcome on unit and time fixed effects plus leads and lags of treatment (coefficients relative to the period before treatment, $t=-1$). The pre-treatment coefficients ($t<-1$) estimate the differential trend; the post-treatment ones ($t\ge0$) mix the true effect with its continuation.

In the robust scenario the effect is large relative to a small trend ($\tau=1.0$, $g=0.1$); in the fragile scenario the effect is small relative to a larger one ($\tau=0.3$, $g=0.3$). Both are simulated, so the truth is available — and worth using rather than eyeballing.

The event-study coefficient at $k=0$ does not estimate $\tau$. It estimates $\tau+g$, since it is measured against $k=-1$ where the treated group already sits one period below its own drift. The naive estimate is overstated by exactly the per-period trend: 10% in the robust scenario, 100% in the fragile one, where the reported effect is more than double the truth.

Meanwhile the statistic an analyst would actually inspect — the largest first difference among the pre-treatment coefficients — comes out at 0.30 and 0.39. Nearly identical, for biases differing tenfold, and both comparable to the ≈0.14 of sampling noise in a first difference. That is the case for Honest DiD in a line: the diagnostic in universal use cannot separate these two scenarios.

import numpy as np, pandas as pd, matplotlib.pyplot as plt, warnings
warnings.filterwarnings("ignore")
BLUE="#2b6cb0"; RED="#c53030"; GREEN="#2f855a"; ORANGE="#dd6b20"; GREY="#a0aec0"; PURP="#6b46c1"
def event_study(seed,tau,g,N=800):
    rng=np.random.default_rng(seed); periods=np.arange(-4,5); treat=(np.arange(N)<N//2).astype(int); ai=rng.normal(0,1,N)
    rows=[(i,t,treat[i],ai[i]+0.3*t+tau*(treat[i]*(t>=0))+g*treat[i]*t+rng.normal(0,1)) for i in range(N) for t in periods]
    d=pd.DataFrame(rows,columns=["id","t","treat","Y"]); evk=[k for k in periods if k!=-1]
    for k in evk: d[f"D{k}"]=((d.t==k)&(d.treat==1)).astype(int)
    Xt=pd.get_dummies(d.t,prefix="t").astype(float).iloc[:,1:]; U=pd.get_dummies(d.id,prefix="u").astype(float).iloc[:,1:]
    M=np.column_stack([np.ones(len(d)), d[[f"D{k}" for k in evk]].values, Xt.values, U.values]).astype(float)
    b=np.linalg.lstsq(M,d.Y.values,rcond=None)[0]; res=d.Y.values-M@b; s2=res@res/(len(d)-M.shape[1]); V=s2*np.linalg.inv(M.T@M)
    ne=len(evk); return np.array(evk), b[1:1+ne], np.sqrt(np.diag(V)[1:1+ne])
TAU_R,G_R = 1.0, 0.1                        # robust: big effect, small trend
TAU_F,G_F = 0.3, 0.3                        # fragile: small effect, bigger trend
evk,bR,seR=event_study(0,tau=TAU_R,g=G_R)
_,bF,seF=event_study(2,tau=TAU_F,g=G_F)
fig,ax=plt.subplots(1,2,figsize=(13,4.4))
for a,(bb,ss,lab) in zip(ax,[(bR,seR,"ROBUST (τ=1.0, small trend)"),(bF,seF,"FRAGILE (τ=0.3, larger trend)")]):
    pre=evk<0; post=evk>=0
    a.errorbar(evk[pre],bb[pre],yerr=1.96*ss[pre],fmt="o",color=BLUE,capsize=3,label="pre (should be ~0)")
    a.errorbar(evk[post],bb[post],yerr=1.96*ss[post],fmt="s",color=RED,capsize=3,label="post (effect)")
    a.axhline(0,color="k",lw=.6); a.axvline(-0.5,color=GREY,ls=":"); a.set_xlabel("event time"); a.set_ylabel("coefficient"); a.set_title(lab); a.legend(fontsize=8)
plt.tight_layout(); plt.show()
k0=list(evk).index(0)
print(f"  {'scenario':10} {'true tau':>9} {'trend g':>8} {'beta at k=0':>12} {'tau + g':>8} {'overstated by':>14}")
for lab,tau,g,b in (("robust",TAU_R,G_R,bR),("fragile",TAU_F,G_F,bF)):
    print(f"  {lab:10} {tau:>9.2f} {g:>8.2f} {b[k0]:>12.3f} {tau+g:>8.2f} {100*g/tau:>13.0f}%")
print()
print("The event-study coefficient at k=0 does not estimate tau. It estimates tau + g, because it is")
print("measured against k=-1, where the treated group already sits one period below its own drift. The")
print("naive estimate is therefore overstated by exactly the per-period trend -- 10% in one scenario,")
print("100% in the other, where the reported effect is more than double the truth.")
print()
fd_R=np.max(np.abs(np.diff(np.r_[bR[evk<0],0]))); fd_F=np.max(np.abs(np.diff(np.r_[bF[evk<0],0])))
noise=np.median(np.r_[seR,seF])*np.sqrt(2)
print(f"Pre-trend max |first-difference|: robust {fd_R:.2f}, fragile {fd_F:.2f}")
print(f"  sampling noise in a first difference is about {noise:.2f} (coefficient SEs ~{np.median(seR):.2f})")
print()
print("Those two numbers ARE the eyeball test, and they are nearly the same -- while the underlying bias")
print("differs by a factor of ten. In the robust scenario the statistic is mostly noise sitting on a small")
print("real drift; in the fragile one it is mostly signal. Nothing in the number itself separates them,")
print("which is the whole problem: the pre-trend test is underpowered, and 'flat enough' is not a property")
print("you can read off a plot.")
print()
print("That is what Honest DiD replaces -- not 'are the pre-trends flat', but: how large a post-treatment")
print("violation would it take to overturn this, measured against the violations already visible?")

  scenario    true tau  trend g  beta at k=0  tau + g  overstated by
  robust          1.00     0.10        1.095     1.10            10%
  fragile         0.30     0.30        0.691     0.60           100%

The event-study coefficient at k=0 does not estimate tau. It estimates tau + g, because it is
measured against k=-1, where the treated group already sits one period below its own drift. The
naive estimate is therefore overstated by exactly the per-period trend -- 10% in one scenario,
100% in the other, where the reported effect is more than double the truth.

Pre-trend max |first-difference|: robust 0.30, fragile 0.39
  sampling noise in a first difference is about 0.14 (coefficient SEs ~0.10)

Those two numbers ARE the eyeball test, and they are nearly the same -- while the underlying bias
differs by a factor of ten. In the robust scenario the statistic is mostly noise sitting on a small
real drift; in the fragile one it is mostly signal. Nothing in the number itself separates them,
which is the whole problem: the pre-trend test is underpowered, and 'flat enough' is not a property
you can read off a plot.

That is what Honest DiD replaces -- not 'are the pre-trends flat', but: how large a post-treatment
violation would it take to overturn this, measured against the violations already visible?

2. The relative-magnitudes sensitivity and the breakdown value

Honest DiD takes the event-study estimates and their covariance and, for each restriction size $\bar M$, computes a robust confidence interval for the (first) post-treatment effect that holds for any differential trend consistent with $\Delta^{RM}(\bar M)$ — i.e., any post-period violation no larger than $\bar M$ times the worst pre-period one. As $\bar M$ grows, more trends are allowed, so the robust CI widens; the breakdown value $\bar M^\star$ is where it first touches zero.

Interpreting $\bar M^\star$ is intuitive:

• $\bar M^\star > 1$ — the effect survives violations larger than anything visible in the pre-period; robust.
• $\bar M^\star < 1$ — the effect cannot even withstand a violation the size of those already in the pre-trend; fragile.

The two scenarios diverge sharply: the robust one tolerates violations well beyond the pre-period ($\bar M^\star\approx2.5$), while the fragile one breaks near $\bar M^\star\approx1.2$. (The robust CIs are computed with the HonestDiD package — a moment-inequality / fixed-length-CI construction — in the R companion; we load its output here.)

hR=pd.read_csv("honest_.csv"); hF=pd.read_csv("honest__fragile.csv")
def breakdown(h):
    s=h[h.Mbar>=0]; hit=s[s.lb<=0]; return hit.Mbar.iloc[0] if len(hit) else np.nan
brR,brF=breakdown(hR),breakdown(hF)
print("Honest DiD relative-magnitudes robust CIs (first post-treatment effect):")
print(f"  ROBUST : original CI [{hR[hR.Mbar<0].lb.iloc[0]:.2f}, {hR[hR.Mbar<0].ub.iloc[0]:.2f}];  breakdown Mbar* = {brR:.2f}")
print(f"  FRAGILE: original CI [{hF[hF.Mbar<0].lb.iloc[0]:.2f}, {hF[hF.Mbar<0].ub.iloc[0]:.2f}];  breakdown Mbar* = {brF:.2f}")
fig,ax=plt.subplots(figsize=(9,4.8))
for h,br,c,lab in [(hR,brR,GREEN,f"robust (Mbar*≈{brR:.1f})"),(hF,brF,ORANGE,f"fragile (Mbar*≈{brF:.1f})")]:
    s=h[h.Mbar>=0]; ax.fill_between(s.Mbar,s.lb,s.ub,color=c,alpha=.2); ax.plot(s.Mbar,s.lb,color=c,lw=2,label=lab); ax.plot(s.Mbar,s.ub,color=c,lw=2)
    if not np.isnan(br): ax.axvline(br,color=c,ls=":")
ax.axhline(0,color=RED,lw=2,ls="--",label="effect = 0"); ax.set_xlabel("relative-magnitudes bound M̄"); ax.set_ylabel("robust CI for the effect")
ax.set_title("Honest DiD: robust CI widens with M̄; it hits 0 at the breakdown value"); ax.legend(fontsize=8)
plt.tight_layout(); plt.show()
print(f"The robust effect stays significant until a violation ~{brR:.1f}x the pre-period worst; the fragile one only to ~{brF:.1f}x.")
print("Same 'flat-ish' pre-trends, very different robustness -- which a pre-trends eyeball test cannot reveal.")

Honest DiD relative-magnitudes robust CIs (first post-treatment effect):
  ROBUST : original CI [0.90, 1.29];  breakdown Mbar* = 2.50
  FRAGILE: original CI [0.50, 0.89];  breakdown Mbar* = 1.25

The robust effect stays significant until a violation ~2.5x the pre-period worst; the fragile one only to ~1.2x.
Same 'flat-ish' pre-trends, very different robustness -- which a pre-trends eyeball test cannot reveal.

3. Summary

Parallel trends is untestable, and the ubiquitous pre-trends "eyeball" is underpowered — a modest, statistically-invisible violation can still overturn a DiD estimate. Rambachan & Roth's Honest DiD replaces the binary assumption with a sensitivity analysis: bound the unseen post-treatment violation by the seen pre-treatment ones (relative magnitudes $\bar M$, or smoothness), compute a robust confidence set, and report the breakdown value $\bar M^\star$ — how large a violation the conclusion can absorb. On two scenarios with equally innocuous-looking pre-trends, one was robust ($\bar M^\star\approx2.5$: survives violations well beyond the pre-period) and one fragile ($\bar M^\star\approx1.2$: cannot withstand much more than the pre-trend already shows).

Guidance: report a breakdown value alongside every DiD/event-study estimate, and interpret it against the magnitude of confounding trends plausible in the application — exactly as the matching subsection paired estimates with Rosenbaum bounds / E-values. Cross-links: Honest DiD is the difference-in-differences analogue of the sensitivity-analysis notebook (both quantify robustness to an untestable assumption — parallel trends here, unconfoundedness there); it operates on the event-study / staggered-adoption estimates from the DiD notebook (subsection 6), and its partial-identification / robust-CI machinery is kin to the Anderson-Rubin confidence sets of the weak-instrument notebook. The R companion runs the full HonestDiD package, including the smoothness restriction and the sensitivity plots. This completes the depth of the Difference-in-Differences subsection.