Causal Inference V — Panel Data & Fixed Effects¶
Differencing away the confounders you can't see: within, first-difference, random effects, and twoway FE¶
The previous designs each found a way to approximate randomization: matching on observed covariates, an instrument, a threshold. Panel data offers a fourth, and one of the most widely used in applied economics: observe the same units over time, and you can difference away every confounder that is constant within a unit — even the ones you never measured. If a state's driving culture, a firm's management quality, or a person's innate ability is stable over the sample, it cannot explain changes within that unit, so a within-unit estimator sweeps it out for free.
This is the workhorse of empirical microeconomics, and the direct ancestor of difference-in-differences (the next notebook). We build the toolkit from the ground up:
- The unobserved-heterogeneity problem — why pooled OLS is biased when a unit-specific term $\alpha_i$ is correlated with the regressor;
- The within (fixed-effects) estimator — demean each unit's data to eliminate $\alpha_i$; and the first-difference estimator, its close cousin;
- Fixed vs random effects and the Hausman test — the efficiency-vs-consistency trade-off, and how to choose;
- Twoway fixed effects — unit and time effects, the estimator that difference-in-differences is a special case of, with panel-robust clustered standard errors.
The running example is a genuine causal question where fixed effects don't just tidy the estimate — they flip its sign. Data: Stock & Watson's U.S. traffic-fatalities panel (48 states, 1982–1988). Python-lead (from-scratch estimators + linearmodels); R companion uses plm and fixest.
1. The problem — unobserved heterogeneity and the wrong-sign puzzle¶
The data (Stock & Watson) record, for each of the 48 continental U.S. states in each year 1982–1988: the traffic-fatality rate (frate, deaths per 10,000 people), the real beer tax (beertax, a proxy for how aggressively a state discourages drinking-and-driving), and covariates (unemployment, income, minimum drinking age, miles driven). The policy question: does a higher beer tax reduce traffic deaths?
The panel model is $$\text{frate}_{it}=\beta\,\text{beertax}_{it}+\alpha_i+\varepsilon_{it},$$ where $\alpha_i$ is a state fixed effect — everything stable about a state that affects its fatality rate (rural road mileage, driving culture, vehicle mix, geography). The danger: $\alpha_i$ is correlated with beer tax (rural, car-dependent states tend to have both high fatality rates and, for other reasons, particular tax levels), so pooled OLS — which ignores $\alpha_i$ — is confounded. Indeed it produces a notorious wrong-sign result: it suggests higher beer taxes go with more deaths. That is omitted-variable bias, and the rest of the notebook removes it.
import numpy as np, pandas as pd, matplotlib.pyplot as plt, warnings
warnings.filterwarnings("ignore")
BLUE="#2b6cb0"; RED="#c53030"; GREEN="#2f855a"; ORANGE="#dd6b20"; GREY="#a0aec0"; PURP="#6b46c1"
d=pd.read_csv("fatalities.csv")
print(f"Stock-Watson traffic fatalities: {d.state.nunique()} states x {d.year.nunique()} years = {len(d)} obs, {d.year.min()}-{d.year.max()}")
print(d.groupby("year")[["frate","beertax"]].mean().round(3).to_string())
# pooled OLS
Xp=np.column_stack([np.ones(len(d)),d["beertax"].values]); bp=np.linalg.lstsq(Xp,d["frate"].values,rcond=None)[0]
print(f"\nPooled OLS: frate = {bp[0]:.2f} + {bp[1]:.3f} * beertax <-- POSITIVE: 'more tax -> more deaths' (confounded!)")
fig,ax=plt.subplots(1,2,figsize=(13,4.6))
ax[0].scatter(d.beertax,d.frate,s=10,c=GREY,alpha=.5); xg=np.linspace(d.beertax.min(),d.beertax.max(),50)
ax[0].plot(xg,bp[0]+bp[1]*xg,color=RED,lw=2.5,label=f"pooled OLS slope {bp[1]:+.2f}")
ax[0].set_xlabel("beer tax"); ax[0].set_ylabel("fatality rate (per 10,000)"); ax[0].set_title("Pooled: the wrong-sign puzzle"); ax[0].legend()
# within-state view: a few states over time, showing the NEGATIVE within relationship
for s,c in zip(["california","texas","new york","florida","ohio"],[BLUE,GREEN,ORANGE,PURP,RED]):
ss=d[d.state==s]; ax[1].plot(ss.beertax,ss.frate,"o-",ms=4,color=c,alpha=.8,label=s)
ax[1].set_xlabel("beer tax"); ax[1].set_ylabel("fatality rate"); ax[1].set_title("Within states over time, the relationship is negative"); ax[1].legend(fontsize=7)
plt.tight_layout(); plt.show()
print("Across states (pooled) high-tax states look deadlier -- but that is state heterogeneity, not causation. WITHIN a state")
print("over time, raising the tax tracks FEWER deaths. Fixed effects formalize that within-state comparison.")
Stock-Watson traffic fatalities: 48 states x 7 years = 336 obs, 1982-1988
frate beertax
year
1982 2.089 0.530
1983 2.008 0.532
1984 2.017 0.530
1985 1.974 0.517
1986 2.065 0.509
1987 2.061 0.495
1988 2.070 0.480
Pooled OLS: frate = 1.85 + 0.365 * beertax <-- POSITIVE: 'more tax -> more deaths' (confounded!)
Across states (pooled) high-tax states look deadlier -- but that is state heterogeneity, not causation. WITHIN a state over time, raising the tax tracks FEWER deaths. Fixed effects formalize that within-state comparison.
2. The within (fixed-effects) and first-difference estimators¶
The within transformation is the whole idea in one line: subtract each unit's time-average from every variable, $$\text{frate}_{it}-\overline{\text{frate}}_i=\beta\,(\text{beertax}_{it}-\overline{\text{beertax}}_i)+(\varepsilon_{it}-\bar\varepsilon_i),$$ and the fixed effect $\alpha_i$ — being constant within $i$ — vanishes. OLS on the demeaned data is the fixed-effects (within) estimator. It uses only variation over time within each state, so any time-invariant confounder is gone. On the beer-tax data this flips the sign to negative: within a state, raising the beer tax is associated with fewer traffic deaths.
The first-difference estimator is the close relative — subtract the previous period instead of the mean, $\Delta\text{frate}_{it}=\beta\,\Delta\text{beertax}_{it}+\Delta\varepsilon_{it}$. It also eliminates $\alpha_i$, and for $T=2$ periods it is identical to within; for $T>2$ the two differ (they weight the dynamics differently and respond differently to serial correlation). We implement both from scratch and confirm the within estimate against linearmodels' PanelOLS.
from linearmodels.panel import PanelOLS
def within(df, y, xs, unit): # from-scratch within (FE) estimator
g=df.copy()
for c in [y]+xs: g[c]=g[c]-g.groupby(unit)[c].transform("mean")
b=np.linalg.lstsq(g[xs].values,g[y].values,rcond=None)[0]; return dict(zip(xs,b))
def firstdiff(df, y, xs, unit, time):
g=df.sort_values([unit,time]); dd=g.groupby(unit)[[y]+xs].diff().dropna()
b=np.linalg.lstsq(dd[xs].values,dd[y].values,rcond=None)[0]; return dict(zip(xs,b))
fe_s=within(d,"frate",["beertax"],"state")["beertax"]
fd_s=firstdiff(d,"frate",["beertax"],"state","year")["beertax"]
# package cross-check
pdd=d.set_index(["state","year"])
fe_pkg=PanelOLS(pdd.frate, pdd[["beertax"]], entity_effects=True).fit(cov_type="clustered",cluster_entity=True)
bp=np.linalg.lstsq(np.column_stack([np.ones(len(d)),d.beertax]),d.frate,rcond=None)[0][1]
print(f"pooled OLS beertax = {bp:+.3f} (confounded)")
print(f"within (FE) scratch beertax = {fe_s:+.3f}")
print(f"PanelOLS (package) beertax = {fe_pkg.params['beertax']:+.3f} (clustered SE {fe_pkg.std_errors['beertax']:.3f}, p {fe_pkg.pvalues['beertax']:.3f})")
print(f"first-difference beertax = {fd_s:+.3f} (=FE only at T=2; differs here, T=7 -> dynamics/serial corr)")
fig,ax=plt.subplots(figsize=(8,3.6))
names=["pooled OLS","first-difference","within (FE)"]; vals=[bp,fd_s,fe_s]
ax.barh(names,vals,color=[RED,ORANGE,GREEN]); ax.axvline(0,color="k",lw=.8)
for i,v in enumerate(vals): ax.text(v+(0.02 if v>=0 else -0.02),i,f"{v:+.3f}",va="center",ha="left" if v>=0 else "right")
ax.set_xlabel("beer-tax coefficient"); ax.set_title("Removing state fixed effects flips the sign of the estimate")
plt.tight_layout(); plt.show()
print("The within estimator matches PanelOLS exactly, which is the implementation check. Controlling for")
print("time-invariant state characteristics turns a spurious POSITIVE association into a NEGATIVE one.")
print()
print("But read the first-difference row before treating that as settled. Under strict exogeneity FD and FE")
print("are BOTH consistent for the same parameter, so they should agree up to sampling noise. Here they do")
print(f"not: FE gives {fe_s:+.3f} and FD gives {fd_s:+.3f} -- a sign flip, and FD is indistinguishable from zero.")
print()
print("That gap is DIAGNOSTIC rather than incidental. It points to serial correlation in the errors, or")
print("dynamics a static specification omits, or measurement error in the tax variable that differencing")
print("amplifies. Whichever it is, the assumption FE leans on is not clean on this panel, and the FE number")
print("should not be read as though it were.")
pooled OLS beertax = +0.365 (confounded) within (FE) scratch beertax = -0.656 PanelOLS (package) beertax = -0.656 (clustered SE 0.289, p 0.024) first-difference beertax = +0.029 (=FE only at T=2; differs here, T=7 -> dynamics/serial corr)
The within estimator matches PanelOLS exactly, which is the implementation check. Controlling for time-invariant state characteristics turns a spurious POSITIVE association into a NEGATIVE one. But read the first-difference row before treating that as settled. Under strict exogeneity FD and FE are BOTH consistent for the same parameter, so they should agree up to sampling noise. Here they do not: FE gives -0.656 and FD gives +0.029 -- a sign flip, and FD is indistinguishable from zero. That gap is DIAGNOSTIC rather than incidental. It points to serial correlation in the errors, or dynamics a static specification omits, or measurement error in the tax variable that differencing amplifies. Whichever it is, the assumption FE leans on is not clean on this panel, and the FE number should not be read as though it were.
3. Fixed vs random effects, and the Hausman test¶
The random-effects (RE) estimator treats $\alpha_i$ as a random draw uncorrelated with the regressors, and applies a partial (quasi-) demeaning — subtracting only a fraction $\theta$ of each unit's mean. If that uncorrelatedness holds, RE is more efficient than FE (it uses both within- and between-unit variation); if it fails — the usual case in causal work — RE is biased, while FE remains consistent. The choice is the classic efficiency-vs-consistency trade-off.
The Hausman test decides: it checks whether the FE and RE estimates differ by more than sampling noise. A significant difference means $\alpha_i$ is correlated with the regressors, RE is inconsistent, and FE is required. On the beer-tax data the RE estimate is pulled back toward the confounded pooled value, and Hausman rejects RE — confirming we must use fixed effects. (The price of FE: it cannot estimate the effect of any time-invariant regressor — a state's fixed geography, a person's sex — because those are swept out with $\alpha_i$.)
from linearmodels.panel import RandomEffects
from scipy.stats import chi2
re=RandomEffects(pdd.frate, pdd[["beertax"]].assign(const=1)).fit()
b_fe,b_re=fe_pkg.params["beertax"], re.params["beertax"]
v_fe,v_re=fe_pkg.cov.loc["beertax","beertax"], re.cov.loc["beertax","beertax"]
H=(b_fe-b_re)**2/abs(v_fe-v_re); pH=1-chi2.cdf(H,1)
print(f"Fixed effects beertax = {b_fe:+.3f}")
print(f"Random effects beertax = {b_re:+.3f} (pulled toward the confounded pooled estimate)")
print(f"\nHausman test: H = {H:.2f}, p = {pH:.4f}")
print(f" p < 0.05 -> FE and RE differ significantly -> RE is inconsistent here -> USE FIXED EFFECTS.")
fig,ax=plt.subplots(figsize=(7.5,3.2))
ax.errorbar([b_fe],[1],xerr=[1.96*np.sqrt(v_fe)],fmt="s",color=GREEN,capsize=6,ms=10,label="fixed effects")
ax.errorbar([b_re],[0.7],xerr=[1.96*np.sqrt(v_re)],fmt="o",color=BLUE,capsize=6,ms=9,label="random effects")
ax.axvline(0,color=RED,ls="--"); ax.set_yticks([]); ax.set_ylim(0.4,1.3); ax.set_xlabel("beer-tax coefficient")
ax.set_title(f"FE vs RE — Hausman rejects RE (p={pH:.3f})"); ax.legend()
plt.tight_layout(); plt.show()
print("RE's near-zero estimate reflects the same omitted-variable contamination as pooled OLS; the Hausman rejection is the")
print("formal signal to trust FE. In causal panel work, FE is usually the default unless Hausman fails to reject.")
Fixed effects beertax = -0.656 Random effects beertax = -0.052 (pulled toward the confounded pooled estimate) Hausman test: H = 5.36, p = 0.0206 p < 0.05 -> FE and RE differ significantly -> RE is inconsistent here -> USE FIXED EFFECTS.
RE's near-zero estimate reflects the same omitted-variable contamination as pooled OLS; the Hausman rejection is the formal signal to trust FE. In causal panel work, FE is usually the default unless Hausman fails to reject.
4. Twoway fixed effects — and the bridge to difference-in-differences¶
State fixed effects remove time-invariant state confounders, but not shocks that hit all states in a given year — a national recession, a federal seat-belt campaign, gas-price swings. Adding time (year) fixed effects absorbs those, giving the twoway fixed-effects (TWFE) model $$\text{frate}_{it}=\beta\,\text{beertax}_{it}+\alpha_i+\gamma_t+\varepsilon_{it},$$ which compares within-state changes net of the common year-to-year movement. It is the standard specification — and it is exactly the model behind difference-in-differences: the canonical 2×2 DiD (two groups, two periods) is a twoway-FE regression with one treatment indicator. The next notebook builds DiD on this foundation and confronts what happens when treatment timing varies across units (the modern staggered-adoption literature).
We estimate TWFE with clustered standard errors (by state), the panel-robust default that accounts for serial correlation within a unit — without which the standard errors are badly overstated in their precision (Bertrand-Duflo-Mullainathan).
The result is worth reading rather than banking. The coefficient barely moves when year effects are added — from −0.656 to −0.640 — but the standard error grows, and p goes from 0.024 to 0.095. With covariates it reaches 0.144. The sign is stable across every within-state specification; the significance is not, and it is the more defensible specification that loses it.
fe2=PanelOLS(pdd.frate, pdd[["beertax"]], entity_effects=True, time_effects=True).fit(cov_type="clustered",cluster_entity=True)
# also the version with controls, for robustness
ctrl=["unemp","income","drinkage","miles"]
fe2c=PanelOLS(pdd.frate, pdd[["beertax"]+ctrl], entity_effects=True, time_effects=True).fit(cov_type="clustered",cluster_entity=True)
print(f"Twoway FE (state+year), beer tax only = {fe2.params['beertax']:+.3f} (clustered SE {fe2.std_errors['beertax']:.3f}, p {fe2.pvalues['beertax']:.3f})")
print(f"Twoway FE + controls (unemp,income,...) = {fe2c.params['beertax']:+.3f} (clustered SE {fe2c.std_errors['beertax']:.3f}, p {fe2c.pvalues['beertax']:.3f})")
rows={"pooled OLS":bp,"random effects":b_re,"state FE":b_fe,"twoway FE":fe2.params['beertax'],"twoway FE\n+controls":fe2c.params['beertax']}
fig,ax=plt.subplots(figsize=(8.5,4))
cols=[RED,ORANGE,GREEN,BLUE,PURP]; nm=list(rows); vv=[rows[k] for k in nm]
ax.bar(nm,vv,color=cols); ax.axhline(0,color="k",lw=.8)
for i,v in enumerate(vv): ax.text(i,v+(0.02 if v>=0 else -0.03),f"{v:+.2f}",ha="center",fontsize=9)
ax.set_ylabel("beer-tax coefficient"); ax.set_title("The specification ladder: from confounded pooled OLS to twoway FE")
plt.setp(ax.get_xticklabels(),fontsize=8); plt.tight_layout(); plt.show()
print()
print(f" {'specification':34} {'estimate':>10} {'SE':>8} {'p':>8}")
print(f" {'one-way FE (state only)':34} {fe_pkg.params['beertax']:>10.3f} "
f"{fe_pkg.std_errors['beertax']:>8.3f} {fe_pkg.pvalues['beertax']:>8.3f}")
print(f" {'twoway FE (state + year)':34} {fe2.params['beertax']:>10.3f} "
f"{fe2.std_errors['beertax']:>8.3f} {fe2.pvalues['beertax']:>8.3f}")
print(f" {'twoway FE + controls':34} {fe2c.params['beertax']:>10.3f} "
f"{fe2c.std_errors['beertax']:>8.3f} {fe2c.pvalues['beertax']:>8.3f}")
print()
print("The SIGN is stable, and that is worth keeping: every within-state specification agrees the")
print("association runs negative, against the pooled estimate's positive.")
print()
print("The SIGNIFICANCE is not, and stopping at the sign would oversell this panel. One-way FE clears 5%")
print(f"at p = {fe_pkg.pvalues['beertax']:.3f}. Add year effects -- the more defensible model, since traffic fatalities move")
print("nationally with the business cycle, fuel prices and federal safety campaigns -- and p rises to")
print(f"{fe2.pvalues['beertax']:.3f}. Add covariates and it reaches {fe2c.pvalues['beertax']:.3f}. Note WHY: the estimate barely moves")
print(f"({fe_pkg.params['beertax']:.3f} to {fe2.params['beertax']:.3f}); it is the standard error that grows, because year effects absorb")
print("much of the variation the tax was competing to explain.")
print()
print("So this panel supports a negative sign and does not, by itself, support a confident magnitude.")
print("Together with the first-difference disagreement earlier, the beer-tax effect is real-looking but")
print("specification-dependent.")
print()
print("This TWFE regression IS difference-in-differences generalized to many groups and periods -- the next")
print("notebook's engine, and where that specification-dependence finally gets a name.")
Twoway FE (state+year), beer tax only = -0.640 (clustered SE 0.382, p 0.095) Twoway FE + controls (unemp,income,...) = -0.517 (clustered SE 0.352, p 0.144)
specification estimate SE p one-way FE (state only) -0.656 0.289 0.024 twoway FE (state + year) -0.640 0.382 0.095 twoway FE + controls -0.517 0.352 0.144 The SIGN is stable, and that is worth keeping: every within-state specification agrees the association runs negative, against the pooled estimate's positive. The SIGNIFICANCE is not, and stopping at the sign would oversell this panel. One-way FE clears 5% at p = 0.024. Add year effects -- the more defensible model, since traffic fatalities move nationally with the business cycle, fuel prices and federal safety campaigns -- and p rises to 0.095. Add covariates and it reaches 0.144. Note WHY: the estimate barely moves (-0.656 to -0.640); it is the standard error that grows, because year effects absorb much of the variation the tax was competing to explain. So this panel supports a negative sign and does not, by itself, support a confident magnitude. Together with the first-difference disagreement earlier, the beer-tax effect is real-looking but specification-dependent. This TWFE regression IS difference-in-differences generalized to many groups and periods -- the next notebook's engine, and where that specification-dependence finally gets a name.
5. Summary¶
Panel data identify causal effects by differencing out the confounders you cannot observe, as long as they are constant within a unit. On Stock & Watson's traffic-fatality panel, pooled OLS gave the wrong sign (higher beer taxes appeared to raise deaths — pure state heterogeneity), and the within / fixed-effects estimator flipped it to a negative effect by using only within-state variation over time. Random effects is more efficient but only if the heterogeneity is uncorrelated with the regressor — and the Hausman test rejected it, mandating FE.
Two results belong beside that headline rather than behind it. Twoway fixed effects, the more defensible specification, leaves the coefficient almost unchanged but pushes p from 0.024 to 0.095, and to 0.144 with covariates: the sign survives, the significance does not. And the first-difference estimator — consistent under the same strict-exogeneity assumption FE relies on — returns essentially zero, a disagreement that signals serial correlation, omitted dynamics or measurement error rather than a second opinion. What this panel defensibly supports is a negative sign whose magnitude is not pinned down.
The lessons that carry forward:
- FE removes time-invariant confounding for free, but at two costs: it cannot identify time-invariant regressors, and it requires the confounders to actually be time-invariant (a shock that changes a state differentially over time still biases it).
- The FE-vs-RE choice is consistency vs efficiency, decided by Hausman; in causal work FE is the usual default.
- Twoway FE is difference-in-differences generalized — the bridge to the next notebook.
Cross-links. The panel structure (units × time) is the same one modelled dynamically in the BVAR / Multivariate Time Series arc — there the question is forecasting the joint dynamics, here it is identifying a single causal coefficient while sweeping out heterogeneity: same data shape, different target. Next, Difference-in-Differences specializes twoway FE to a treatment that switches on at a point in time, adds the parallel-trends diagnostic, and confronts the recent staggered-adoption critique (Goodman-Bacon; Callaway & Sant'Anna) that has reshaped applied econometrics.