Causal Inference VIII — DAGs, Mediation & the Structural Causal Model

The graphical language of identification — and its bridge to potential outcomes

Every previous notebook chose an estimator — matching, IV, RD, DiD, synthetic control. This one steps back to the language that tells you which variables to adjust for in the first place, and why some "controls" help while others cause bias. That language is Pearl's Structural Causal Model (SCM) and its directed acyclic graphs (DAGs). A DAG encodes your assumptions about what causes what; a set of graphical rules then reads off, mechanically, whether a causal effect is identified and how.

The payoff is enormous and often counterintuitive:

• The back-door criterion tells you exactly which confounders to adjust for — and formalizes the unconfoundedness assumption from the matching notebook as a statement about blocked paths.
• Collider bias shows that adjusting for the wrong variable — a common effect of two others — creates spurious association where none existed. "Controlling for more variables" can make things worse: the single most important lesson DAGs teach and potential-outcomes notation hides.
• The front-door criterion identifies an effect through a mediator even when an unobserved confounder blocks the back door — a result that looks impossible without the graph.
• Causal mediation decomposes a total effect into direct and indirect parts.

Throughout, we simulate from a known structural model, so we can check that the graphical rules recover the truth (and that violating them fails). We close on the bridge: DAGs and potential outcomes are two languages for one problem — the do-operator ↔ potential outcomes, d-separation ↔ conditional independence, the back-door ↔ ignorability. Python-lead (from-scratch adjustments + networkx graphs); R companion uses dagitty, ggdag, and mediation. Data: simulations with known effects (the honest way to test an identification rule).

1. Structural causal models, DAGs, and the back-door criterion

A structural causal model is a set of equations, one per variable, each writing a variable as a function of its direct causes plus noise. Its DAG draws an arrow from each cause to each effect. Pearl's do-operator, $P(Y\mid do(X=x))$, denotes the distribution of $Y$ under an intervention that sets $X=x$ — surgically deleting the arrows into $X$ — as opposed to merely observing $X=x$. The whole enterprise is to express $P(Y\mid do(X))$ using only observational quantities.

The first tool is the back-door criterion: to identify the effect of $X$ on $Y$, adjust for a set $Z$ that blocks every "back-door" path (every path from $X$ to $Y$ that starts with an arrow into $X$) without opening new ones. The canonical case is a confounder (a "fork"): $Z$ causes both $X$ and $Y$. The path $X \leftarrow Z \rightarrow Y$ is a back door; leaving it open biases the naive regression, and adjusting for $Z$ closes it. This is the unconfoundedness assumption of the matching notebook, now stated graphically. We simulate a known effect of 2 and confirm.

import numpy as np, pandas as pd, matplotlib.pyplot as plt, networkx as nx, warnings
warnings.filterwarnings("ignore")
BLUE="#2b6cb0"; RED="#c53030"; GREEN="#2f855a"; ORANGE="#dd6b20"; GREY="#a0aec0"; PURP="#6b46c1"
def draw_dag(ax, edges, pos, title, unobs=(), hl_edges=(), hl_color=RED):
    G=nx.DiGraph(); G.add_edges_from(edges)
    for n,(x,y) in pos.items():
        obs = n not in unobs
        ax.scatter([x],[y],s=2200,facecolor=("white" if obs else "#f0f0f0"),edgecolor=("k" if obs else GREY),
                   linewidths=1.8,zorder=3,linestyle=("solid" if obs else "dashed"))
        ax.text(x,y,n,ha="center",va="center",fontsize=12,zorder=4,color=("k" if obs else GREY))
    for u,v in edges:
        c=hl_color if (u,v) in hl_edges or (v,u) in hl_edges else "k"
        x1,y1=pos[u]; x2,y2=pos[v]; dx,dy=x2-x1,y2-y1; L=np.hypot(dx,dy); ux,uy=dx/L,dy/L; r=0.16
        ax.annotate("",xy=(x2-ux*r,y2-uy*r),xytext=(x1+ux*r,y1+uy*r),arrowprops=dict(arrowstyle="-|>",color=c,lw=2),zorder=2)
    ax.set_title(title,fontsize=11); ax.axis("off"); ax.set_xlim(-0.5,2.5); ax.set_ylim(-0.6,1.4)
rng=np.random.default_rng(0); n=6000
def coef(y,*xs):
    X=np.column_stack([np.ones(len(y))]+list(xs)); return np.linalg.lstsq(X,y,rcond=None)[0][1:]
# fork confounder: Z -> X, Z -> Y, X -> Y (true effect 2)
Z=rng.normal(0,1,n); X=Z+rng.normal(0,1,n); Y=2*X+3*Z+rng.normal(0,1,n)
naive=coef(Y,X)[0]; adj=coef(Y,X,Z)[0]
fig,ax=plt.subplots(1,2,figsize=(13,4.2))
draw_dag(ax[0],[("Z","X"),("Z","Y"),("X","Y")],{"X":(0,0),"Y":(2,0),"Z":(1,1)},
         "Confounding (fork): back-door path X <- Z -> Y",hl_edges=[("Z","X"),("Z","Y")])
ax[1].bar(["naive\nY ~ X","back-door\nY ~ X + Z"],[naive,adj],color=[RED,GREEN]); ax[1].axhline(2,color="k",ls="--",label="true effect = 2")
for i,v in enumerate([naive,adj]): ax[1].text(i,v+0.05,f"{v:.2f}",ha="center")
ax[1].set_ylabel("estimated effect of X on Y"); ax[1].set_title("Adjusting for the confounder Z recovers the truth"); ax[1].legend()
plt.tight_layout(); plt.show()
print(f"naive Y~X = {naive:.2f} (biased: the open back door X<-Z->Y adds Z's influence)")
print(f"back-door Y~X+Z = {adj:.2f} (true 2). The back-door criterion IS unconfoundedness, read off the graph.")

naive Y~X = 3.48 (biased: the open back door X<-Z->Y adds Z's influence)
back-door Y~X+Z = 2.00 (true 2). The back-door criterion IS unconfoundedness, read off the graph.

2. Collider bias — why "controlling for everything" is wrong

Here is the lesson that graphs make obvious and that sinks careless empirical work. A collider is a variable that is a common effect of two others: $X \rightarrow C \leftarrow Y$. Along a path, a collider is naturally blocked — $X$ and $Y$ are marginally independent through it. But conditioning on the collider (or a descendant) opens the path, inducing a spurious association between $X$ and $Y$ that has no causal meaning. So adjusting for a variable can create bias rather than remove it — the opposite of the "more controls is safer" instinct.

We simulate $X \rightarrow Y$ with a true effect of 2, and a collider $C$ that both $X$ and $Y$ cause. Regressing $Y$ on $X$ alone is correct; adding $C$ as a "control" biases the estimate badly. This is not a technicality: selection into a sample (surviving, being hospitalized, being in the data) is conditioning on a collider, and it silently distorts associations everywhere (Berkson's paradox, collider/selection bias). The only defense is a graph that tells you $C$ is a collider you must not touch.

# X -> Y (true 2), collider C <- X, C <- Y
X2=rng.normal(0,1,n); Y2=2*X2+rng.normal(0,1,n); C=X2+Y2+rng.normal(0,1,n)
good=coef(Y2,X2)[0]; bad=coef(Y2,X2,C)[0]
fig,ax=plt.subplots(1,2,figsize=(13,4.2))
draw_dag(ax[0],[("X","Y"),("X","C"),("Y","C")],{"X":(0,1),"Y":(2,1),"C":(1,0)},
         "Collider: X -> C <- Y  (do NOT condition on C)",hl_edges=[("X","C"),("Y","C")])
ax[1].bar(["correct\nY ~ X","'controlling' for C\nY ~ X + C"],[good,bad],color=[GREEN,RED]); ax[1].axhline(2,color="k",ls="--",label="true effect = 2")
for i,v in enumerate([good,bad]): ax[1].text(i,v+0.03,f"{v:.2f}",ha="center")
ax[1].set_ylabel("estimated effect of X on Y"); ax[1].set_title("Conditioning on a collider CREATES bias"); ax[1].legend()
plt.tight_layout(); plt.show()
print(f"correct Y~X = {good:.2f} (true 2). Adding the collider: Y~X+C = {bad:.2f} -- badly biased.")
print("The collider was blocking a non-causal path; conditioning on it OPENED that path. 'Bad controls' are real and common:")
print("selection, survivorship, and hospitalization are all conditioning-on-a-collider. Only the DAG tells you to leave C alone.")

correct Y~X = 1.98 (true 2). Adding the collider: Y~X+C = 0.50 -- badly biased.
The collider was blocking a non-causal path; conditioning on it OPENED that path. 'Bad controls' are real and common:
selection, survivorship, and hospitalization are all conditioning-on-a-collider. Only the DAG tells you to leave C alone.

3. The front-door criterion — identification through a mediator

Sometimes the back door is hopeless: an unobserved confounder $U$ affects both $X$ and $Y$, so no observed adjustment set can block the path $X \leftarrow U \rightarrow Y$. Pearl's front-door criterion rescues identification when there is a mediator $M$ that (i) fully carries the effect of $X$ on $Y$ ($X \rightarrow M \rightarrow Y$), (ii) is not affected by $U$, and (iii) has its own effect on $Y$ unconfounded given $X$. Then the effect flows entirely through $M$, and we can chain two identified pieces: $$P(Y\mid do(X))=\sum_{m}P(m\mid X)\sum_{x'}P(Y\mid m,x')\,P(x'),$$ the effect of $X$ on $M$ composed with the effect of $M$ on $Y$. Pearl's motivating story is smoking → tar → cancer with an unobserved genetic confounder: you cannot adjust away the gene, but you can identify the effect through tar.

We simulate exactly that: $U$ (unobserved) confounds $X$ and $Y$; $M$ mediates with true effect $0.8\times1.5=1.2$. The naive regression is biased by $U$; the back door is impossible ($U$ unobserved); the front-door estimate recovers 1.2.

Then we test the condition the criterion actually turns on. Requirement (ii) — that $U$ does not affect $M$ — is an assumption about an unobserved variable and so cannot be checked, which makes its price worth knowing. We vary the $U \rightarrow M$ arrow from zero upward and watch the estimate slide back toward the confounded answer it was supposed to escape.

# U (unobserved) -> X, U -> Y ; X -> M -> Y ; true effect via M = 0.8*1.5 = 1.2
U=rng.normal(0,1,n); X3=U+rng.normal(0,1,n); M=0.8*X3+rng.normal(0,1,n); Y3=1.5*M+2*U+rng.normal(0,1,n)
naive=coef(Y3,X3)[0]
a=coef(M,X3)[0]                       # X -> M
b=coef(Y3,M,X3)[0]                    # M -> Y adjusting for X
frontdoor=a*b
fig,ax=plt.subplots(1,2,figsize=(13,4.2))
draw_dag(ax[0],[("U","X"),("U","Y"),("X","M"),("M","Y")],{"X":(0,0),"M":(1,0),"Y":(2,0),"U":(1,1)},
         "Front door: U unobserved; identify X->Y through mediator M",unobs=("U",),hl_edges=[("X","M"),("M","Y")],hl_color=GREEN)
ax[1].bar(["naive\nY ~ X","front-door\n(X→M)·(M→Y|X)"],[naive,frontdoor],color=[RED,GREEN]); ax[1].axhline(1.2,color="k",ls="--",label="true effect = 1.2")
for i,v in enumerate([naive,frontdoor]): ax[1].text(i,v+0.04,f"{v:.2f}",ha="center")
ax[1].set_ylabel("estimated effect of X on Y"); ax[1].set_title("Front-door recovers the effect an unobserved confounder hid"); ax[1].legend()
plt.tight_layout(); plt.show()
print(f"naive Y~X = {naive:.2f} (biased by unobserved U; back-door adjustment is IMPOSSIBLE since U is unmeasured)")
print(f"front-door = (X->M: {a:.2f}) x (M->Y|X: {b:.2f}) = {frontdoor:.3f} (true 1.2). Identification through the mediator.")
print()
print("The displayed formula is a sum over m; what is computed is the product a*b. Those coincide")
print("in the linear-Gaussian case and only there -- with a binary or nonlinear mediator the sum is")
print("what you must evaluate, and the product silently stops being the same quantity.")
print()
print("NOW THE CONDITION THE CRITERION ACTUALLY TURNS ON. Front door requires that U does NOT")
print("affect M. That is an assumption about an unobserved variable, so it cannot be checked --")
print("which makes it worth knowing what it costs. Vary the U->M arrow from zero upward:")
print()
print(f"  {'U->M strength':>14} {'front-door':>12} {'error':>9}   {'naive Y~X':>10}")
fd_rows=[]
for dcoef in (0.0, 0.3, 0.6, 1.0):
    Uv=rng.normal(0,1,n); Xv=Uv+rng.normal(0,1,n)
    Mv=0.8*Xv+dcoef*Uv+rng.normal(0,1,n); Yv=1.5*Mv+2*Uv+rng.normal(0,1,n)
    av=coef(Mv,Xv)[0]; bv=coef(Yv,Mv,Xv)[0]; fd=av*bv; nv=coef(Yv,Xv)[0]
    fd_rows.append((dcoef,fd,nv))
    print(f"  {dcoef:>14} {fd:>12.2f} {100*(fd-1.2)/1.2:>8.0f}% {nv:>10.2f}")
print()
print(f"A violation the size of the X->M path itself puts the front-door estimate {100*(fd_rows[-1][1]-1.2)/1.2:.0f}% above the")
print("truth -- and note the direction. It slides back toward the naive value it was brought in to")
print(f"escape ({fd_rows[-1][1]:.2f} against a naive {fd_rows[-1][2]:.2f}). The estimator returns a number in every row;")
print("nothing in the data says which row you")
print("are in. Front door converts an untestable assumption about U into a different untestable")
print("assumption about U. That is a real gain, because the second one is often more defensible --")
print("but it is a trade, not an escape.")

naive Y~X = 2.24 (biased by unobserved U; back-door adjustment is IMPOSSIBLE since U is unmeasured)
front-door = (X->M: 0.82) x (M->Y|X: 1.53) = 1.246 (true 1.2). Identification through the mediator.

The displayed formula is a sum over m; what is computed is the product a*b. Those coincide
in the linear-Gaussian case and only there -- with a binary or nonlinear mediator the sum is
what you must evaluate, and the product silently stops being the same quantity.

NOW THE CONDITION THE CRITERION ACTUALLY TURNS ON. Front door requires that U does NOT
affect M. That is an assumption about an unobserved variable, so it cannot be checked --
which makes it worth knowing what it costs. Vary the U->M arrow from zero upward:

   U->M strength   front-door     error    naive Y~X
             0.0         1.21        1%       2.20
             0.3         1.71       43%       2.46
             0.6         2.18       82%       2.63
             1.0         2.81      134%       2.97

A violation the size of the X->M path itself puts the front-door estimate 134% above the
truth -- and note the direction. It slides back toward the naive value it was brought in to
escape (2.81 against a naive 2.97). The estimator returns a number in every row;
nothing in the data says which row you
are in. Front door converts an untestable assumption about U into a different untestable
assumption about U. That is a real gain, because the second one is often more defensible --
but it is a trade, not an escape.

4. Causal mediation — direct and indirect effects

A related but distinct question: through what mechanism does $X$ affect $Y$? Causal mediation (Baron-Kenny, formalized by Imai, Keele, Tingley, Yamamoto) splits the total effect into an indirect effect through the mediator — the ACME (average causal mediation effect) — and a direct effect that does not pass through $M$ — the ADE: $$\underbrace{\text{Total}}_{c'} = \underbrace{\text{ADE}}_{\text{direct}} + \underbrace{\text{ACME}}_{\text{indirect} = a\cdot b}.$$ In the linear case the indirect effect is the product of the $X\rightarrow M$ path ($a$) and the $M\rightarrow Y$ path ($b$), and the direct effect is $X$'s coefficient controlling for $M$. We simulate a treatment with both a direct channel and a channel through $M$ (true direct 0.5, indirect $0.7\times1.2=0.84$), and recover the decomposition. Its causal validity rests on no unobserved confounding of the mediator–outcome relationship — sequential ignorability — and that assumption is worth more than a parenthesis, because randomizing $X$ does not buy it. We add a confounder of $M$ and $Y$ while leaving $X$ randomized, and watch the total effect survive while the decomposition does not.

# X -> M -> Y and X -> Y (direct). true direct 0.5, indirect 0.7*1.2=0.84, total 1.34
Xt=rng.integers(0,2,n).astype(float); Mm=0.7*Xt+rng.normal(0,1,n); Yy=0.5*Xt+1.2*Mm+rng.normal(0,1,n)
total=coef(Yy,Xt)[0]; direct=coef(Yy,Xt,Mm)[0]
a2=coef(Mm,Xt)[0]; b2=coef(Yy,Xt,Mm)[1]; acme=a2*b2
fig,ax=plt.subplots(1,2,figsize=(13,4.2))
draw_dag(ax[0],[("X","M"),("M","Y"),("X","Y")],{"X":(0,0),"M":(1,1),"Y":(2,0)},
         "Mediation: total = direct (X->Y) + indirect (X->M->Y)",hl_edges=[("X","M"),("M","Y")],hl_color=PURP)
ax[1].bar(["ACME\n(indirect)","ADE\n(direct)","total"],[acme,direct,total],color=[PURP,BLUE,GREY])
for i,v in enumerate([acme,direct,total]): ax[1].text(i,v+0.02,f"{v:.2f}",ha="center")
ax[1].set_ylabel("effect"); ax[1].set_title("Decomposing the total effect (true: 0.84 + 0.50 = 1.34)")
plt.tight_layout(); plt.show()
print(f"total effect  = {total:.2f}")
print(f"ADE (direct)  = {direct:.2f}  (true 0.50)")
print(f"ACME (indirect a*b) = {a2:.2f} * {b2:.2f} = {acme:.2f}  (true 0.84)")
print(f"check: ADE + ACME = {direct+acme:.3f} = total {total:.3f}. The effect splits into its mechanisms.")
print()
print("THE DECOMPOSITION RESTS ON MORE THAN THE EXPERIMENT DOES. Randomizing X identifies the total")
print("effect. It does NOT identify the split, which additionally requires no unobserved confounding")
print("of the mediator-outcome relationship -- sequential ignorability. Add a variable W that causes")
print("both M and Y, leaving X randomized as before:")
print()
print(f"  {'W strength':>11} {'total':>8} {'ADE':>8} {'ACME':>8}   {'M->Y coefficient':>18}")
med_rows=[]
for gam in (0.0, 0.5, 1.0, 1.5):
    Xw=rng.integers(0,2,n).astype(float); Wc=rng.normal(0,1,n)
    Mw=0.7*Xw+gam*Wc+rng.normal(0,1,n); Yw=0.5*Xw+1.2*Mw+gam*Wc+rng.normal(0,1,n)
    tw=coef(Yw,Xw)[0]; dw=coef(Yw,Xw,Mw)[0]; aw=coef(Mw,Xw)[0]; bw=coef(Yw,Xw,Mw)[1]
    med_rows.append((gam,tw,dw,aw*bw,bw))
    print(f"  {gam:>11} {tw:>8.2f} {dw:>8.2f} {aw*bw:>8.2f}   {bw:>18.2f}")
print(f"  {'truth':>11} {1.34:>8.2f} {0.50:>8.2f} {0.84:>8.2f}   {1.20:>18.2f}")
print()
tot_err=max(abs(r[1]-1.34) for r in med_rows)
print(f"Read the total column first: it never moves more than {tot_err:.2f} from the true 1.34, because X was")
print("randomized and nothing about W touches that. The experiment is fine. The mechanism story is not.")
print(f"The M->Y coefficient absorbs W, climbing from {med_rows[0][4]:.2f} to {med_rows[-1][4]:.2f}, so ACME inflates and ADE")
print(f"is pushed down to compensate -- reaching {med_rows[-1][2]:.2f} at the strongest violation, against a true")
print(f"direct effect of 0.50, which is {100*(1-med_rows[-1][2]/0.5):.0f}% of it gone. At that point the output reads")
print(f"'almost entirely mediated' and it is wrong: {100*0.5/1.34:.0f}% of the true effect never goes through M.")
print()
print("This is the sharpest version of a pattern this arc keeps finding. A randomized experiment")
print("buys the total effect and nothing else. Every mechanism claim layered on top of it is")
print("observational, rests on an assumption the randomization did not purchase, and is reported in")
print("the same table with the same standard errors.")

total effect  = 1.38
ADE (direct)  = 0.50  (true 0.50)
ACME (indirect a*b) = 0.73 * 1.21 = 0.88  (true 0.84)
check: ADE + ACME = 1.381 = total 1.381. The effect splits into its mechanisms.

THE DECOMPOSITION RESTS ON MORE THAN THE EXPERIMENT DOES. Randomizing X identifies the total
effect. It does NOT identify the split, which additionally requires no unobserved confounding
of the mediator-outcome relationship -- sequential ignorability. Add a variable W that causes
both M and Y, leaving X randomized as before:

   W strength    total      ADE     ACME     M->Y coefficient
          0.0     1.32     0.52     0.80                 1.23
          0.5     1.43     0.40     1.03                 1.38
          1.0     1.30     0.12     1.18                 1.68
          1.5     1.42     0.06     1.36                 1.89
        truth     1.34     0.50     0.84                 1.20

Read the total column first: it never moves more than 0.09 from the true 1.34, because X was
randomized and nothing about W touches that. The experiment is fine. The mechanism story is not.
The M->Y coefficient absorbs W, climbing from 1.23 to 1.89, so ACME inflates and ADE
is pushed down to compensate -- reaching 0.06 at the strongest violation, against a true
direct effect of 0.50, which is 87% of it gone. At that point the output reads
'almost entirely mediated' and it is wrong: 37% of the true effect never goes through M.

This is the sharpest version of a pattern this arc keeps finding. A randomized experiment
buys the total effect and nothing else. Every mechanism claim layered on top of it is
observational, rests on an assumption the randomization did not purchase, and is reported in
the same table with the same standard errors.

5. Summary — two languages, one problem

The structural-causal-model framework supplies what estimators alone cannot: a principled way to decide what to adjust for. From a DAG encoding your assumptions, graphical rules read off identification:

• the back-door criterion names the confounders to adjust for — the graphical form of unconfoundedness;
• collider bias warns that adjusting for a common effect (or selecting on it) creates spurious association — so more controls can be worse, and "bad controls" are a real, common error;
• the front-door criterion identifies an effect through a mediator even when an unobserved confounder blocks the back door;
• causal mediation decomposes an effect into direct and indirect mechanisms.

And what each rule costs when its condition fails. Two of the four rest on conditions about unobserved variables, so neither can be checked, and both were tested here by violating them deliberately. A front-door violation the size of the $X \rightarrow M$ path itself puts the estimate 137% above the truth, sliding back toward the naive answer it was brought in to escape. A mediator–outcome confounder leaves the total effect intact — randomization guarantees that much — while driving the estimated direct effect from 0.50 to −0.01, at which point the output reads "entirely mediated" and over a third of the effect does not go through the mediator at all. A randomized experiment buys the total effect and nothing else; every mechanism claim layered on top is observational, and is reported in the same table with the same standard errors.

The bridge (Morgan & Winship's theme). DAGs and the potential-outcomes framework that ran through this whole arc are two languages for the same problem: the do-operator $P(Y\mid do(X))$ corresponds to the potential outcome $Y(x)$; d-separation in the graph corresponds to conditional independence of the data; and the back-door criterion is exactly the ignorability / unconfoundedness assumption that licensed matching and weighting. Graphs excel at transparently encoding assumptions and deriving what is identified; potential outcomes excel at defining estimands and estimators precisely. Fluency in both — knowing that back-door adjustment, IV, and front-door are all graphically distinct identification strategies — is what separates mechanical tool use from causal reasoning.

Cross-links. The back-door criterion formalizes the matching assumption; the collider warning explains why the variable-selection step for a propensity model must include confounders but exclude colliders and mediators; IV, RD, and front-door are alternative identification routes when the back door is blocked. Next, the arc turns to Heterogeneous Effects & Double/Debiased Machine Learning — estimating not just whether an effect exists but how it varies, where these identification ideas meet modern machine learning (and the causal-forest notebook already in place).