DAGs and Discovery on Survey Data

Variables You Can Judge Yourself

The two pages before this one made their arguments on simulations and on the Sachs protein-signalling network, and both were the right choice. A simulation is the only honest way to test whether an identification rule recovers a known effect, and Sachs is the field's benchmark precisely because molecular biology supplies a consensus network to score against.

Both share a limitation, though. You cannot grade the answer yourself. When an algorithm proposes an edge between Plcg and PIP2, a reader without a background in cell signalling has no independent view on whether that is sensible and has to accept the scoring on trust. So this page runs the same two methods on NHANES — the CDC's national health survey — where every variable is one you already have opinions about. Age, sex, income, education, BMI, blood pressure, smoking, exercise, cholesterol, HDL, diabetes, sleep, on 4,048 adults from the 2017–2018 cycle.

The Control That Should Not Be There

Start with a question anyone might ask of this survey: does recreational exercise lower systolic blood pressure, and by how much? The raw comparison is not the answer, because exercisers are younger, richer and better educated, and all three move blood pressure independently. Those are confounders on back-door paths, and blocking them is exactly what the criterion prescribes — age alone takes the raw −4.573 mmHg to −1.600.

The interesting part is what happens next. The near-universal instinct is to keep going: also adjust for BMI, for diabetes, for cholesterol. More controls, more careful. But exercise causes lower BMI and BMI causes higher blood pressure, so BMI sits on the causal path. It is a mediator, not a confounder, and adjusting for it does not remove bias — it removes part of the effect being estimated.

Every row is a defensible-looking specification. The standard errors barely move.

State the obvious before going further, because the rest of this section is easy to misread. The raw association is decisive — a 95% interval of [−5.778, −3.368], nowhere near zero. The adjusted one is not: [−1.970, +0.294], which includes zero. So the honest headline of this cross-section is that essentially the entire apparent benefit of exercise for blood pressure is confounding, most of it age, and what survives is not distinguishable from nothing. That does not make what follows academic — it makes it cleaner. The question below is why the number moves when a variable is added, and that has an exact answer whether or not the number being moved is significant.

Adding BMI costs the estimate 71% of what the defensible set left. And the arithmetic identifies exactly where it went:

The effect travelling through BMI is −0.601 mmHg and the amount the estimate lost is −0.599 — the same number to within 0.002 mmHg. That is not a coincidence to be noted; it is the definition of what happened. Adjusting for a mediator deletes a real channel of the thing being measured and reports the remainder as though it were the whole. Diabetes is downstream too (exercise moves it by −0.324 in log-odds), and cholesterol and HDL repeat the mistake.

So name the three quantities, because “do not adjust for a mediator” is too blunt a rule.

In words: about 72% of what exercise does to blood pressure in this cross-section, it does by way of weight. The remainder — the part that is not about weight — is the direct effect of −0.240 mmHg, with an interval of [−1.360, +0.881] that comfortably contains zero.

Which sharpens what actually went wrong in row five. Adjusting for BMI is not meaningless and does not produce a corrupted number — it produces the direct effect, a real and well-defined quantity somebody might genuinely want. Does exercise lower blood pressure? and does exercise lower blood pressure other than by making you lighter? are different questions with different answers, and row five answers the second. The mistake is not computing it. The mistake is reporting it as an answer to the first — which is what happens whenever BMI enters the model because it was available and seemed relevant, rather than because somebody decided the direct effect was the estimand. Nothing in the regression output records which of the two was intended.

Nothing in the regression output distinguishes the fourth row from the seventh. No fit statistic, no standard error, no significance test. The seventh looks more thorough and is further from the answer. Only the graph separates them, which is the entire case for having drawn one.

Discovery, Graded Without a Consensus Network

Now the second method on the same twelve variables. On Sachs the recovered graph had to be scored against a consensus network from decades of molecular biology. Here something better is available, and much harder to argue with. Two of these variables cannot be caused by anything else in the table. Nothing you do makes you older; nothing in the survey determines your sex. Every arrow pointing into $\texttt{age}$ or $\texttt{female}$ is therefore wrong — not implausible or unsupported but impossible — and no expertise is needed to see it.

PC at $\alpha = 0.01$ returns 27 directed and 1 undirected edge out of 66 possible pairs. Of the eleven edges touching age and sex, it oriented eleven backwards and zero correctly.

Among them: systolic → age, smoker → age, bmi → female, sleep → female.

Not most of them wrong. All of them. And this is not a defect a better algorithm would repair: the orientation rules work from conditional independencies alone, and those are symmetric in a way the world is not. The observational distribution contains no trace of the fact that time runs forwards. Everything the algorithm knows about direction it deduced from collider patterns, and here those patterns point the wrong way every single time.

What it does get right is worth stating, because the failure is specific rather than total. The skeleton is largely sensible — BMI with diabetes, cholesterol with HDL, income with education, age with blood pressure. Discovery is far better at these two are connected than at this one causes that one, which is the Markov-equivalence limit made visible on variables anyone can check. Loosening the test level does not help: at $\alpha = 0.05$ the impossible orientations rise to 12.

What the Corrected Graph Says About the Original Question

The two halves have run side by side without touching. The first estimated an effect under a graph drawn by hand; the second learned a graph and repaired it. The obvious question has not been asked: take the learned graph seriously, read the adjustment set off it, and see whether it agrees. Adjusting for a treatment's parents satisfies the back-door criterion, so the learned graph specifies its own set — and it is not the one drawn by hand.

They agree, 0.135 mmHg apart, about a quarter of a standard error. Two routes to an adjustment set — one from domain reasoning, one learned from data under a constraint — landing in the same place. That is the reassuring result.

The unreserved one is that the learned graph does not contain the edge at all. There is no $\texttt{exercise}$–$\texttt{systolic}$ adjacency at any test level from 0.001 to 0.20. Which is not a contradiction of the first section but its own answer in a different language: the partial correlation runs from $p = 0.004$ given age alone to $p = 0.147$ given the full confounder set, so an algorithm testing for conditional independence deletes the edge. Regression says it with an interval spanning zero; discovery says it by drawing no arrow.

But the two graphs disagree about BMI, and the disagreement matters. The hand-drawn graph has $\texttt{exercise} \rightarrow \texttt{BMI} \rightarrow \texttt{systolic}$, making BMI a mediator. The learned graph puts both exercise and systolic among BMI's parents — $\texttt{exercise} \rightarrow \texttt{BMI} \leftarrow \texttt{systolic}$ — making it a collider. Both forbid adjusting for it, for opposite reasons: under one you would delete a real causal channel, under the other you would open a closed path and manufacture association from nothing. Same instruction, incompatible explanations, identical observable consequences, and no amount of this data settles it. Note too what the learned graph asserts to get there: $\texttt{systolic} \rightarrow \texttt{BMI}$, which runs backwards to the physiology and survived the repair untouched because the only knowledge supplied was that nothing causes age or sex. The repair is exactly as good as what goes into it, and nothing in the output distinguishes the edges knowledge corrected from the ones it never reached.

The Repair Is Knowledge, Not More Data

The obvious response to eleven impossible arrows is to want a better algorithm, more observations, or a more careful test level — and the sweep already showed the last of those making things worse. What actually fixes it is the thing the reader supplied in the first place: nothing causes your age. That fact is not in the data and never will be at any sample size. It is background knowledge, and constraint-based discovery has a formal slot for exactly that.

Forbidding arrows into those two variables removes all eleven errors, which was guaranteed since they were forbidden by hand. The interesting number is the other one. Thirteen edges changed direction in total, so two were repaired that nobody touched — $\texttt{educ} \rightarrow \texttt{exercise}$ and $\texttt{educ} \rightarrow \texttt{income}$ — and the one edge PC could not orient at all is now resolved, leaving zero undirected.

Constraining two variables propagated through the orientation rules and settled parts of the graph having nothing to do with age or sex. Knowledge is not a patch applied to the output; it enters the search and changes what the data is able to say. What comes back reads like something a person might have drawn — age to blood pressure, age to diabetes, sex to BMI — and none of it is a discovery. Every one of those arrows was obvious before the algorithm ran, which is the honest summary: the graph is credible exactly to the extent that a human constrained it, and the parts nobody constrained are the parts to distrust.

The two halves are one argument. In the first, the graph was known and the data could not tell you that a control was wrong — every diagnostic looked fine while the estimate lost most of its magnitude. In the second, the data could not tell you which way an arrow points even when the answer is beyond dispute. Both are the same limit seen from opposite sides. Association is symmetric and causation is not, so the asymmetry has to come from somewhere other than the joint distribution: from an intervention, or from knowing something about the world.

Where this sits

The bad-control result is the mediator material of DAGs & the SCM on real data rather than a simulation, and it is the same “adjusting made it worse” pattern matching found when balance diagnostics selected the wrong estimator — a diagnostic that improves while the answer degrades. The discovery half is causal discovery's Markov-equivalence limit checked against a ground truth needing no consensus network, and it reaches the conclusion Sachs et al. reached by perturbing proteins: identification comes from design and from knowledge, not from analysing the observational distribution harder.

Notebook

Downloads

References

• Pearl, J. (1995). Causal diagrams for empirical research. Biometrika 82(4), 702–710. — the graph and its back-door criterion
• Greenland, S., Pearl, J. & Robins, J. M. (1999). Causal Diagrams for Epidemiologic Research. Epidemiology 10(1), 37–48. — causal diagrams in applied research
• Chickering, D. M. (2002). Optimal structure identification with greedy search. Journal of Machine Learning Research. — structure learning from data