Albert-Dodd (2004) A Cautionary Note on the Robustness of Latent Class Models for Estimating Diagnostic Error without a Gold Standard

Summary

Albert and Dodd (2004) study four latent class models for estimating diagnostic accuracy (sensitivity/specificity) when no gold standard exists. They show that even when models fit the observed data equally well by every standard diagnostic, their estimated sensitivities can diverge by 0.2 or more. Misspecified maximum likelihood estimators (MLEs) converge to pseudo-true parameters that are nearly indistinguishable in likelihood yet yield very different sensitivity estimates, and the number of diagnostic tests required to distinguish models in practice is unrealistically large ($J \geq 10$).

Key Claims

• A latent class model with conditional independence (CI) is identifiable with $J \geq 3$ tests; beta-binomial (BB) and finite mixture (FM) are identifiable at $J \geq 3$; Gaussian random effects (GRE) requires $J \geq 4$.
• The four models produce identical or near-identical fits to data generated by any one of them, yet their sensitivity estimates diverge substantially.
• On the Handelman dentistry data (5 raters, 3,869 X-rays): FM sensitivity $= 0.645$, BB $= 0.518$, GRE $= 0.457$ — a range of ${\approx}0.19$ across equally-fitting models.
• Asymptotic bias example: when true sensitivity $= 0.75$ under GRE, BB estimates $0.55$; expected log-likelihoods agree to 5 decimal places.
• Simulations with $I = 1{,}000$ subjects and $J = 5$ tests: FM model has higher log-likelihood 74.3% of the time even when the true model is GRE; chi-squared ($\chi^2$) goodness-of-fit rejects the wrong model only 4.9%–3.4% of the time.
• Reliable model identification requires $J \geq 10$ tests, which is unrealistic in most diagnostic accuracy studies.

Concepts Introduced or Extended

• Latent Class Model
• Diagnostic Accuracy
• Misspecification

Entities Mentioned

• Paul S. Albert
• Lori E. Dodd

Quotes

"We have shown that even when models for estimating diagnostic accuracy with no gold standard are not identifiable from the data, they are distinguishable from the true model with more raters."

"We caution that if such a study is not feasible, the best one can do is to fit all reasonable models and present sensitivity analyses."

My Take

A sobering identifiability paper. The core insight — that maximum likelihood can converge to a wrong but nearly indistinguishable pseudo-true parameter — is not new, but the concrete numerical illustration (0.2 divergence in sensitivity, expected log-likelihoods equal to 5 decimal places) makes the problem viscerally clear. The practical upshot is pessimistic: with $J = 5$ raters you simply cannot know which latent class structure is correct. The three guidelines (use gold standard when possible; sensitivity analysis across models; collect $J \geq 10$ if no gold standard) are sensible but the third is rarely achievable.