Diagnostic Accuracy

Definition

Diagnostic accuracy refers to the ability of a binary or ordinal test to correctly classify subjects as diseased or non-diseased. The canonical measures are sensitivity (probability of a positive test given disease) and specificity (probability of a negative test given no disease). When a gold-standard reference classification is available, these are estimable from a contingency table. When no gold standard exists, latent class models treat true disease status as a latent variable and estimate sensitivity and specificity jointly from the marginal agreement pattern across $J \geq 3$ fallible tests.

Key Ideas

• Sensitivity and specificity: For a binary test $T$ and true status $D \in \{0,1\}$:
  • Sensitivity: $\pi_e = P(T=1 \mid D=1)$
  • Specificity: $\pi_c = P(T=0 \mid D=0)$
  • These are properties of the test, not the population, but prevalence $\rho = P(D=1)$ must also be estimated.
• No-gold-standard problem: In many diagnostic studies (e.g., radiological assessment, psychiatric diagnosis), no criterion classifier exists. Four latent class models are commonly used:
  1. Conditional Independence (CI): $J$ tests independent given latent $D$; identifiable at $J \geq 3$.
  2. Beta-Binomial (BB): Test responses exchangeable given $D$; identifiable at $J \geq 3$.
  3. Finite Mixture (FM): Two components within each disease class; identifiable at $J \geq 3$.
  4. Gaussian Random Effects (GRE): Continuous frailty for each subject; identifiable at $J \geq 4$.
• Pseudo-true parameter problem (Albert-Dodd 2004): All four models can fit observed data equally well (identical log-likelihoods to 5 decimal places) yet produce sensitivity estimates differing by 0.2 or more. Maximum likelihood estimation (MLE) converges to the wrong pseudo-true parameters when the model is misspecified. Reliable model identification requires $J \geq 10$ raters — unrealistic in practice.
• Practical implication: With $J = 5$ tests, there is no statistical basis for preferring one model over another. The FM model has higher log-likelihood than the true GRE model 74% of the time. The only safe strategy is to fit all plausible models and report sensitivity across them as a sensitivity analysis.

How It Works

With $J$ conditionally independent binary tests and a two-class latent variable $D$, the observed cell probability for pattern $(t_1,\ldots,t_J)$ is: $$P(T_1=t_1,\ldots,T_J=t_J) = \rho \prod_j \pi_{ej}^{t_j}(1-\pi_{ej})^{1-t_j} + (1-\rho)\prod_j (1-\pi_{cj})^{t_j}\pi_{cj}^{1-t_j}$$ Parameters $(\rho, \pi_{e1},\ldots,\pi_{eJ}, \pi_{c1},\ldots,\pi_{cJ})$ are estimated by MLE (via the Expectation-Maximization [EM] algorithm) or Markov Chain Monte Carlo (MCMC) with Dirichlet/Beta priors. The Bayesian approach propagates uncertainty in all parameters, including prevalence.

Why It Matters

Misestimated sensitivity by 0.2 has direct clinical consequences: a screening test declared 0.65 sensitive rather than 0.85 changes downstream treatment protocols, resource allocation, and disease burden estimates. The Albert-Dodd (2004) result implies that no-gold-standard latent class studies can produce widely varying conclusions that are statistically indistinguishable, which counsels caution when interpreting published sensitivity/specificity estimates from small rater panels.

Open Questions

• Bayesian model averaging over the four model classes could incorporate model uncertainty explicitly rather than using ad hoc sensitivity analysis.
• Semi-parametric frailty models allow the subject-level random effect distribution to be estimated nonparametrically (Dirichlet Process Mixture), potentially providing robustness to the GRE distributional assumption.
• Ordinal test outcomes (e.g., Likert-scale ratings) require a Receiver Operating Characteristic (ROC) generalization; the latent class framework extends to polytomous tests but identifiability conditions become stricter.

Related

• Latent Class Model
• Misspecification
• Albert-Dodd (2004)
• Paul S. Albert
• Lori E. Dodd