Diagnostic Accuracy

Definition

Diagnostic accuracy refers to the ability of a binary or ordinal test to correctly classify subjects as diseased or non-diseased. The canonical measures are sensitivity (probability of a positive test given disease) and specificity (probability of a negative test given no disease). When a gold-standard reference classification is available, these are estimable from a contingency table. When no gold standard exists, latent class models treat true disease status as a latent variable and estimate sensitivity and specificity jointly from the marginal agreement pattern across $J \geq 3$ fallible tests.

Key Ideas

• Sensitivity and specificity: For a binary test $T$ and true status $D \in \{0,1\}$:
  • Sensitivity: $\pi_e = P(T=1 \mid D=1)$
  • Specificity: $\pi_c = P(T=0 \mid D=0)$
  • These are properties of the test, not the population, but prevalence $\rho = P(D=1)$ must also be estimated.
• Youden's $J$ index (Youden 1950): the classic single-number summary $J = \text{sensitivity} + \text{specificity} - 1$, ranging from 0 (worthless — equal positive rate in both groups) to 1 (no false positives or negatives). It is the vertical distance from an ROC operating point to the chance diagonal, so the threshold maximizing $J$ (the "Youden index point") is a standard cutpoint-selection rule. Key virtue: prevalence-invariance (unlike accuracy or predictive values, $J$ does not move with the diseased/control mix); key assumption: the simple average weights false positives and false negatives equally — a value judgment Youden explicitly flags as non-statistical.
• Hui-Walter two-population identification (Vacek 1985 on Hui-Walter 1980): applying two imperfect tests to two populations of differing prevalence makes both tests' error rates and both prevalences identifiable by ML — the classic no-gold-standard design — but only under conditional independence.
• Conditional-dependence bias (Vacek 1985): when the two tests' errors are correlated given true status (shared failure modes), fitting the independence model underestimates both tests' error rates — i.e. overstates sensitivity and specificity — and biases prevalence estimates up or down depending on the two conditional covariances. The bias is optimistic, the worst direction for screening; this is the concrete reason the dependence-structure models below (BB, GRE, FM) exist.
• No-gold-standard problem: In many diagnostic studies (e.g., radiological assessment, psychiatric diagnosis), no criterion classifier exists. Four latent class models are commonly used:
  1. Conditional Independence (CI): $J$ tests independent given latent $D$; identifiable at $J \geq 3$.
  2. Beta-Binomial (BB): Test responses exchangeable given $D$; identifiable at $J \geq 3$.
  3. Finite Mixture (FM): Two components within each disease class; identifiable at $J \geq 3$.
  4. Gaussian Random Effects (GRE): Continuous frailty for each subject; identifiable at $J \geq 4$.
• Pseudo-true parameter problem (Albert-Dodd 2004): All four models can fit observed data equally well (identical log-likelihoods to 5 decimal places) yet produce sensitivity estimates differing by 0.2 or more. Maximum likelihood estimation (MLE) converges to the wrong pseudo-true parameters when the model is misspecified. Reliable model identification requires $J \geq 10$ raters — unrealistic in practice.
• Practical implication: With $J = 5$ tests, there is no statistical basis for preferring one model over another. The FM model has higher log-likelihood than the true GRE model 74% of the time. The only safe strategy is to fit all plausible models and report sensitivity across them as a sensitivity analysis.

How It Works

With $J$ conditionally independent binary tests and a two-class latent variable $D$, the observed cell probability for pattern $(t_1,\ldots,t_J)$ is: $$P(T_1=t_1,\ldots,T_J=t_J) = \rho \prod_j \pi_{ej}^{t_j}(1-\pi_{ej})^{1-t_j} + (1-\rho)\prod_j (1-\pi_{cj})^{t_j}\pi_{cj}^{1-t_j}$$ Parameters $(\rho, \pi_{e1},\ldots,\pi_{eJ}, \pi_{c1},\ldots,\pi_{cJ})$ are estimated by MLE (via the Expectation-Maximization [EM] algorithm) or Markov Chain Monte Carlo (MCMC) with Dirichlet/Beta priors. The Bayesian approach propagates uncertainty in all parameters, including prevalence.

Why It Matters

Misestimated sensitivity by 0.2 has direct clinical consequences: a screening test declared 0.65 sensitive rather than 0.85 changes downstream treatment protocols, resource allocation, and disease burden estimates. The Albert-Dodd (2004) result implies that no-gold-standard latent class studies can produce widely varying conclusions that are statistically indistinguishable, which counsels caution when interpreting published sensitivity/specificity estimates from small rater panels.

Open Questions

• Bayesian model averaging over the four model classes could incorporate model uncertainty explicitly rather than using ad hoc sensitivity analysis.
• Semi-parametric frailty models allow the subject-level random effect distribution to be estimated nonparametrically (Dirichlet Process Mixture), potentially providing robustness to the GRE distributional assumption.
• Ordinal test outcomes (e.g., Likert-scale ratings) require a Receiver Operating Characteristic (ROC) generalization; the latent class framework extends to polytomous tests but identifiability conditions become stricter.

Related

• Latent Class Model
• Misspecification
• Albert-Dodd (2004)
• Paul S. Albert
• Lori E. Dodd
• Youden (1950) Index for Rating Diagnostic Tests
• W. J. Youden
• Vacek (1985) The Effect of Conditional Dependence on the Evaluation of Diagnostic Tests
• Pamela M. Vacek
• Beath (2017) randomLCA: An R Package for Latent Class with Random Effects Analysis
• Ken J. Beath
• Probability Calibration