Liddell-Kruschke (2019) Analyzing Ordinal Data with Metric Models: What Could Possibly Go Wrong?

Summary

Liddell and Kruschke (2019) prove, analytically and by example, that applying metric models (t-tests, analysis of variance (ANOVA), regression) to ordinal data can produce all three types of inferential error: spurious effects (Type I), missed effects (Type II), and reversals of effect direction. The root cause is that the ordinal mean is a sigmoidal function of the latent continuous mean, so two groups sitting on different sigmoid curves — because they differ in latent variance — can have equal latent means but different ordinal means, or vice versa. The paper argues for Bayesian ordered-probit models as the principled alternative and provides worked JAGS code. An audit of three major psychology journals (the Journal of Personality and Social Psychology (JPSP), Psychological Science (PS), and Journal of Experimental Psychology: General (JEP:G); 2016 volumes) finds that 100% of articles using Likert data analysed it with metric models.

Key Claims

• Every article in JPSP, PS, and JEP:G (2016) that used Likert-scale data applied metric methods; N = 68 articles.
• False alarm (Type I): two groups with equal latent means $\mu$ but different latent standard deviations (SDs) can have different ordinal means $\bar{Y}$ — a metric model detects a spurious effect.
• Miss (Type II): two groups with different latent means can have equal ordinal means — a metric model misses the effect.
• Inversion: ordinal mean can show a positive group difference while the latent mean shows a negative one, and vice versa. Extends to factorial interactions (a genuine cross-over interaction can appear non-crossing) and polynomial trend contrasts.
• Root cause — sigmoidal mapping: $\bar{Y} = \sum_{j} j \cdot [\Phi(\theta_j - \mu/\sigma) - \Phi(\theta_{j-1} - \mu/\sigma)]$ is a sigmoid in $\mu$ with curvature controlled by $\sigma$. Two groups with different $\sigma$ trace different curves; the three failure modes are generic features of any two distinct curves.
• Averaging does not help: computing the mean of multiple correlated Likert items does not remove the sigmoidal distortion; all three failure modes persist after averaging.
• Equal metric SDs are irrelevant: having equal observed $\sigma$ across groups does not imply equal latent $\sigma$.
• Ordered-probit solution: models ordinal data via a latent continuous variable $z \sim N(\mu, \sigma^2)$ and $K-1$ thresholds $\theta_k$; $\Pr(Y=k) = \Phi((\theta_k - \mu)/\sigma) - \Phi((\theta_{k-1}-\mu)/\sigma)$. Estimated via Markov chain Monte Carlo (MCMC) (JAGS code provided; Stan also applicable).
• Real-data illustration: 36 Amazon drama-movie ratings (1–5 stars). Ordered-probit histogram fits are excellent; metric (normal) density fits are visually poor. Multiple inversions and false alarms demonstrated.
• HDI as credible interval: posterior highest density interval used throughout; Bayesian approach allows heterogeneous $\sigma$ across groups without additional complication.

Concepts Introduced or Extended

• Ordinal Regression — establishes failure modes of metric alternatives; motivates the ordered-probit model as the correct framework for Likert/rating scale data.
• Binary Probit — ordered probit generalizes the binary probit's latent-variable structure to $K > 2$ categories with multiple thresholds.

Entities Mentioned

• Torrin M. Liddell
• John K. Kruschke

Quotes

"We show that metric analysis of ordinal data can yield the full range of inferential errors: false alarms, misses, and inversions of direction." (Abstract)

"The ordinal mean is a sigmoidal function of the latent mean. Because of the sigmoidal relationship, groups that differ only in their latent standard deviations will have different ordinal means even though their latent means are the same." (p. 8)

"None of the 68 articles ... used an ordinal analysis of their Likert or ratings data. Instead, all of them used metric analyses." (p. 3)

My Take

The paper's core pedagogical contribution — the sigmoid-curve diagram (Figure 4) — is genuinely clarifying: it reduces the three failure modes to a single geometric fact about two intersecting S-curves. The audit of 68 articles is damning but not surprising given the dominance of ANOVA in psychology training. The ordered-probit solution is well-known in econometrics (Albert-Chib 1993b, 1997b, 2001) but the paper correctly notes it is largely unknown in psychology. The JAGS code is a practical contribution for psychologists unfamiliar with Bayesian computation. One limitation: the paper focuses on group-comparison designs; the extension to regression, mediation, and multi-item scales with correlated errors is mentioned but not fully worked out. The claim that averaging multiple Likert items doesn't help deserves more formal treatment — the argument relies on simulations rather than a theorem. The choice of the Bayesian framework, while principled, conflates two separable issues: (1) using an ordinal model vs. a metric model, and (2) Bayesian vs. frequentist inference; a frequentist ordered-probit (proportional-odds logistic regression) would solve problem (1) without requiring MCMC.