Latent Class Analysis
The discrete-data mixture model: a population is treated as a blend of a few unobserved classes, and each subject's categorical responses are generated by whichever class they belong to. Its defining assumption is local independence — within a class the items are independent, so every association between items is attributed to the class structure itself. That single idea does a surprising amount of work: it underpins market segmentation and psychological typologies, and it is what makes it possible to evaluate diagnostic tests when no gold standard exists. The arc builds the model from scratch three ways — maximum likelihood by EM, a Bayesian data-augmentation Gibbs sampler in which the latent class is imputed and every conditional becomes conjugate, and PyMC, where the discrete class must instead be marginalised away because gradient samplers cannot touch it — checked against R reference packages where they exist (poLCA, BayesLCA, dirichletprocess, randomLCA) and against independent base-R implementations where they do not. From there it takes on what actually makes mixtures hard: label switching; how many classes there are — by information criteria, bootstrap likelihood-ratio tests and marginal likelihoods, or by refusing to choose at all and inferring the count from a Dirichlet process; letting covariates predict who lands in which class; testing whether local independence actually holds, and relaxing it with random effects when it does not; and finally setting the classes in motion, so each subject follows a hidden Markov chain across waves.
How many classes — and what the criteria are arguing about. Every criterion below scores the same quantity, fit minus a penalty for complexity, and they differ only in how hard they penalise. AIC charges 2 per parameter regardless of sample size, so on large samples it is the most permissive and the most prone to adding a class that buys little. BIC charges log n per parameter, which grows with the data and makes it markedly more conservative. SABIC softens that using an effective sample size, and CAIC hardens it further. WAIC and LOO come at it from a different direction entirely, estimating out-of-sample prediction rather than counting parameters. When they disagree it is worth knowing which way each leans; when they agree, as they do on the carcinoma data, the answer is about as settled as model selection gets.
Entropy, and why a well-fitting model can still be useless. Choosing the number of classes and being able to use them are different achievements. Entropy measures the second: it summarises how confidently subjects are assigned, running from 0 when membership probabilities are so diffuse that everyone is a blend of every class, to 1 when each subject belongs decisively to one. A model can fit well and still separate badly — the classes are then a real description of the population that cannot be turned into an assignment for any individual, which matters enormously if the point was to segment customers or triage patients. Values above about 0.8 are usually taken as good separation. It is a diagnostic, never a selection criterion: adding classes tends to lower it, so choosing by entropy would systematically under-count.
Measuring accuracy with nothing to measure against. The most surprising thing here is that a diagnostic test’s sensitivity can be estimated when no test is known to be right. The trick is arithmetic. Two tests on one population give three free cells in the table against five unknowns — a prevalence, two sensitivities, two specificities — so the model is under-identified and the posterior is a ridge rather than a peak, a direction the data simply cannot pin down. Adding a third test gives seven cells against seven unknowns. Or, keeping two tests, run them in a second population with a different prevalence: six cells against six unknowns, which is Hui and Walter’s device. Both are exactly identified and both collapse the ridge. The reason to state the counting explicitly is that an informative prior will produce a tidy-looking posterior for the under-identified design too — it just reports the analyst’s assumption back to them.
Why the discrete class is the hard part. The latent class is a categorical unknown, and that single fact shapes every implementation here. A Gibbs sampler loves it: impute each subject’s class, and with the labels in hand every remaining block is conjugate. Gradient samplers cannot touch it, because there is no derivative with respect to a category — so PyMC has to marginalise it away, summing the likelihood over all classes before handing a purely continuous problem to NUTS. When the classes are also allowed to move over time, that sum becomes a sum over every possible path through the waves, which is exponential if taken literally and linear if organised as the forward recursion — the same dynamic-programming pass that makes hidden Markov models tractable. It also explains label switching: nothing in the likelihood distinguishes “class 1” from “class 2”, so chains that disagree about the naming are equally correct, and the pooled posterior is bimodal until the draws are relabelled.
Choosing the classes, reading them, and the assumption underneath
All three panels are committed notebook output. A and B come from the model-selection project on Congdon’s carcinoma ratings; C from the conditional-dependence project’s simulation, where the truth is known.
A plots each criterion as its excess over its own minimum, which is the only way to see six differently-scaled numbers at once. They disagree about almost everything — how steeply to punish a fourth class, whether a second is nearly as good — and agree on the one thing that matters: all six bottom out at C = 3. That agreement is what licenses the answer, because on data this sparse any single criterion would be easy to doubt. The one-class model is left off the chart; it is worse by around 400.
B is what those three classes turn out to be, and it is the step that separates a class count from a finding. Read across the rows: 44% of slides that essentially every pathologist calls carcinoma, 37% that essentially none do, and an 18% remainder where the readers split — high for B, E and G, at or near zero for C, D and F. The third class is not a third kind of tumour. It is the set of slides that are genuinely hard to read, and the model found it without being told that such a thing exists.
C is the assumption everything above depends on. Local independence says that within a class the items are unrelated, and when it is false the model has nowhere to put the leftover association except into the items themselves. In this simulation tests 1–3 are correlated within class and tests 4–5 are not, and the naive fit over-states the accuracy of exactly the correlated ones — sensitivity 0.938 where the truth is 0.90, 0.889 where it is 0.85 — while leaving the independent pair roughly alone. Agreement between correlated tests gets credited to each test's skill. The random-effects correction is worth reporting honestly: it moves the estimates back down, but on this run it overshoots, landing at 0.862 and 0.811 below the truth rather than on it, and its mean absolute error (0.0316) barely improves on the naive fit's (0.0320). It relocates the bias rather than removing it — which is a more useful thing to know than a claim that the fix works.
How the seven examples relate
One model, three ways to fit it, then five things that make mixtures hard. Read the trunk first; each branch relaxes a single assumption the trunk had to make.
The model
Classes are unobserved; within a class the items are independent.
1 · FoundationsEM, data-augmentation Gibbs and marginalised NUTS — and label switchingHow many classes?
The one question a mixture cannot answer from inside itself.
2 · Model selectionsix criteria, a bootstrap LRT and marginal likelihoods — all pointing at three 3 · Dirichlet-process LCArefuse to choose: infer the count in a single fitRelaxing the assumptions
Membership can depend on who you are; items need not be independent; classes need not stand still.
4 · Covariate membershipa multinomial logit on class membership — identified only as contrasts 5 · Conditional dependencetest local independence, then relax it with random effects 6 · Latent transitionsclasses move between waves — a hidden Markov chain per subjectWhere it pays off
Accuracy without a gold standard.
7 · Diagnostic testingHui–Walter: sensitivity and specificity with nothing known to be rightPart 5 and Part 7 are worth reading together. The diagnostic model assumes local independence between tests; the conditional-dependence project shows what that assumption costs when tests share a latent difficulty — and on the carcinoma readers it finds mean loadings of 0.82, which is the same dependence that surfaced in Part 2 as a third class.
Latent Class Analysis — Foundations
The core model, built three times over and made to agree. EM with random
restarts (the likelihood is genuinely multimodal), a from-scratch data-augmentation
Gibbs sampler in which treating the latent class as a variable to impute makes every
conditional conjugate — Dirichlet for the prevalences, Beta for the item profiles — and
PyMC, where NUTS cannot sample a discrete label at all, so the class is
marginalised analytically through a log-sum-exp over components. R's poLCA and
BayesLCA supply independent reference fits. Applied to the classic
Stouffer–Toby role-conflict data (216 respondents, 4 binary items), all five
engines land on the same two-class solution: a large type leaning universalistic and a
smaller one answering particularistically on every item, with EM and poLCA
agreeing to three decimals across the whole parameter matrix. Label switching
is confronted rather than hidden — individual chains each stay put, but different chains
disagree about which class is "class 1", so the pooled posterior is visibly bimodal until the
draws are relabelled. A closing section separates two questions that are easy to conflate:
BIC selects two classes, G² confirms the model reproduces all sixteen response-pattern
frequencies, and entropy R² — which measures class separation rather than fit —
actually falls when a third class is added.
Choosing the Number of Classes
The crux of any latent class analysis, and the hard part: adding a class always improves the likelihood, the surface is multimodal, and with J items generating 2J response patterns the classical full-table G² test collapses into a sea of empty cells. The project assembles the whole practical toolkit — AIC, BIC, SABIC, CAIC, WAIC and LOO, the bootstrap likelihood-ratio test (needed because the χ² null is invalid when the smaller model sits on the parameter boundary), Chib's marginal likelihood with Bayes factors, and bivariate residuals for when the table is too sparse to fit any other way. Its organising principle is validation before application: everything is first run on simulated data with three known classes, where BIC, SABIC, WAIC, LOO and the BLRT all recover the truth and AIC over-extracts to four — a caution then carried into the real analysis. Applied to the carcinoma ratings (118 slides, 7 pathologists), every criterion converges on three slide types, and the third is the finding: alongside clear carcinoma and clear benign sits an ambiguous class the raters genuinely split on, which a two-class model would erase. Chib's estimator is itself validated against the exact one-class marginal likelihood, and plotting it against −BIC/2 shows concretely that BIC is the large-N approximation to the log evidence.
View example →Nonparametric Latent Class Analysis
The direct answer to the model-selection problem: rather than choosing the number of classes, put a prior over infinitely many and let K be a posterior quantity. A Dirichlet-process latent class model takes the finite mixture's Dirichlet prior to its C → ∞ limit, giving stick-breaking weights and a Chinese Restaurant Process for how subjects cluster. Because the base measure is conjugate the class profiles integrate out, so the partition is sampled directly (Neal's Algorithm 3) and everything is summarised label-invariantly through the posterior of K and a co-clustering matrix. On the carcinoma ratings it puts P(K=3) ≈ 0.84 in a single fit with no information criteria — the same answer the previous project assembled from six criteria and a bootstrap test — with the concentration α learned from the data rather than fixed. The most valuable section is where it breaks: holding the truth at three classes and growing N through 200, 800 and 3200, the posterior mass on K > 3 increases. That is the Miller–Harrison inconsistency, and it is why the DP is the right tool for flexible clustering but the wrong one when K is meant to be a small, interpretable count of types.
View example →Latent Class Regression
Every model so far gives each subject the same prior class weights. This one lets covariates predict membership, replacing the fixed weights with a multinomial logit — the concomitant-variable model. The measurement half is untouched, so the classes are still defined by response patterns; what changes is that the mix now bends with who the respondent is, and zeroing the coefficients recovers ordinary LCA exactly. From scratch it costs precisely one new ingredient: the Gibbs sampler keeps its conjugate Dirichlet and Categorical blocks, but the membership coefficients are a multinomial-logit regression on the imputed labels and need a random-walk Metropolis step — the single non-conjugate piece. Applied to the 2000 American National Election Study (1,294 respondents rating Gore and Bush on six traits each), three vote-classes emerge — pro-Gore, pro-Bush, and lukewarm-on-both — and party identification re-weights them sharply enough that the membership curves cross between strong Democrat and strong Republican. A fixed-weight LCA reports one average mix for everyone and misses it. The notebooks are also careful about what is identified: multinomial-logit coefficients only mean something as contrasts, so the reported figure is +1.38 log-odds per party point for pro-Bush against pro-Gore — a number the from-scratch Gibbs, PyMC and the R engines all agree on.
View example →Diagnostic Testing Without a Gold Standard
The Hui–Walter model, and the arc's most direct payoff: how accurate is a medical test when nobody can observe who is truly diseased? Sensitivity and specificity are defined by conditioning on a truth you never see — so treat that truth as a latent class, apply several imperfect tests, and estimate prevalence and every test's accuracy jointly. It is a two-class LCA rewritten in the language of epidemiology. The core lesson is identifiability, demonstrated rather than asserted: two tests in one population give 3 free cells against 5 parameters, and the notebook shows what that means by plotting the joint posterior as a ridge the sampler wanders along. A third test (7 vs 7) or a second population with a different prevalence (6 vs 6) — Hui and Walter's original device — collapses the ridge onto the truth. Where neither is available, informative priors are shown to make the design estimable but not identified: the posterior tightens and stops drifting, yet its location comes from the prior, not the data. Applied to the carcinoma ratings, it audits seven pathologists with no ground truth at all, separating accurate readers from aggressive and conservative ones in ROC space — and closes by naming its own weak point, since the three-class structure found earlier in the arc is precisely the signature of the conditional dependence this model assumes away.
View example →Relaxing Local Independence
The arc's closing project, and the one that audits everything before it. Every earlier model
assumed local independence — that items are independent given the
class. It is the load-bearing assumption of LCA and it is frequently wrong: two assays
reacting to the same antibody, or two pathologists trained alike, stay correlated within a
class. The random-effects (probit factor) model fixes it by giving each
subject a continuous latent severity that nudges all their responses together, with a
loading per item saying how strongly it responds — zero loadings recover
ordinary LCA exactly. Fitted from scratch by Albert–Chib data augmentation,
it is a latent class and a latent trait at once. On simulated data the naive model
over-states precisely the correlated tests, and the loadings correctly flag which ones they
are. On the carcinoma readers every loading is clearly positive, randomLCA's BIC
drops from 706.1 to 668.9, and the corrected sensitivities come down — which finally explains
the puzzling third class found earlier in the arc as conditional dependence rather than a
real type. The notebooks are also candid about the method's limits: the factor-mixture
posterior is multimodal, so the loadings are the dependable output while point
estimates can land in a collapsed mode.
Latent Transition Analysis
Latent class analysis with a clock. Every model before this one is a snapshot — a subject sits in one class. Here the classes move: each subject occupies a class at every wave and evolves as a Markov chain, making the model a per-subject hidden Markov model with parameters shared across the sample. The measurement model is held invariant over time, so that a "class" means the same thing at every wave — without that, movement between classes and drift in their meaning would be confounded. What that buys is the transition matrix, the object a snapshot analysis simply cannot produce: who stays, who escalates, who recovers. The sampler's new ingredient is forward-filter backward-sample, drawing each subject's whole trajectory jointly rather than wave by wave, after which everything is conjugate Dirichlet again — and it is the same latent-state sampler that drives the Markov-switching models, where the hidden state switches a variance regime instead of a response profile. On five waves of National Youth Survey marijuana data (237 adolescents) the dynamics come out crisp: non-use is stable but leaks about one in six per year, occasional use is the volatile middle with roughly a quarter escalating annually, and frequent use is the stickiest state at 88%. A PyMC forward-marginalised HMM reproduces the whole matrix to 0.003.
View example →