← All examples

Item Response Theory

A test score is a summary; item response theory is a model of how that score came about. It is the core of modern psychometrics — the measurement branch of psychology and education, which deals with quantities nobody can observe directly (ability, attitude, symptom severity) and has to infer them from how people answer a set of questions. Its defining difficulty is that the questions are part of the instrument: a low score can mean a weak trait or a hard set of items, and the two are only separable with a model that carries both at once. Every person carries a latent trait, every item has a curve giving the probability of a given response as a function of that trait, and the two are estimated on one common scale — which is what lets a hard item and a high-ability examinee be compared at all. The arc builds outward from the simplest case: dichotomous items (2PL and 3PL, with Rasch folded in as the equal-discrimination special case), then ordered responses for Likert data, then multiple traits, where IRT turns out to be factor analysis for categorical items. From there it becomes diagnostic — separating impact (do the groups differ in the trait?) from bias (does an item behave differently at the same trait level?), which is the statistical machinery behind test fairness, and turning the question around to ask what makes an item hard in the first place. It closes by asking how to choose among the models it has built — and by getting the likelihood right first, since scoring an IRT model on the wrong unit reports 278 effective parameters for one that has six.

The from-scratch engine throughout is Albert–Chib data augmentation, which is fitting: the 1992 paper that introduced that trick introduced it for IRT, before it spread to probit, Tobit and multivariate probit models elsewhere in this collection. Working psychometrically also means confronting how much of a reported number is convention rather than result — the same item can carry two legitimate discrimination values depending on whether the model is written in the normal-ogive or logistic metric, a loading table can look unrelated to a discrimination table that it exactly equals, and a guessing parameter can be quoted to two decimals from a fit that never converged. Running two independent implementations of every model is what makes those distinctions visible rather than leaving them as apparent disagreements.

What an item characteristic curve is. Every model in this section is a variation on one object, so it is worth drawing in words first. Put ability on the horizontal axis and the probability of a correct or endorsing response on the vertical, and an item traces an S-shaped curve between them: low-ability people rarely get it, high-ability people usually do, and the interesting part is the transition. Three numbers describe that curve and each answers a different question. Difficulty is where the transition happens — the ability at which a person has a 50% chance — and because it is measured on the ability scale, an item and a person can be compared directly, which is the whole point. Discrimination is how steeply it happens: a steep item separates people either side of it sharply but says little about anyone far away, while a shallow item grades everybody coarsely. Guessing is where the curve bottoms out — on a five-option multiple-choice item it should be near 1 in 5, because even someone who knows nothing will sometimes tick the right box. The models below add these one at a time (1PL, 2PL, 3PL), then let the curve describe an ordered response rather than a right/wrong one, then give it a second ability axis, then ask whether it sits in the same place for two groups.

What a total score cannot do. Counting right answers works only as long as everyone answered the same questions. The moment two people face different items — different test forms, an adaptive test that chose questions as it went, a scale revised between waves — their totals are not comparable, because a low score might mean a weak trait or a hard set of items and the total cannot tell them apart. Estimating people and items on one scale is what dissolves that: an ability estimate is adjusted for the difficulty of the items actually taken, so two examinees with no questions in common can still be placed on the same axis. That is why item response theory underpins adaptive testing and test equating, and it is the practical payoff for all the machinery that follows. A related consequence runs the other way: because the scale is fixed by convention (ability is given mean 0 and variance 1 in the sample), an ability of +1 means one standard deviation above these respondents and nothing absolute. What is real is the ordering and the distances, not the zero point.

Where a test measures, and where it does not. A test does not have one accuracy; it has a different accuracy at every level of the trait. Information measures that — how sharply the responses pin down ability at a given point — and it is largest where an item’s curve is steepest, which is near its own difficulty. Its inverse square root is the standard error of the ability estimate, so more information means a tighter estimate, and a test measures precisely only in the range where its items are pitched. Two consequences show up repeatedly below. A test whose items are all easy, like the LSAT section used here, measures somewhat-below-average ability well and is blunt for the strongest candidates — the very people a selection test most needs to separate. And modelling guessing honestly lowers the measured precision rather than raising it, because a correct answer that might have been a lucky guess carries less evidence about ability than the 2PL assumes it does.

How the models get compared. More parameters always fit the data in hand better, so the closing question is out-of-sample: would the extra parameters improve prediction for someone new? WAIC and PSIS-LOO estimate that, reporting an elpd — roughly, the log-probability the model would assign to data it has not seen — minus a penalty for effective complexity. Only differences between models on the same data mean anything, and each difference comes with a standard error, so a gap of a couple of standard errors is not evidence. The subtlety that opens the comparison is what counts as one observation. Scoring response-by-response treats each person’s ability as a parameter and the penalty counts all of them — reporting 278 effective parameters for a model with six. Integrating ability out makes the person the unit, and the penalty falls to 6.5, which is what the model really costs.

A continuous trait, or a set of types? Everything here assumes the latent variable is continuous — ability, neuroticism and aggression are quantities you can have more or less of, and the models give each person a position on a smooth scale, conventionally standard normal. That assumption is doing real work and it is not the only option. Latent class analysis asks the same question of the same kind of data — several categorical responses, one unobserved cause behind them — but takes the latent variable to be discrete and unordered: a finite set of classes, with each respondent belonging to one and each class having its own response probabilities. The two are the same idea about measurement with different commitments about what is being measured. Ask how much and you want a trait model; ask which kind and you want classes. The choice is substantive rather than technical, and it is testable: a genuinely graded trait fitted with classes needs more and more of them to approximate the continuum, while genuinely distinct types fitted on one dimension show up as items that refuse to order consistently. Between the two sit the hybrids — ordered classes, or a continuous trait within each class — which is why the distinction is worth stating explicitly rather than letting the choice of software make it.