Differential Item Functioning & Explanatory IRT

Is a test item fair — and what makes items hard?

Two group questions about a test. Impact: do the groups differ in the trait itself? — a legitimate difference, modelled by a latent regression $\theta_i\sim N(\beta g_i,1)$. Differential item functioning (DIF): does an item behave differently across groups at the same trait level? — a bias. An item with uniform DIF is a group-specific shift in difficulty, $$P(x_{ij}=1\mid\theta_i)=\Phi(a_j\theta_i+d_j-\delta_j\,g_i),$$ where $\delta_j$ is how much harder item $j$ is for group $g=1$ at the same ability. Separating impact ($\beta$) from bias ($\delta_j$) is the point: a group can be genuinely higher on the trait and some items can still be biased.

From scratch it is the Albert–Chib augmentation once more: draw $z_{ij}$ truncated by the sign of $x_{ij}$, after which $\theta$, the item parameters, the DIF shifts $\delta_j$ and the regression coefficient $\beta$ are all conjugate Gaussian. The DIF parameters are identified by centring ($\sum_j\delta_j=0$, DIF relative to the average item); items whose $\delta_j$ credible interval excludes zero are flagged. We validate, run the classic verbal-aggression data (do items work the same for men and women?), then do explanatory IRT — regressing item difficulties on item features to explain what makes an item easy to endorse — and cross-check in PyMC.

import numpy as np, pandas as pd, matplotlib.pyplot as plt
from scipy.special import ndtr as Phi
import dif as D
rng = np.random.default_rng(3)
BLUE="#2b6cb0"; RED="#c53030"; GREEN="#2f855a"; ORANGE="#dd6b20"; PURP="#6b46c1"; GREY="#718096"
print("DIF asks whether an item is biased across groups at the same trait level; impact asks whether the trait differs.")

DIF asks whether an item is biased across groups at the same trait level; impact asks whether the trait differs.

1. What DIF looks like

Two items, one fair and one with DIF, plotted for two groups. The fair item has the same curve for both groups — at a given ability, everyone endorses it equally. The DIF item is shifted: at the same ability, one group endorses it more. That shift, net of any real trait difference, is item bias.

th=np.linspace(-4,4,200); a=1.3
fig,ax=plt.subplots(1,2,figsize=(12,4.2))
ax[0].plot(th, Phi(a*th+0.3), color=BLUE, lw=2.4, label="group 0"); ax[0].plot(th, Phi(a*th+0.3), color=RED, lw=1.4, ls="--", label="group 1")
ax[0].set_title("Fair item (no DIF): one curve for both groups"); ax[0].set_xlabel(r"ability $\theta$"); ax[0].set_ylabel("P(endorse)"); ax[0].legend(frameon=False,fontsize=8)
ax[1].plot(th, Phi(a*th+0.3), color=BLUE, lw=2.4, label="group 0"); ax[1].plot(th, Phi(a*th+0.3-1.0), color=RED, lw=2.4, ls="--", label="group 1 (item harder)")
ax[1].set_title("Uniform DIF: item is harder for group 1 at every ability"); ax[1].set_xlabel(r"ability $\theta$"); ax[1].set_ylabel("P(endorse)"); ax[1].legend(frameon=False,fontsize=8)
plt.tight_layout(); plt.show()
print("A DIF item's curves are separated at the same theta -- a bias the ordinary IRT model would blame on ability.")
print("The model adds a group shift delta_j per item to catch it, while a latent regression absorbs true trait differences.")

A DIF item's curves are separated at the same theta -- a bias the ordinary IRT model would blame on ability.
The model adds a group shift delta_j per item to catch it, while a latent regression absorbs true trait differences.

2. Does it work? — recovering DIF and impact

Simulated: 2000 people, 15 items, three of them with DIF (two harder for group 1, one easier), and a genuine group trait difference $\beta=0.5$. The sampler should recover the DIF shifts and separate them from the real group difference.

J=15; a_t=rng.uniform(0.8,1.8,J); d_t=rng.uniform(-1,1,J); dif_t=np.zeros(J); dif_t[[3,7]]=[1.0,0.9]; dif_t[12]=-0.8; dif_t-=dif_t.mean()
X,th,g=D.simulate_dif(2000,a_t,d_t,dif_t,0.5,rng)
r=D.dif_gibbs(X,g,rng,draws=2000,burn=1000); de=r["delta"]; dm=de.mean(0); lo,hi=np.percentile(de,[2.5,97.5],axis=0)
flag=(lo>0)|(hi<0)
print(f"DIF recovery correlation {np.corrcoef(dif_t,dm)[0,1]:.3f};  group trait difference beta: true 0.5, estimated {r['beta'].mean():.2f}")
fig,ax=plt.subplots(1,2,figsize=(12,4.2))
ax[0].scatter(dif_t,dm,color=BLUE); ax[0].plot([-1,1.2],[-1,1.2],"k--",lw=1); ax[0].set_xlabel("true DIF"); ax[0].set_ylabel("estimated DIF"); ax[0].set_title("DIF shifts recovered")
y=np.arange(J); ax[1].errorbar(dm,y,xerr=[dm-lo,hi-dm],fmt="o",color=GREY); ax[1].scatter(dm[flag],y[flag],color=RED,zorder=3,label="CI excludes 0")
ax[1].scatter(dm[[3,7,12]],y[[3,7,12]],facecolors="none",edgecolors=BLUE,s=120,label="true DIF item")
ax[1].axvline(0,color="k",lw=.8); ax[1].set_yticks(y); ax[1].set_yticklabels([f"item {j}" for j in range(J)],fontsize=7); ax[1].set_xlabel("DIF shift delta"); ax[1].set_title("DIF forest plot"); ax[1].legend(frameon=False,fontsize=8)
plt.tight_layout(); plt.show()
print("The three true DIF items carry the largest shifts and are flagged; beta recovers the real group difference,")
print("kept separate from the item bias. (Sum-to-zero identification gives DIF-free items tiny compensating shifts.)")

DIF recovery correlation 0.981;  group trait difference beta: true 0.5, estimated 0.50

The three true DIF items carry the largest shifts and are flagged; beta recovers the real group difference,
kept separate from the item bias. (Sum-to-zero identification gives DIF-free items tiny compensating shifts.)

3. Verbal aggression — do items work the same for men and women?

316 people report, for 24 situations, whether they would want to or do curse / scold / shout — the classic DIF benchmark. We take the focal group to be men ($g=1$) and ask which items show gender DIF once the overall aggression level is controlled for.

d=pd.read_csv("verbagg.csv"); items=list(d.columns[1:]); g=(d.Gender=="M").astype(float).to_numpy(); X=d.drop(columns="Gender").to_numpy().astype(float)
r=D.dif_gibbs(X,g,rng,draws=4000,burn=2000); de=r["delta"]; dm=de.mean(0); lo,hi=np.percentile(de,[2.5,97.5],axis=0); flag=(lo>0)|(hi<0)
beta=r["beta"]; bmean,blo,bhi=beta.mean(),np.percentile(beta,2.5),np.percentile(beta,97.5)
print(f"group trait difference (men - women aggression): beta = {bmean:+.2f}  95% CrI [{blo:+.2f}, {bhi:+.2f}]  ({'significant' if blo>0 or bhi<0 else 'not significant -- little overall difference'})")
print(f"{flag.sum()} of {len(items)} items flagged with gender DIF.")
order=np.argsort(dm)
fig,ax=plt.subplots(figsize=(8.5,7)); y=np.arange(len(items))
cols=np.where(flag[order], RED, GREY)
ax.errorbar(dm[order],y,xerr=[dm[order]-lo[order],hi[order]-dm[order]],fmt="o",ecolor=GREY,mfc="none",mec="none")
ax.scatter(dm[order],y,color=cols,zorder=3)
ax.axvline(0,color="k",lw=.8); ax.set_yticks(y); ax.set_yticklabels(np.array(items)[order],fontsize=7)
ax.set_xlabel("DIF shift delta  (>0: harder for men)"); ax.set_title(f"Gender DIF in verbal-aggression items (red = flagged, n={flag.sum()})")
plt.tight_layout(); plt.show()
# Report the FLAGGED items -- those whose credible interval excludes zero -- not simply the
# largest and smallest point estimates. Naming an item biased when the analysis has not flagged
# it is the one mistake a fairness analysis cannot afford.
it_arr = np.array(items)
hard_M = it_arr[flag & (dm > 0)]          # positive delta: harder for men, women over-endorse
easy_M = it_arr[flag & (dm < 0)]          # negative delta: easier for men, men over-endorse
print("Flagged HARDER for men (women over-endorse): " + (", ".join(hard_M) if len(hard_M) else "none"))
print("Flagged EASIER for men (men over-endorse):  " + (", ".join(easy_M) if len(easy_M) else "none"))
print("   (%d + %d = %d flagged, matching the count above.)" % (len(hard_M), len(easy_M), len(hard_M)+len(easy_M)))
r_flagged = ["S2WantShout", "S2DoCurse", "S2DoScold", "S3DoCurse"]      # difR, BH-adjusted (R notebook)
overlap = [t for t in r_flagged if t in set(hard_M) | set(easy_M)]
print("\nR's difR flags %d items by an entirely different route -- a logistic DIF test with a" % len(r_flagged))
print("Benjamini-Hochberg correction -- and %d of those %d appear above. Two methods that share no" % (len(overlap), len(r_flagged)))
print("machinery agreeing on which items misbehave is worth more than either flag count on its own.")
n_m = int(g.sum()); n_w = int(len(g) - g.sum())
print("\nRead all of this against the sample: %d respondents, %d men and %d women. That imbalance is" % (len(g), n_m, n_w))
print("why most intervals in the figure are wide enough to span zero -- the study has limited power to")
print("detect DIF, so absence of a flag is weak evidence of fairness, and the flagged set is the")
print("conservative end of what may be there.")
print("\nThe pattern in the flagged items is still legible: the one item women over-endorse is a WANT-to-")
print("shout item, while every item men over-endorse is a DO-curse or DO-scold item. The genders differ")
print("less in overall aggression -- beta is not distinguishable from zero -- than in HOW it is expressed,")
print("which is precisely the distinction a plain total score cannot make and DIF analysis is built for.")

group trait difference (men - women aggression): beta = +0.22  95% CrI [-0.05, +0.50]  (not significant -- little overall difference)
5 of 24 items flagged with gender DIF.

Flagged HARDER for men (women over-endorse): S2WantShout
Flagged EASIER for men (men over-endorse):  S2DoCurse, S2DoScold, S3DoCurse, S3DoScold
   (1 + 4 = 5 flagged, matching the count above.)

R's difR flags 4 items by an entirely different route -- a logistic DIF test with a
Benjamini-Hochberg correction -- and 4 of those 4 appear above. Two methods that share no
machinery agreeing on which items misbehave is worth more than either flag count on its own.

Read all of this against the sample: 316 respondents, 73 men and 243 women. That imbalance is
why most intervals in the figure are wide enough to span zero -- the study has limited power to
detect DIF, so absence of a flag is weak evidence of fairness, and the flagged set is the
conservative end of what may be there.

The pattern in the flagged items is still legible: the one item women over-endorse is a WANT-to-
shout item, while every item men over-endorse is a DO-curse or DO-scold item. The genders differ
less in overall aggression -- beta is not distinguishable from zero -- than in HOW it is expressed,
which is precisely the distinction a plain total score cannot make and DIF analysis is built for.

Reading a DIF shift, and what it cannot tell you

The model reports two quantities that are easy to conflate, and the whole point of the analysis is that they are different things.

Impact is $\beta$, and it moves people: the difference in the underlying trait between groups, in trait standard deviations. Here $\beta = +0.22$ with a credible interval of $[-0.05, +0.50]$, so men score slightly higher on latent verbal aggression, by about a fifth of a standard deviation, and the interval includes zero — this sample cannot establish an overall difference.

DIF is $\delta_j$, and it moves items: a shift in item easiness that applies at the same trait level. It is on the intercept's scale, so dividing by that item's discrimination, $\delta_j/a_j$, re-expresses it in trait units — the horizontal distance between the two groups' curves for that item. A positive shift here means the item is easier for men to endorse than their aggression level alone would predict.

Three limits are worth stating plainly, because DIF results are often over-read.

The shifts are identified only up to a constant: they are constrained to sum to zero across items, so each one is a departure from the average item, not an absolute measure of bias. There is no such thing as an unbiased benchmark item here, and a claim that "item 7 is biased" always means "relative to how the rest of this test behaves". If every item were biased the same way, this model would report no DIF at all.

Flagging is not ranking. An item is flagged when its interval excludes zero, which depends on the shift's precision as much as its size, so a large but uncertain shift can go unflagged while a smaller, tightly-estimated one is caught. A list ordered by magnitude and a list of flagged items need not agree, or even be the same length.

And DIF is not unfairness. It says an item behaves differently across groups at matched trait levels; whether that is a defect depends on why. An item that is genuinely harder for one group for reasons irrelevant to the construct is a fairness problem; an item that taps a real sub-facet the groups differ on may be measuring something the test intends to measure. The statistics locate the items; deciding what to do about them is a judgement about the construct.

4. Explanatory IRT — what makes an item easy to endorse?

The ordinary model treats each item's difficulty as a free parameter. Explanatory IRT (the linear logistic test model idea) instead predicts the difficulties from item features. Each verbal-aggression item is described by its behaviour type (curse/scold/shout), whether it is a want or a do, and whether the situation blames self or other. Regressing the estimated item easinesses on these features shows which properties make an act easy to endorse.

feat=pd.read_csv("verbagg_items.csv"); feat=feat.set_index("item").loc[items].reset_index()
d_hat=r["d"].mean(0)                                   # item easiness (higher = more readily endorsed)
Xf=pd.get_dummies(feat[["btype","situ","mode"]], drop_first=True).astype(float)
Xf.insert(0,"intercept",1.0); Xm=Xf.to_numpy()
b=np.linalg.lstsq(Xm, d_hat, rcond=None)[0]; r2=1-np.var(d_hat-Xm@b)/np.var(d_hat)
coef=pd.Series(b, index=Xf.columns)
print("regression of item easiness on item features (LLTM-style):"); print(coef.round(2).to_string())
print(f"\nR^2 = {r2:.2f}: item features explain most of the difficulty variation.")
fig,ax=plt.subplots(figsize=(8,4)); c=coef.drop("intercept"); ax.barh(range(len(c)), c.values, color=[GREEN if v>0 else RED for v in c.values])
ax.set_yticks(range(len(c))); ax.set_yticklabels(c.index); ax.axvline(0,color="k",lw=.8); ax.set_xlabel("effect on item easiness"); ax.set_title("What makes a verbal-aggression act easy to endorse")
plt.tight_layout(); plt.show()
print("Wanting is far easier to endorse than doing; cursing is easier than scolding, and shouting is hardest of all;")
print("blaming others is easier than blaming oneself. The difficulties are not arbitrary -- item features explain most of them, which")
print("is exactly what the linear logistic test model formalises: item difficulty as a regression on item properties.")

regression of item easiness on item features (LLTM-style):
intercept      0.55
btype_scold   -0.66
btype_shout   -1.15
situ_self     -0.64
mode_want      0.52

R^2 = 0.88: item features explain most of the difficulty variation.

Wanting is far easier to endorse than doing; cursing is easier than scolding, and shouting is hardest of all;
blaming others is easier than blaming oneself. The difficulties are not arbitrary -- item features explain most of them, which
is exactly what the linear logistic test model formalises: item difficulty as a regression on item properties.

Reading the item-feature regression

These coefficients are on the easiness scale, not difficulty, so a positive value means easier to endorse. They are contrasts against the baseline item — cursing, in an other-to-blame situation, in the do mode — so the intercept of 0.55 is that baseline item's easiness and each coefficient is a shift from it in the same latent units the difficulties live in.

Read that way the pattern is a substantive finding, not a fit statistic. Moving from doing to merely wanting makes an item 0.52 easier to endorse; from cursing to scolding, 0.66 harder; to shouting, 1.15 harder — shouting is the most reluctantly admitted behaviour on the test. Self-blaming situations are 0.64 harder than other-blaming ones. People will admit to wanting what they will not admit to doing, and to blaming others more readily than themselves.

The $R^2 = 0.88$ says three binary design features reproduce most of the variation in difficulty across 24 items. That is the claim the linear logistic test model makes: item difficulty is not an arbitrary property to be estimated one item at a time, but a function of how the item was built. Where it holds, difficulty becomes predictable in advance, which is what lets test developers write a new item to a target difficulty rather than pilot it and hope. The residual 12% is the part of an item's behaviour its design features do not explain — and on a test where that residual were large, the LLTM would be the wrong model however good its coefficients looked.

5. Cross-check in PyMC

The DIF-plus-impact model in PyMC: standard-normal abilities shifted by a group regression $\beta$, item discriminations and easinesses, and a per-item DIF shift constrained to sum to zero, through the probit link. We compare the DIF estimates and $\beta$ with the from-scratch sampler.

import pymc as pm, pytensor.tensor as pt
N,J=X.shape
with pm.Model() as mod:
    beta_=pm.Normal("beta",0,3)
    theta=pm.Normal("theta", mu=beta_*g, sigma=1, shape=N)
    a_=pm.HalfNormal("a",2,shape=J); d_=pm.Normal("d",0,3,shape=J)
    dlt=pm.Normal("dlt",0,1,shape=J); dlt=pm.Deterministic("delta", dlt-dlt.mean())   # sum-to-zero DIF
    eta=theta[:,None]*a_[None,:] + d_[None,:] - dlt[None,:]*g[:,None]
    pm.Bernoulli("x", p=pm.math.invprobit(eta), observed=X)
    idata=pm.sample(600, tune=1000, chains=4, target_accept=0.9, random_seed=5, progressbar=False)
de_pm=idata.posterior["delta"].mean(("chain","draw")).values; beta_pm=float(idata.posterior["beta"].mean())
print(f"group trait difference beta:  from-scratch {bmean:+.2f}   PyMC {beta_pm:+.2f}")
print(f"DIF agreement: correlation {np.corrcoef(dm,de_pm)[0,1]:.3f}")
fig,ax=plt.subplots(figsize=(6.6,4)); ax.scatter(dm,de_pm,color=BLUE)
for j in np.where(flag)[0]: ax.annotate(items[j].replace("Want","W").replace("Do","D"),(dm[j],de_pm[j]),fontsize=6)
ax.plot([-.8,.8],[-.8,.8],"k--",lw=1); ax.set_xlabel("from-scratch DIF"); ax.set_ylabel("PyMC DIF"); ax.set_title("DIF shifts: from-scratch vs PyMC")
plt.tight_layout(); plt.show()
print("Same DIF pattern, same near-zero overall gender difference: the augmentation Gibbs and PyMC's NUTS agree on")
print("which items are biased and on the impact-vs-bias split.")

g++ not available, if using conda: `conda install gxx`

Initializing NUTS using jitter+adapt_diag...

Multiprocess sampling (4 chains in 4 jobs)

NUTS: [beta, theta, a, d, dlt]

Sampling 4 chains for 1_000 tune and 600 draw iterations (4_000 + 2_400 draws total) took 11 seconds.

The rhat statistic is larger than 1.01 for some parameters. This indicates problems during sampling. See https://arxiv.org/abs/1903.08008 for details

The effective sample size per chain is smaller than 100 for some parameters.  A higher number is needed for reliable rhat and ess computation. See https://arxiv.org/abs/1903.08008 for details

group trait difference beta:  from-scratch +0.22   PyMC +0.23
DIF agreement: correlation 1.000

Same DIF pattern, same near-zero overall gender difference: the augmentation Gibbs and PyMC's NUTS agree on
which items are biased and on the impact-vs-bias split.

6. Summary

Group differences in a test come in two kinds, and IRT keeps them apart. Impact — a real difference in the trait — is a latent regression $\theta\sim N(\beta g,1)$; differential item functioning — item bias at the same trait level — is a per-item shift $\delta_j$. On the verbal-aggression data the overall gender difference in aggression was small, but several items showed clear DIF: women more readily endorse wanting to shout, men actually cursing or scolding — a difference in how aggression is expressed that the plain trait score would hide. Explanatory IRT then predicted the item difficulties from item features (want vs do, shout vs curse, self vs other), explaining most of the variation — the linear-logistic-test-model view of difficulty as a regression on item properties. A PyMC fit confirmed the DIF pattern and the impact estimate.

The connections: DIF is an interaction (group × item) laid over the 2PL; the latent regression and the difficulty-on-features model are regressions attached to the item response model — the same explanatory move as latent class regression (Latent Class Regression), and the augmentation is Albert (1992), shared across the probit models. Next, the optional capstone: IRT model comparison (1PL vs 2PL vs 3PL) reusing the WAIC/LOO machinery from the latent-class model-selection notebook.