Agresti and Hitchcock (2005) Bayesian Inference for Categorical Data Analysis

Summary

A comprehensive survey of Bayesian methods for analyzing categorical data, tracing developments from Bayes (1763) and Laplace (1774) through the Markov chain Monte Carlo (MCMC) era. Organized by data structure — binomial and multinomial parameters, contingency table cell probabilities, loglinear models, and generalized linear models (GLMs) for binary and multi-category responses — the paper documents how conjugate beta-Dirichlet priors, Leonard's logit-normal hierarchy (1970s), and computationally tractable MCMC methods progressively unlocked fully Bayesian analysis for discrete data. A recurring theme is the shrinkage-toward-model character of Bayesian estimators and the practical difficulty of prior specification in non-conjugate settings.

Key Claims

Beta-Dirichlet conjugacy: For a binomial $y \sim \text{Bin}(n,\pi)$ with beta $(\alpha,\beta)$ prior, the posterior mean is $E(\pi|y) = (y+\alpha)/(n+\alpha+\beta)$ , a weighted average of the sample proportion $y/n$ and the prior mean $\alpha/(\alpha+\beta)$ with weight $w = n/(n+\alpha+\beta)$ . The Dirichlet prior generalizes this to the multinomial: $E(\pi_i|\mathbf{n}) = (n_i+\alpha_i)/(n+K)$ where $K=\sum\alpha_i$ is Good's (1965) "flattening constant" controlling shrinkage toward the equi-probability estimate.
Prior choice for the binomial: The Jeffreys prior beta $(0.5,0.5)$ is invariant under reparametrization; the uniform beta $(1,1)$ gives posterior mean $(y+1)/(n+2)$ (Laplace's rule of succession); the Haldane improper prior ( $\alpha=\beta=0$ ) recovers the maximum likelihood estimate (MLE) but yields an improper posterior when $y=0$ or $y=n$ . The logistic-normal (normal prior on $\text{logit}(\pi)$ , Leonard 1972) is more flexible than the beta family but non-conjugate and requires numerical integration or MCMC.
Altham (1969) result: Fisher's exact test one-sided P-value exactly equals a Bayesian P-value for independence in a $2\times2$ table when the Dirichlet prior hyperparameters are $\alpha_{11}=\alpha_{22}=0$ and $\alpha_{12}=\alpha_{21}=1$ — a prior distribution that favors $H_0$ . For a symmetric Dirichlet with $\alpha_{ij}=\gamma\in[0,1]$ , the Bayesian P-value is strictly smaller than Fisher's; the difference is bounded by the null probability of the observed data.
Leonard's logit-normal hierarchy (1972, 1975): Placing independent $N(\mu,\sigma^2)$ priors on the logits of $r$ binomial parameters (with second-stage improper priors on $\mu$ and $\log\sigma^2$ ) induces shrinkage of each logit toward the weighted mean of the others; the posterior mean is approximately a weighted average of $\text{logit}(p_i)$ and a pooled estimate. For loglinear models in two-way tables, normal priors on the interaction parameters $\{\lambda_{ij}^{XY}\}$ shrink fitted counts toward the independence fit $\{p_{i+}p_{+j}\}$ .
Laird (1978) empirical Bayes: In a loglinear model, the posterior mode of the interaction parameters is computed using an improper uniform prior for main effects and $N(0,\sigma^2)$ for interactions; $\sigma^2$ is then estimated by marginal likelihood. As $\sigma\to\infty$ , estimates converge to sample proportions; as $\sigma\to0$ , they converge to the independence model $\{p_{i+}p_{+j}\}$ .
Albert and Chib (1993) probit regression: Latent normal variables $Z_i$ (with $y_i=1$ iff $Z_i>0$ ) convert binary regression into a two-block Normal/truncated-Normal Gibbs sampler; a Lindley-Smith hierarchical prior $\beta\sim N(\mathbf{A}\beta_0,\sigma^2\mathbf{I})$ adds a second Gibbs block. The same data augmentation extends to ordered multinomial responses via cutpoints. This paper (Agresti-Hitchcock) cites it as the key computational advance for Bayesian binary GLMs.
MCMC enabling: Gelfand and Smith (1990) Gibbs sampling and Tierney (1994) Metropolis-Hastings algorithm enabled fully Bayesian analysis for non-conjugate categorical models. Zellner and Rossi (1984) importance sampling — using a multivariate $t$ importance function $I(\beta)$ and weights $w_i=f(\beta_i|\mathbf{y})/I(\beta_i)$ — was the dominant pre-MCMC approach.
Bayesian model averaging: Madigan and Raftery (1994) developed model averaging for loglinear model selection via Bayes factors, avoiding commitment to a single model; the Bayesian information criterion (BIC) approximates $-2\log$ Bayes factor for large samples (Raftery 1986).
Practical gap (§7): Despite an extensive literature, Bayesian inference is rarely used in practice for basic categorical analyses (tests of independence, confidence intervals for association parameters) due to software barriers, difficulty specifying priors on GLM parameters under non-linear link functions, and the plethora of hyperparameters in multinomial models.

Concepts Introduced or Extended

Entities Mentioned

Quotes

"The beta and logistic-normal priors sometimes do not provide sufficient flexibility." (p. 300)

"The ordinary P-value for Fisher's exact test corresponds to a Bayesian P-value with a conservative prior distribution, which some have taken to reflect the conservative nature of Fisher's exact test." (p. 314)

"Despite the advances summarized in this paper and the increasingly extensive literature, Bayesian inference does not seem to be commonly used yet in practice for basic categorical data analyses such as tests of independence and confidence intervals for association parameters." (p. 322)

My Take

A useful orientation to the categorical-data Bayesian literature, but tangential to this wiki's time-series/finance core. The most transferable ideas are: (1) the Altham-Fisher connection (P-values as Bayesian P-values under conservative priors) — a clean calibration result; (2) Leonard's logit-normal shrinkage as the conceptual precursor to the hierarchical probit work (Albert-Chib 1993b) already ingested; (3) the empirical Bayes / Laird (1978) approach as a historical foil to full Bayes. The claim in §7 about the practical gap remains true and is a useful reminder that having a rich methodology does not imply adoption. The survey predates Stan and brms; the software gap has since narrowed considerably.