George-Makov-Smith (1993) Conjugate Likelihood Distributions

Summary

George, Makov, and Smith (1993) identify and characterize the obstacle to full Gibbs sampling in conjugate exponential-family hierarchical models: the hyperparameter full conditional is proportional to a conjugate likelihood distribution — the likelihood of the hyperparameters treating the group-level parameters as i.i.d. draws from the conjugate prior. They prove this distribution is log-concave (enabling Gilks-Wild adaptive rejection sampling) and jointly proper whenever the number of groups $p \geq 2$, provide explicit formulas for the Gamma-Poisson and Beta-Binomial cases, and illustrate the complete Gibbs cycle on the classic pump-failure dataset.

Key Claims

• In an exponential-family hierarchical model with conjugate priors, the hyperparameter full conditional takes the form $\pi(\alpha,\beta|\theta_1,\ldots,\theta_p) \propto L(\alpha,\beta|\theta_1,\ldots,\theta_p)\cdot\pi(\alpha,\beta)$ where $L$ is the conjugate likelihood distribution — the likelihood of $(\alpha,\beta)$ from viewing $\theta_1,\ldots,\theta_p$ as i.i.d. draws from the conjugate prior (eqs. 5–8).
• Theorem 3.3: The conjugate likelihood $L(x_0,n_0|\theta_1,\ldots,\theta_p) \propto \exp\!\left[x_0\sum_i\theta_i - n_0\sum_i\psi(\theta_i) - p\varphi(x_0,n_0)\right]$ is log-concave in $(x_0,n_0)$ (i.e., in $(\alpha,\beta)$); proper marginally in each argument for all $p\geq 1$; and proper jointly in $(\alpha,\beta)$ iff $p\geq 2$.
• Log-concavity enables Gilks-Wild (1992) adaptive rejection sampling (ARS) — an exact, tuning-free algorithm requiring only evaluation of the log-density and its derivative — for hyperparameter blocks that lack closed-form conditionals.
• Corollary 3.5 (Gamma case): For $\theta_i|\alpha,\beta\sim\text{Gamma}(\alpha,\beta)$: $L^G(\alpha,\beta|\theta_1,\ldots,\theta_p) \propto \left(\frac{\beta^\alpha}{\Gamma(\alpha)}\right)^p\!\left(\prod_i\theta_i\right)^{\alpha-1}\exp\!\left(-\beta\sum_i\theta_i\right)$. The $\alpha$ marginal conditional is log-concave (→ ARS); the $\beta$ conditional given $\alpha$ is Gamma (→ standard conjugate draw).
• Corollary 3.6 (Beta case): Analogous results for the Beta-Binomial hierarchy. When $p=1$, each marginal conditional in $\alpha$ and $\beta$ is individually proper, but the joint is improper — a cautionary single-group edge case.
• Pump failure application (§4): $p=10$ industrial pumps; $x_i|\theta_i\sim\text{Poisson}(\theta_i t_i)$; $\theta_i|\alpha,\beta\sim\text{Gamma}(\alpha,\beta)$. Complete Gibbs cycle: (41) $\theta_i|\cdot\sim\text{Gamma}(\alpha+x_i,\beta+t_i)$ — closed-form; (44) $\alpha|\cdot$ — sampled by ARS from $L^G$; (45) $\beta|\cdot\sim\text{Gamma}$ — closed-form. 100,000 draws under four hyperprior specifications (Table 2).
• Bayes estimates shrink pump-specific maximum-likelihood estimates (MLEs) toward a common center; the degree of shrinkage is calibrated by the data through $\alpha$ and $\beta$. All four prior specifications give qualitatively similar results.
• The conjugate likelihood representation is universal: any exponential-family hierarchy with a natural conjugate prior admits this decomposition, making the paper's results directly applicable across the entire class.

Concepts Introduced or Extended

• Bayesian Hierarchical Model
• Gibbs Sampler

Entities Mentioned

• Edward I. George
• Udi E. Makov
• A.F.M. Smith

Quotes

"The key to the Gibbs sampling approach to this problem lies in the availability of conjugate prior distributions."

"In the case where [the conjugate likelihood distribution] is log-concave, the Gilks and Wild (1992) method of adaptive rejection sampling can be used to sample from these distributions."

My Take

A technically tight paper that converts a computational bottleneck — the non-standard hyperparameter conditional — into a principled, general solution. The log-concavity result is the key: it makes exact ARS possible without tuning, which was a significant advance over Griddy-Gibbs for this class of models. The $p\geq 2$ properness condition is a useful reminder that single-group hierarchies need a proper joint hyperprior. The lasting contribution is the vocabulary ("conjugate likelihood distribution") and explicit formulas that let practitioners immediately identify when their hyperparameter sampler reduces to ARS + one conjugate draw — a complete, exact Gibbs cycle with no grid approximation or Metropolis step.