Bayesian Hierarchical Model

bayesianhierarchical-modelmcmccredibilityactuarialgeneralized-additive-modelspatial-econometricsenvironmental-epidemiologypoisson-regressionmeta-analysisr-squaredshrinkageprobitecological-fallacypublic-opinionconjugate-priorexponential-familyconsumer-heterogeneitydiscrete-choicepanel-dataconjoint-analysismarketinglongitudinal-datablockingrandom-effectsmixture-of-normalslatent-variabledecision-theory

Definition

A Bayesian hierarchical model organizes uncertainty into two or more levels: observable data governed by parameters, and those parameters governed by hyperparameters with their own prior distribution. Each level borrows information from the others through the joint posterior, enabling partial pooling — estimates for sparse groups are shrunk toward a common mean while rich groups remain data-driven.

Key Ideas

How It Works

Three-Level Structure

Let YijY_{ij} be the jjth observation in group ii:

Level 1 (data model):Yij∣θi∼f(y∣θi)\text{Level 1 (data model):} \quad Y_{ij} \mid \theta_i \sim f(y \mid \theta_i) Level 2 (latent parameters):θi∣μ,τ∼g(θ∣μ,τ)\text{Level 2 (latent parameters):} \quad \theta_i \mid \mu, \tau \sim g(\theta \mid \mu, \tau) Level 3 (hyperprior):μ,τ∼h(μ,τ)\text{Level 3 (hyperprior):} \quad \mu, \tau \sim h(\mu, \tau)

Bayes' theorem ties the levels together:

p(θ1,…,θn,μ,τ∣Y)∝h(μ,τ)∏ig(θi∣μ,τ)∏i,jf(yij∣θi)p(\theta_1,\ldots,\theta_n, \mu, \tau \mid Y) \propto h(\mu,\tau) \prod_i g(\theta_i|\mu,\tau) \prod_{i,j} f(y_{ij}|\theta_i)

Three-Component Marketing Framework and Heterogeneity Distributions (Rossi-Allenby 2003)

Rossi and Allenby articulate that every Bayesian marketing model has three coherent components: (1) a within-unit likelihood p(yi∣θi)p(y_i|\theta_i) modelling individual consumer behavior, (2) a cross-unit heterogeneity distribution p(θi∣τ)p(\theta_i|\tau), and (3) an action rule derived from a loss function applied to the posterior.

Normal hierarchical model. The canonical three-level specification (where WW denotes the Wishart distribution):

θi∼N(θˉ,Vθ),θˉ∼N(θˉˉ,A−1),Vθ−1∼W(ν,V)\theta_i \sim N(\bar\theta, V_\theta), \quad \bar\theta \sim N(\bar{\bar\theta}, A^{-1}), \quad V_\theta^{-1} \sim W(\nu, V)

Two-block Gibbs: draw {θi∣τ,yi}\{\theta_i|\tau, y_i\} (conjugate given the likelihood) and τ∣{θi}\tau|\{\theta_i\} (conjugate from the Wishart-Normal hyperprior). Shrinkage toward θˉ\bar\theta is strong when within-unit data are sparse and between-unit variation is small.

Observable heterogeneity. When unit-level covariates ziz_i are available:

θi=Bzi+ui,ui∼N(0,Vθ)\theta_i = Bz_i + u_i, \quad u_i \sim N(0, V_\theta)

BB and VθV_\theta are estimated jointly; residual heterogeneity uiu_i is the unexplained part after covariate adjustment.

Mixture-of-normals heterogeneity. To accommodate multimodal or heavy-tailed heterogeneity:

p(θi∣τ)=∑k=1Krk ϕ(θi∣θˉk,Vk)p(\theta_i|\tau) = \sum_{k=1}^K r_k\, \phi(\theta_i|\bar\theta_k, V_k)

A normal mixture avoids the convex-hull limitation of finite-mixture models (in which predictions are constrained to the convex hull of the component means). Component assignments are auxiliary variables in the Gibbs sampler; the number of components KK is selected by marginal likelihood, not BIC (Bayesian Information Criterion, which is "extremely inaccurate" for correlated-parameter models).

Structural heterogeneity. When individuals switch between behavioral regimes within the panel:

p(yit∣{θik})=∑k=1Krk pk(yit∣θik)p(y_{it}|\{\theta_{ik}\}) = \sum_{k=1}^K r_k\, p_k(y_{it}|\theta_{ik})

The mixture operates at the observation level, not the parameter level — each θik\theta_{ik} is a complete behavioral type, and the assignment probability rkr_k governs regime prevalence.

Diagnostic check (Allenby-Rossi 1999). Compare unit posterior means θ^i\hat\theta_i to the marginal predictive:

p(θi′∣data)=∬ϕ(θ∣θˉ,Vθ) p(θˉ,Vθ∣data) dθˉ dVθp(\theta_{i'}|\text{data}) = \iint \phi(\theta|\bar\theta, V_\theta)\, p(\bar\theta, V_\theta|\text{data})\, d\bar\theta\, dV_\theta

Unit posterior means that are outliers relative to this predictive distribution signal that the heterogeneity model is too restrictive. Properly accounting for hyperparameter uncertainty (integrating over θˉ\bar\theta and VθV_\theta) widens the predictive relative to a plug-in check.

Bayesian decision theory. The optimal action is:

a∗=arg⁡min⁡a∫l(a,θ) p(θ∣data) dθa^* = \arg\min_a \int l(a, \theta)\, p(\theta|\text{data})\, d\theta

Evaluating profit only at the posterior mean (plug-in) overstates expected profit because it ignores parameter uncertainty — the loss function is typically nonlinear in θ\theta. Full posterior integration is required for valid targeting decisions. The value of disaggregate (unit-level) information is quantified by Πdisagg−Πagg\Pi_{\text{disagg}} - \Pi_{\text{agg}} where Πdisagg\Pi_{\text{disagg}} uses individual posterior θi\theta_i and Πagg\Pi_{\text{agg}} uses a pooled estimate.

Workers Compensation Credibility (Scollnik ~1996, Example 4)

133 occupation classes, 7 years of claim frequency data; Klugman (1992) Data Set 2. Let YijY_{ij} = claim count, PijP_{ij} = scaled exposure, xij=Yij/Pijx_{ij} = Y_{ij}/P_{ij} = relative frequency.

HNM (Hierarchical Normal Model):

xij∣θi,τ1∼N(θi, Pij/τ1),θi∣μ,τ2∼N(μ, 1/τ2)x_{ij} \mid \theta_i, \tau_1 \sim \mathcal{N}(\theta_i,\, P_{ij}/\tau_1), \quad \theta_i \mid \mu, \tau_2 \sim \mathcal{N}(\mu,\, 1/\tau_2)

Diffuse proper priors: μ∼N(0,0.001)\mu \sim \mathcal{N}(0, 0.001); τ1,τ2∼Gamma(0.001,0.001)\tau_1, \tau_2 \sim \text{Gamma}(0.001, 0.001). Full conditionals are Normal/Gamma — closed-form Gibbs. Drawback: allows negative frequencies.

HFLPM (Hierarchical First-Level Poisson Model):

Yij∣λi∼Poisson(λiPij),λi∣μ,τ∼N(μ,1/τ)Y_{ij} \mid \lambda_i \sim \text{Poisson}(\lambda_i P_{ij}), \quad \lambda_i \mid \mu, \tau \sim \mathcal{N}(\mu, 1/\tau)

Non-negative counts by construction. Best predictive performance on the hold-out year: overall measure of prediction success (OMPS) posterior mean = 12.96 (sd = 0.64).

VSHNM (Variance-Stabilized HNM):

Apply zij=2xijz_{ij} = 2\sqrt{x_{ij}} before fitting a Normal hierarchy. Stabilizes the variance of Poisson-like data. OMPS = 13.72 — better than HNM but worse than HFLPM.

Gibbs Sampling for Hierarchical Models

The block structure of the posterior corresponds naturally to Gibbs blocks. For the HNM:

  1. θi∣μ,τ1,τ2,Y∼N\theta_i \mid \mu, \tau_1, \tau_2, Y \sim \mathcal{N} — weighted average of group data and prior mean
  2. μ∣θ1:n,τ2∼N\mu \mid \theta_{1:n}, \tau_2 \sim \mathcal{N} — grand mean update
  3. τ1∣θ1:n,Y∼Gamma\tau_1 \mid \theta_{1:n}, Y \sim \text{Gamma} — within-group precision
  4. τ2∣θ1:n,μ∼Gamma\tau_2 \mid \theta_{1:n}, \mu \sim \text{Gamma} — between-group precision

All four blocks have closed-form conjugate conditionals. BUGS (Bayesian Inference Using Gibbs Sampling) implements this automatically from the model declaration.

Gamma-Poisson Model for Geographic Mortality Data (Tsutakawa 1988)

For count data yijy_{ij} (deaths in region ii, demographic group jj) with exposure nijn_{ij}, the model adds two random-effect layers above the Poisson observation:

yij∣pij∼Poisson(nijpij)y_{ij} \mid p_{ij} \sim \text{Poisson}(n_{ij} p_{ij}) pij∣θj,σj2,zi∼Gamma(σj−2,  σj2θjzi)p_{ij} \mid \theta_j, \sigma_j^2, z_i \sim \text{Gamma}(\sigma_j^{-2},\; \sigma_j^2 \theta_j z_i) zi∼Inv-Gamma-1(γ),E(zi)=1z_i \sim \text{Inv-Gamma-1}(\gamma), \quad E(z_i) = 1

Marginalising over pijp_{ij} gives a negative binomial for yijy_{ij} with variance =nijθjzi+σj2(nijθjzi)2= n_{ij}\theta_j z_i + \sigma_j^2 (n_{ij}\theta_j z_i)^2 — Poisson plus extra-Poisson. The regional effect ziz_i is shared across all demographic groups jj in region ii, inducing positive cross-age correlation within a county.

Posterior mean (partial pooling): E(pij∣yij,ξ,zi)E(p_{ij} \mid y_{ij}, \xi, z_i) is a weighted average of the observed rate yij/nijy_{ij}/n_{ij} (weight ∝nijpˉij\propto n_{ij}\bar{p}_{ij}) and the expected rate θjzi\theta_j z_i (weight ∝σj−2\propto \sigma_j^{-2}). Small counties shrink heavily toward expected; large counties stay data-driven.

Estimation: Empirical Bayes via profile likelihood — estimate hyperparameters ξ=(θ,σ2,γ)\xi = (\theta, \sigma^2, \gamma) by ML (profile over γ\gamma, integrate over ziz_i), then estimate rates by posterior means given ξ^\hat{\xi}. Missouri lung cancer: 115 counties × 4 age groups, 1972–1981; likelihood-ratio (LR) test vs. fixed-effects Poisson = 51.4; Empirical Bayes (EB) standard deviations up to 15% smaller than raw rates.

Hierarchical Pooling of Air Pollution–Mortality Evidence (Dominici-Samet-Zeger 2000)

The NMMAPS (National Morbidity, Mortality, and Air Pollution Study) exemplifies the two-stage approach where Stage 1 delivers sufficient statistics and Stage 2 applies the hierarchical model to those statistics rather than to raw data.

Stage 1 — City-specific Poisson generalized additive model (GAM). For each of 20 US cities, city cc, age group aa, and day tt:

log⁡(μtac)=xtac′βc+γcDOWt+∑k=15Skc(⋅)+δac+∑aSac(timet,8/yr)\log(\mu^{ac}_t) = x^{ac\prime}_t \beta_c + \gamma_c \text{DOW}_t + \sum_{k=1}^{5} S^c_k(\cdot) + \delta^c_a + \sum_a S^{ac}(\text{time}_t,8/\text{yr})

where DOWt\text{DOW}_t is a day-of-week indicator, SkcS^c_k are natural splines for long-run time trend (7 df/yr), same-day temperature (6 df), 1–3 day lagged temperature (6 df), same-day dew point (3 df), lagged dew point (3 df), and xtacx^{ac}_t = (PM₁₀, O₃) with lags 0, 1, 2. Output per city: β^c∈Rp\hat{\beta}_c \in \mathbb{R}^p and sample covariance VcV_c. The large sample (T=550T = 550–25502550 days) justifies the normal approximation β^c∣βc∼Np(βc,Vc)\hat{\beta}_c \mid \beta_c \sim N_p(\beta_c, V_c).

Stage 2 — Bayesian hierarchical pooling. Base-line independence model:

β^c∣βc∼Np(βc,Vc)\hat{\beta}_c \mid \beta_c \sim N_p(\beta_c, V_c) βc=zc′α+εc,εc∣Σ∼Np(0,Σ)\beta_c = z^\prime_c \alpha + \varepsilon_c, \quad \varepsilon_c \mid \Sigma \sim N_p(0, \Sigma) α∼N(0,100I),Σ∼IWp(df=p+1=3,  D=3I)(IW = Inverse-Wishart)\alpha \sim N(0, 100I), \quad \Sigma \sim \mathrm{IW}_p(df=p+1=3,\; D=3I) \quad \text{(IW = Inverse-Wishart)}

where zc=(1,Ppovertyc,P>65c,XˉPM10c)z_c = (1, P^c_\text{poverty}, P^c_{>65}, \bar{X}^c_{\text{PM}_{10}}). All full conditionals are conjugate (normal for α\alpha and βc\beta_c, IW for Σ\Sigma): a pure Gibbs sampler runs 3000 iterations with burn-in 100 and thinning by 5. Convergence via Raftery-Lewis (1992) CODA (Nmin≈2000N_\text{min} \approx 2000).

Spatial extension. City correlations follow an exponential decay model:

corr(βc,βc′)=exp⁡{−θ d(c,c′)}\text{corr}(\beta_c, \beta_{c'}) = \exp\{-\theta\, d(c,c')\}

The decay parameter θ∼LogNormal(0.2,0.52)\theta \sim \text{LogNormal}(0.2, 0.5^2) lacks a conjugate conditional; a Metropolis-Hastings (M-H) step with gamma proposal (acceptance 0.3–0.5) is required. A regional step-function alternative recovers closed-form conditionals: τ2∼IG(5,8.5)\tau^2 \sim IG(5, 8.5) (IG = Inverse-Gamma).

Key results. Univariate base-line, lag 0: +10 μg/m³ PM₁₀ → +0.48% daily mortality (95% credible interval [CI]: 0.05–0.92), Pr(α₀ > 0) = 0.98. Bivariate (PM₁₀ + O₃): +0.52% (0.16–0.85). Between-city standard deviation σ^≈0.76\hat{\sigma} \approx 0.76 (0.41–1.37), approximately twice the pooled effect — substantial heterogeneity. City-level covariates do not explain it. Robust across 27 sensitivity scenarios (3 models × 3 lags × 3 priors).

Limitation. The IW prior with only p+1=3p+1=3 degrees of freedom is influential relative to 20 cities. The profile likelihood for σ\sigma has 95% mass approximately on (0,0.4)(0, 0.4), narrower than the IW posterior's (0.41,1.37)(0.41, 1.37) — a standard small-nn problem for IW hyperpriors.

Level-Specific R² and Pooling Factor λ (Gelman-Pardoe 2004)

For a multilevel model with level-m units k=1,…,Kmk = 1,\ldots,K_m, let θk\theta_k be the quantity of interest and εk=θk−μk\varepsilon_k = \theta_k - \mu_k the residual after the linear predictor μk\mu_k. Two Markov Chain Monte Carlo (MCMC)-computed summary measures (posterior expectations of variance ratios) are:

Level-m R2R^2 (eq. 7): R2=1−E[Vk(εk)]E[Vk(θk)]R^2 = 1 - \frac{E[V_k(\varepsilon_k)]}{E[V_k(\theta_k)]}

Measures the fraction of level-m variance explained by the predictor μ\mu. VkV_k denotes variance across units kk. Reduces to classical ordinary least squares (OLS) R² with no pooling (Xu 2003).

Pooling factor λ (eq. 8): λ=1−Vk(E(εk))E[Vk(εk)]\lambda = 1 - \frac{V_k(E(\varepsilon_k))}{E[V_k(\varepsilon_k)]}

Measures the fraction of residual variance reduced by the hierarchical model for the residuals εk\varepsilon_k. λ≈0\lambda \approx 0: each group is estimated independently (no pooling). λ≈1\lambda \approx 1: all residual structure is captured by the group-level model.

Parallel structure: R² and λ have the same denominator structure. R² measures predictor fit to θk\theta_k; λ measures hierarchical shrinkage of εk\varepsilon_k.

Bayesian adjusted R2R^2 for linear regression (nn obs., pp predictors): Radj,Bayes2=1−n−3n−p−2⋅yT(I−H)yyTIcyR^2_{\text{adj,Bayes}} = 1 - \frac{n-3}{n-p-2} \cdot \frac{y^T(I-H)y}{y^T I_c y}

Strictly lower than the classical Wherry (1931) adjusted R2R^2 (factor (n−1)/(n−p)(n-1)/(n-p)). For the balanced one-way hierarchical model, λ reduces to the Bühlmann credibility weight ω=1−σ^α2/(σ^α2+σ^y2/n)\omega = 1 - \hat\sigma^2_\alpha / (\hat\sigma^2_\alpha + \hat\sigma^2_y/n).

Minnesota radon example (919 houses, J=85 counties; Model 4 with uranium predictor):

Level R2R^2 λ\lambda
Household (yy) 0.21 0.03
County intercept (α\alpha) 0.53 0.81
County slope (β\beta) 0.83 0.97

Uranium explains 83% of cross-county slope variation; hierarchical pooling accounts for 97% of slope residual variance (slopes are nearly entirely determined by the county-level uranium model).

Both measures are free MCMC byproducts (no null model needed) and can be negative (misspecification signal).

Varying-Slope Multilevel Logistic Regression: Rich-State/Poor-State Voting Paradox (Gelman-Shor-Bafumi-Park 2007)

The apparent contradiction — rich states vote Democratic, rich individuals vote Republican — is resolved by allowing the income slope to vary across states and to depend on average state income.

Varying-intercept model (eq. 1): Constant slope β\beta, state-specific intercepts:

Pr⁡(yi=1)=logit−1(αs[i]+βxi)\Pr(y_i = 1) = \text{logit}^{-1}(\alpha_{s[i]} + \beta x_i)

Varying-intercept, varying-slope model (eq. 2): Both intercept and slope vary by state, each modeled as a linear function of average state income usu_s:

Pr⁡(yi=1)=logit−1(αs[i]+βs[i]xi)\Pr(y_i = 1) = \text{logit}^{-1}(\alpha_{s[i]} + \beta_{s[i]} x_i) αs=aα+bαus+εsα,βs=aβ+bβus+εsβ\alpha_s = a^\alpha + b^\alpha u_s + \varepsilon^\alpha_s, \quad \beta_s = a^\beta + b^\beta u_s + \varepsilon^\beta_s

Key finding: b^β<0\hat b^\beta < 0 — the income–Republican slope is steepest in poor states (Mississippi) and near-zero in rich states (Connecticut). The aggregate relationship runs in the opposite direction because richer states have higher αs\alpha_s (baseline Democratic lean), which dominates the cross-state comparison even as βs>0\beta_s > 0 within each state.

County-level model (§2.4): yc=αs[c]+βs[c]xc+εcy_c = \alpha_{s[c]} + \beta_{s[c]} x_c + \varepsilon_c fits county average income and vote shares with the same hierarchical structure — confirms individual-level inference from aggregated data.

Pattern emergence: Essentially absent 1968–1988; emerged post-1992 (Figures 14–16). The geographic income–vote correlation is a post-Cold-War phenomenon. Estimation via WinBUGS; model checking via binned residual plots.

Random-Effects Logit for Conjoint Analysis (Allenby-Rossi-McCulloch 2005, eqs. 16–19)

The standard hierarchical Bayes (HB) conjoint model applies the three-level hierarchy to a logit observation layer:

Pr⁡h(i)=exp⁡(xi′βh)∑jexp⁡(xj′βh),βh=Γzh+ξh,ξh∼N(0,Vβ)\Pr_h(i) = \frac{\exp(x_i'\beta_h)}{\sum_j \exp(x_j'\beta_h)}, \quad \beta_h = \Gamma z_h + \xi_h, \quad \xi_h \sim \mathcal{N}(0, V_\beta)

Priors: Γ∼N(a,A)\Gamma \sim \mathcal{N}(a, A); Vβ−1∼Wishart(w,W)V_\beta^{-1} \sim \text{Wishart}(w, W). Gibbs cycle — (1) each βh\beta_h one respondent at a time (Metropolis for non-conjugate logit likelihood); (2) Γ∣{βh},Vβ∼N\Gamma|\{\beta_h\}, V_\beta \sim \mathcal{N}; (3) Vβ∣{βh},Γ∼IWV_\beta|\{\beta_h\}, \Gamma \sim \text{IW}; (4) repeat.

Credit card application: N=946 bank customers, 14,799 paired comparisons, 7 attributes; 20,000 iterations, convergence by ~6,000. Key finding: comparing low annual fee vs. low fixed interest as incentives to overcome the −3.614 out-of-state penalty, despite low interest having higher mean utility, the low-fee incentive produces 7.5% of respondents above a net-utility profitability threshold vs. 4.5% for low interest — nearly 2×. The explanation is the off-diagonal of V^β\hat V_\beta: Cov(low fee, out-of-state)=8.5\text{Cov}(\text{low fee, out-of-state}) = 8.5 (ρ=0.80) vs. Cov(low interest, out-of-state)=1.9\text{Cov}(\text{low interest, out-of-state}) = 1.9 (ρ=0.23). Respondents who are fee-sensitive also tend to be less penalized by the out-of-state attribute.

Blocked MCMC for Longitudinal Models (Chib-Carlin 1999)

The standard Gaussian linear mixed model yi=Xiβ+Wibi+εiy_i = X_i\beta + W_i b_i + \varepsilon_i, bi∼Nq(0,D)b_i\sim\mathcal{N}_q(0,D), εi∼N(0,σ2I)\varepsilon_i\sim\mathcal{N}(0,\sigma^2 I) with Wishart prior on D−1D^{-1} and Gamma prior on σ−2\sigma^{-2}, admits progressively blocked Gibbs samplers that dramatically reduce autocorrelation relative to the standard 4-block algorithm (Gelfand-Smith 1990).

Algorithm 2 (partial blocking): Marginalize yiy_i over bib_i: yi∣β,σ2,D∼Nni(Xiβ,Vi)y_i|\beta,\sigma^2,D\sim\mathcal{N}_{n_i}(X_i\beta, V_i) where Vi=σ2I+WiDWi′V_i=\sigma^2I+W_iDW_i'. This gives a closed-form β∣y,σ2,D∼Np(β^,B)\beta|y,\sigma^2,D\sim\mathcal{N}_p(\hat\beta,B) that does not condition on the random effects. After sampling β, draw bi∣β,σ2,Db_i|\beta,\sigma^2,D from its standard conditional. Autocorrelation time κ(β)\kappa(\beta) drops to 1.0 (essentially iid) in the HIV CD4 example.

Algorithm 3 (single block): The joint marginal π(σ2,D−1∣y)\pi(\sigma^2,D^{-1}|y) has no closed form, but the integrated likelihood f(y∣σ2,D)f(y|\sigma^2,D) can be evaluated via the Chib (1995) identity as a ratio of three tractable Normal densities: f(y∣σ2,D)=ϕp(β∗∣β0,B0)∏iϕni(yi∣Xiβ∗,Vi)ϕp(β∗∣β^,B)f(y|\sigma^2,D) = \frac{\phi_p(\beta^*|\beta_0,B_0)\prod_i\phi_{n_i}(y_i|X_i\beta^*,V_i)}{\phi_p(\beta^*|\hat\beta,B)} evaluated at the Algorithm 2 posterior mean β∗\beta^*. Update θ=(σ2,ψ)\theta=(\sigma^2,\psi) jointly by M-H with a multivariate-t proposal calibrated from a short Algorithm 2 warmup run. Reduces κ(D21)\kappa(D_{21}) from 60 → 11.

Binary longitudinal extension: For binary responses with probit link, data augmentation (Albert-Chib 1993) introduces latent zit∣bi∼N(xit′β+wit′bi,1)z_{it}|b_i\sim\mathcal{N}(x_{it}'\beta+w_{it}'b_i,1). Marginalizing ZiZ_i over bib_i yields Zi∼Nni(Xiβ,Ini+WiDWi′)Z_i\sim\mathcal{N}_{n_i}(X_i\beta, I_{n_i}+W_iDW_i') — a special case of the Chib-Greenberg (1998) multivariate probit. Algorithm 7 marginalizes over both bib_i and ZZ by computing f(yi∣β,D)f(y_i|\beta,D) via the GHK (Geweke–Hajivassiliou–Keane) method (Cholesky + sequential truncated-Normal draws + product of Normal CDFs); lag-10 β autocorrelation drops from 0.934 to 0.009 (PSID (Panel Study of Income Dynamics), two random effects (REs)) but at high computational cost. Algorithm 5 (partial blocking only) is the recommended practical choice.

Universal rule: fixed effects should always be sampled marginalized over random effects. See Gibbs Sampler and Chib-Carlin (1999).

Conjugate Likelihood Distributions and Hyperparameter Gibbs Sampling (George-Makov-Smith 1993)

In a conjugate exponential-family hierarchical model — data layer xi∣θi∼f(x∣θi)x_i|\theta_i\sim f(x|\theta_i) and prior layer θi∣α,β∼π(θ∣α,β)\theta_i|\alpha,\beta\sim\pi(\theta|\alpha,\beta) — the Gibbs conditionals for θi\theta_i are standard conjugate draws. The obstacle is the hyperparameter block: its full conditional is proportional to the conjugate likelihood distribution,

π(α,β∣θ1,…,θp)∝L(α,β∣θ1,…,θp)⋅π(α,β)\pi(\alpha,\beta|\theta_1,\ldots,\theta_p) \propto L(\alpha,\beta|\theta_1,\ldots,\theta_p)\cdot\pi(\alpha,\beta)

where L(x0,n0∣θ1:p)∝exp⁡ ⁣[x0∑iθi−n0∑iψ(θi)−pφ(x0,n0)]L(x_0,n_0|\theta_{1:p}) \propto \exp\!\left[x_0\sum_i\theta_i - n_0\sum_i\psi(\theta_i) - p\varphi(x_0,n_0)\right].

Theorem 3.3 (George-Makov-Smith 1993): LL is log-concave in (α,β)(\alpha,\beta) — enabling Gilks-Wild (1992) adaptive rejection sampling (ARS) — and jointly proper iff p≥2p\geq 2. This resolves the hyperparameter bottleneck: the shape hyperparameter samples by ARS; the scale hyperparameter often draws conjugately; every step is exact.

Gamma-Poisson pump failure application (p=10p=10, §4): With xi∣θi∼Poisson(θiti)x_i|\theta_i\sim\text{Poisson}(\theta_i t_i) and θi∣α,β∼Gamma(α,β)\theta_i|\alpha,\beta\sim\text{Gamma}(\alpha,\beta), the explicit conjugate likelihood is LG(α,β∣θ1:p)∝(βα/Γ(α))p(∏iθi)α−1exp⁡(−β∑iθi)L^G(\alpha,\beta|\theta_{1:p})\propto(\beta^\alpha/\Gamma(\alpha))^p(\prod_i\theta_i)^{\alpha-1}\exp(-\beta\sum_i\theta_i). The complete Gibbs cycle: (a) θi∣⋅∼Gamma(α+xi,β+ti)\theta_i|\cdot\sim\text{Gamma}(\alpha+x_i,\beta+t_i); (b) α∣⋅\alpha|\cdot by ARS; (c) β∣⋅∼Gamma\beta|\cdot\sim\text{Gamma}. 100,000 draws across four hyperprior specifications confirm shrinkage of pump-specific rates toward a common value — shrinkage calibrated by data through α\alpha and β\beta.

Why It Matters

Open Questions

Related