Spike-and-Slab Prior

Definition

A spike-and-slab prior is a two-component mixture prior on a parameter $\beta_i$ that simultaneously models inclusion uncertainty and coefficient magnitude. The "spike" component concentrates mass near zero (either a point mass $\delta_0$ or a tight Gaussian $\mathcal{N}(0, \tau^2)$ with small $\tau$ ); the "slab" component is a diffuse Gaussian $\mathcal{N}(0, c^2\tau^2)$ with $c \gg 1$ that allows $\beta_i$ to take large values. A latent binary indicator $\gamma_i \in \{0,1\}$ selects the component. The marginal prior is $\beta_i \mid \gamma_i \sim (1-\gamma_i)\mathcal{N}(0,\tau_i^2) + \gamma_i\mathcal{N}(0,c_i^2\tau_i^2)$ .

Key Ideas

Inclusion indicator: $\gamma_i \sim \text{Bernoulli}(p_i)$ ; $p_i$ is the prior inclusion probability (often $1/2$ or a Beta hyperprior for automatic multiplicity correction)
Hard vs. soft spike: The point-mass spike ( $\delta_0$ ) produces exact sparsity and is the limit $\tau \to 0$ ; the Gaussian spike ( $\tau$ small but positive) keeps the Gibbs sampler ergodic and avoids a reducibility pathology
Stochastic Search Variable Selection (SSVS): George-McCulloch (1993, 1997) — three-block Gibbs over $(\beta, \sigma^2, \gamma)$ ; $\gamma_i$ updates via a Bernoulli draw whose probability is the ratio of spike and slab densities evaluated at the current $\beta_i$
Conjugate formulation: Tying $\beta$ 's prior variance to $\sigma^2$ (George-McCulloch 1997 §4) allows analytical integration of $\beta$ and $\sigma^2$ , reducing the sampler to a single Gibbs step over $\gamma$ and preventing the stuck- $\gamma$ pathology of the non-conjugate version
Multiplicity correction: A Beta $(a,b)$ hyperprior on the inclusion probability $q$ integrates out the unknown sparsity level and automatically penalises large models; Beta $(1,1)$ (flat on $q$ ) achieves the Scott-Berger (2006, 2010) automatic Bayesian multiplicity correction
Factor loading sparsity: Hahn-Carvalho-Scott (2012) apply spike-and-slab to factor loading matrices $(b_{js} \mid \nu_s, q_s) \sim q_s \mathcal{N}(0,\nu_s) + (1-q_s)\delta_0(b_{js})$ ; Beta $(1,1)$ on $q_s$ per-factor induces multiplicity correction across the $p \times k$ loading entries

How It Works

Given current draws $(\beta^j, \sigma^{2j})$ , the indicator $\gamma_i$ is updated via:

$P(\gamma_i=1 \mid \beta^j, \sigma^{2j}) = \frac{p_i \cdot \phi(\beta^j_i; 0, c_i^2\tau_i^2)}{p_i \cdot \phi(\beta^j_i; 0, c_i^2\tau_i^2) + (1-p_i) \cdot \phi(\beta^j_i; 0, \tau_i^2)}$

where $\phi(\cdot\,;\mu,\sigma^2)$ is the Normal density. The ratio of the two terms is the Bayes factor for inclusion. When the current $\beta^j_i$ is large relative to $\tau_i$ , the slab dominates and $\gamma_i = 1$ is likely; when $\beta^j_i$ is small, the spike dominates. After all $\gamma_i$ are updated, $\beta \mid \gamma, \sigma^2$ is multivariate Normal and $\sigma^2 \mid \beta, \gamma$ is Inverse-Gamma — both conjugate.

Posterior summaries: the frequency with which $\gamma_i = 1$ across Gibbs iterations estimates the marginal posterior inclusion probability (PIP) for variable $i$ . High-probability models are those visited frequently by the sampler.

Why It Matters

Spike-and-slab priors convert model selection — an inherently combinatorial $2^p$ search — into a continuous Bayesian computation tractable by Gibbs sampling. The latent $\gamma$ indicator acts as a data-augmentation device: conditional on $\gamma$ , all blocks are standard conjugate families. The resulting PIPs are interpretable as posterior model-averaging weights and are more honest about uncertainty than classical step-wise selection or Lasso point estimates. The conjugate version's analytical marginalization over $\beta$ and $\sigma^2$ (George-McCulloch 1997) dramatically improves mixing when information is weak.

Open Questions