O'Hara-Sillanpää (2009) A Review of Bayesian Variable Selection Methods: What, How and Which

Summary

This Bayesian Analysis review surveys the main Bayesian approaches to variable selection in regression and — crucially — compares how they behave in practice when implemented in BUGS. O'Hara and Sillanpää organize the field around five methods: Kuo & Mallick's indicator model, Gibbs Variable Selection (GVS), Stochastic Search Variable Selection (SSVS), adaptive shrinkage with a Jeffreys' or Laplacian (Bayesian-LASSO-type) prior, and reversible-jump MCMC (RJMCMC). They reframe selection as parameter estimation — estimating the marginal posterior probability that each variable belongs in the model, rather than searching for a single "best" model — and run all methods on simulated and real data. Their bottom line: SSVS, RJMCMC, and adaptive shrinkage can all work well, but which is best depends on the priors chosen and on implementation details (proposal/pseudo-prior tuning, mixing), not on the method label alone.

Key Claims

• Selection = estimating posterior model/variable probabilities. In the Bayesian framing, one does not pick one model; one estimates the posterior probability over models (or, in variable-specific form, the marginal posterior probability that each variable is included). MCMC is the standard computational engine, which is why the paper emphasizes methods that slot easily into BUGS/WinBUGS/OpenBUGS.
• The methods differ mainly in how the "in/out" indicator relates to the coefficient.
  • Kuo & Mallick (1998): write $\beta_j = I_j\,\theta_j$ with the inclusion indicator $I_j$ and the effect $\theta_j$ given independent priors. Simplest to code, but with correlated predictors or a poorly scaled $\theta$ prior it can mix badly, because the sampler rarely proposes sensible values for a currently-excluded coefficient.
  • Gibbs Variable Selection (GVS; Dellaportas–Forster–Ntzoufras): as Kuo–Mallick but the priors on $I_j$ and $\theta_j$ are linked, and a pseudo-prior governs $\theta_j$ when $I_j=0$, keeping the excluded coefficient in a plausible region to improve mixing — at the cost of having to specify that pseudo-prior.
  • SSVS (George & McCulloch 1993): a spike-and-slab mixture — the coefficient is always in the model but drawn from a two-component normal, a tight spike near zero (excluded) or a diffuse slab (included). It never sets coefficients exactly to zero; performance is sensitive to the spike variance.
  • Adaptive shrinkage (Jeffreys' or Laplacian prior): continuous shrinkage priors with no explicit indicator; the Laplacian case is the Bayesian LASSO, the Jeffreys' prior gives coefficient-specific adaptive shrinkage.
  • Reversible-jump MCMC (Green 1995): a trans-dimensional sampler that jumps directly between models of different dimension; flexible but needs well-designed between-model proposals.
• Sparse vs. tapering priors matter. The right prior on effect sizes differs by problem: a leptokurtic, sparse prior (a few large effects, most near zero — as in gene mapping) versus a slow taper with no clear tail. The paper focuses on the exploratory, sparse case.
• Empirical verdict. Across the examples, SSVS, RJMCMC, and adaptive shrinkage all perform well; Kuo–Mallick and GVS are workable but more sensitive to tuning. The practical choice hinges on the priors and their implementation rather than on theoretical superiority of any one method.
• Caveat against uncritical use. If variables were chosen because they are expected to matter, fitting the full model and interpreting posterior coefficient sizes may beat formal selection; selection is most appropriate for genuinely exploratory, high-dimensional, sparse problems.

Concepts Introduced or Extended

• Variable Selection
• Spike-and-Slab Prior
• Reversible Jump MCMC

Entities Mentioned

• R.B. O'Hara
• Mikko J. Sillanpää

Quotes

"Rather than searching for the single optimal model, a Bayesian will attempt to estimate the posterior probability of all models within the considered class of models."

"Our results suggest that SSVS, reversible jump MCMC and adaptive shrinkage methods can all work well, but the choice of which method is better will depend on the priors that are used, and also on how they are implemented."

My Take

This is the practical map of the Bayesian variable-selection landscape: it lines up the indicator-based methods (Kuo–Mallick, GVS, SSVS) against continuous-shrinkage and trans-dimensional (RJMCMC) alternatives and — unusually for a review — actually runs them in BUGS so the reader sees the tuning pain (pseudo-priors, spike variances, between-model proposals) that theory papers gloss over. For this wiki it is the connective tissue of the selection cluster: it situates SSVS, RJMCMC, and Bayesian-LASSO-style shrinkage as members of one family distinguished by how the inclusion indicator couples to the coefficient. The honest headline — that method ranking is dominated by prior choice and implementation, not by the method name — is the durable lesson, and its gene-mapping framing is a reminder that "sparse, leptokurtic effects" is the regime where selection earns its keep.