Importance Sampling

Definition

Importance sampling is a Monte Carlo technique for estimating an expectation $E_\pi[h(\theta)] = \int h(\theta)\,\pi(\theta)\,d\theta$ under a target density $\pi$ that is hard to sample from, by drawing instead from a tractable proposal (importance) density $g$ and reweighting each draw by the importance weight $w(\theta) = \pi(\theta)/g(\theta)$. The identity $\int h(\theta)\pi(\theta)\,d\theta = \int h(\theta)\,w(\theta)\,g(\theta)\,d\theta$ turns an intractable integral into an expectation over $g$, so it can be approximated by a weighted average of draws $\theta_i \sim g$. When $\pi$ is known only up to a normalising constant — the usual case in Bayesian work, where $\pi(\theta)\propto f(y|\theta)\,p(\theta)$ — the self-normalised estimator $\hat{h} = \sum_i w_i h(\theta_i) / \sum_i w_i$ is used, which needs the weights only up to proportionality.

Key Ideas

• Change of measure, not of the estimand. The proposal $g$ is a computational device; the answer is still an expectation under $\pi$. Any $g$ with support covering $\pi$ gives a consistent estimator, but efficiency varies enormously.
• Efficiency hinges on the $g$–$\pi$ match. The estimator variance is governed by the variance of the weights $w(\theta)$. A good $g$ mimics $\pi$ closely, especially in the tails and around the mode.
• Fat tails are safer. A proposal with heavier tails than the target keeps weights bounded; a proposal with thinner tails than $\pi$ produces occasional enormous weights, giving an infinite-variance estimator and weight degeneracy — a few draws dominate the sum and the effective sample size collapses.
• Self-normalisation removes the need for the target's normalising constant but introduces a small $O(1/n)$ bias, negligible relative to the variance reduction it enables.
• Diagnostics. The distribution of the weights (e.g. the maximum weight, or effective sample size $(\sum w_i)^2/\sum w_i^2$, or relative numerical efficiency) reveals whether the sampler has collapsed. Uhlig (1997) Bayesian Vector Autoregressions with Stochastic Volatility recommends monitoring the maximum weight; Kadiyala-Karlsson (1997) Numerical Methods for Estimation and Inference in Bayesian VAR-Models tracks relative numerical efficiency (RNE), which drops toward zero as dimension grows.
• Variance reduction add-ons. Antithetic variates (sign-flipped draws through the mode) sharply cut variance for near-symmetric posteriors, as used in the Bayesian VAR importance samplers of Kadiyala and Karlsson.

How It Works

1. Choose an importance density $g$ that is easy to sample and approximates $\pi$; a common choice is a fat-tailed multivariate $t$ or a mixture centred at the posterior mode with curvature from the Hessian (a Laplace-type proposal; see Laplace Approximation).
2. Draw $\theta_1,\dots,\theta_n \sim g$.
3. Form weights $w_i = \pi(\theta_i)/g(\theta_i)$ (up to a constant when $\pi$ is unnormalised).
4. Estimate $E_\pi[h] \approx \sum_i w_i h(\theta_i) / \sum_i w_i$.
5. Assess reliability from the weight distribution; if degenerate, refit $g$ (heavier tails, better location/scale) or switch to a Markov chain method.

The same machinery estimates a marginal likelihood (normalising constant): $m(y) = \int f(y|\theta)p(\theta)\,d\theta = E_g[f(y|\theta)p(\theta)/g(\theta)]$, the basis of the Gelfand–Dey estimator on the Marginal Data Density. The notorious harmonic-mean estimator is the special case $g=$ prior, which is why it is unstable. Sequentially propagating and reweighting draws through a state-space model turns importance sampling into sequential importance sampling / SIR, the ancestor of the particle filter.

Why It Matters

Before Markov chain Monte Carlo became standard, importance sampling was the workhorse for Bayesian integration in econometrics. Early Bayesian VAR posteriors that lacked closed forms — the Normal-Diffuse and Extended Natural Conjugate priors of Kadiyala-Karlsson (1993) Forecasting with Generalized Bayesian Vector Autoregressions — were evaluated by the Kloek–Van Dijk importance sampler, and Kadiyala-Karlsson (1997) Numerical Methods for Estimation and Inference in Bayesian VAR-Models then compared competing importance functions (2-0 poly-$t$ vs. a mode-centred conditional-$t$) against Gibbs sampling, documenting that IS becomes infeasible in large systems while Gibbs stays efficient. Kleibergen-van-Dijk (1993) Non-Stationarity in GARCH Models: A Bayesian Analysis used Bayesian importance sampling to compute posterior odds for GARCH non-stationarity and a unit root in the mean under fat-tailed errors, and Uhlig (1997) Bayesian Vector Autoregressions with Stochastic Volatility used it for the marginal posterior of the VAR coefficients when integrating over latent stochastic-volatility paths. Rao-Blackwellisation of Sampling Schemes shows importance sampling can even be layered onto a Metropolis chain to cut estimator variance by 70–95%, casting doubt on the raw ergodic mean. As a marginal-likelihood tool it is the comparison benchmark against Chib's Gibbs-output method and bridge sampling on the Marginal Data Density. It is a close relative of Metropolis-Hastings Algorithm: both propose from a tractable $g$ and correct for the mismatch with $\pi$ — importance sampling reweights every draw, Metropolis-Hastings accepts or rejects.

Open Questions

• Proposal design in high dimensions. As the parameter dimension grows the weight variance explodes (the curse of dimensionality that sank IS for the 7-variable VAR in Kadiyala–Karlsson 1997). How to build adaptive or defensive-mixture proposals that stay efficient at scale remains open.
• When to prefer IS over MCMC. IS gives independent draws and a clean marginal-likelihood estimate but degenerates easily; MCMC scales better but needs convergence diagnostics. The trade-off is problem-specific and not fully characterised.
• Reliable degeneracy detection. Effective sample size and maximum-weight diagnostics can mask an infinite-variance estimator whose worst draw simply has not been seen yet.
• Adaptive and marginalised variants. Adaptive importance sampling, annealed IS, and Rao-Blackwellised particle methods improve robustness, but general guidance on tuning them for econometric posteriors is still thin.

Related

• Markov Chain Monte Carlo
• Metropolis-Hastings Algorithm
• Gibbs Sampler
• Marginal Data Density
• Bayes Factor
• Simulation-Based Estimation
• Stochastic Simulation
• Laplace Approximation
• Widely Applicable Information Criterion
• Vehtari-Gelman-Gabry (2017) — Pareto-smoothed IS
• Bridge Sampling