Laplace Approximation

Definition

The Laplace approximation is a family of analytic methods for approximating intractable integrals in Bayesian inference, based on Laplace's asymptotic expansion of integrals of the form $\int\exp\{-nf(\theta)\}d\theta$ . For a joint density $p(\theta|y) \propto b(\theta)\exp\{-H(\theta)\}$ with $H(\theta)=nh(\theta)$ and a dominant interior mode at $\hat\theta$ , the key idea is to expand $H$ to second order around $\hat\theta$ — producing a Gaussian approximation — and integrate this approximation analytically. The method generalises from coordinate margins (Tierney-Kadane 1986) to arbitrary smooth nonlinear functions $g(\theta)$ (Tierney-Kass-Kadane [TKK] 1989a) and achieves saddlepoint accuracy: relative errors O(n⁻¹) uniformly on compact sets.

Key Ideas

Basic Laplace integral: $\int\exp\{-nf(\theta)\}d\theta \approx (2\pi/n)^{d/2}|f''(\hat\theta)|^{-1/2}\exp\{-nf(\hat\theta)\}$ with relative error O(n⁻¹), where $\hat\theta$ is the global minimiser of f.
Coordinate margin (Tierney-Kadane 1986): For the joint density $p(\theta_1,\theta_2)\propto\exp\{-H(\theta_1,\theta_2)\}$ , the marginal of $\theta_1$ is: $\tilde{p}(\theta_1) \propto \frac{[\det\Omega(\theta_1)]^{1/2}}{[\det\Omega]^{1/2}} b(\theta_1,\hat\theta_2(\theta_1))\exp\{-H(\theta_1,\hat\theta_2(\theta_1))\}$ where $\hat\theta_2(\theta_1)$ minimises $H(\theta_1,\cdot)$ and $\Omega(\theta_1)=[\nabla^2_{\theta_2}H(\theta_1,\cdot)|_{\hat\theta_2}]^{-1}$ .
Nonlinear function margin (TKK 1989a): For any smooth $g:\mathbb{R}^d\to\mathbb{R}^k$ with full-rank Jacobian near $\hat\theta$ , the marginal density of $g(\theta)$ at $y$ is approximated by equation (3) of TKK (1989a) — see Tierney-Kass-Kadane (1989a) for the full formula. Requires only constrained minimisation of $H$ subject to $g(\theta)=y$ , Hessian evaluation at the constrained mode, and the Jacobian $Dg$ .
Posterior expectations (TK 1986; TKK 1989b): $E[g(\theta)|y] \approx$ (ratio of two Laplace approximations to numerator and denominator integrals). TKK (1989b) "fully exponential" form incorporates $b(\theta)$ into the exponent, achieving O(n⁻²) relative error vs. O(n⁻¹) for the basic form.
Marginal likelihood and Bayesian Information Criterion (BIC): $\int p(y|\theta)p(\theta)d\theta \approx p(y|\hat\theta)p(\hat\theta)(2\pi/n)^{d/2}|H''(\hat\theta)/n|^{-1/2}$ . Taking logs and dropping O(1) terms gives BIC $= \log p(y|\hat\theta) - (d/2)\log n$ — the BIC is the leading-term Laplace approximation to the log marginal likelihood.
Saddlepoint accuracy: In both the Bayesian and sampling contexts, relative errors are O(n⁻¹) uniformly on compact sets — the same rate as the Barndorff-Nielsen-Cox saddlepoint approximation. Renormalized forms achieve O(n⁻³/²) on shrinking neighbourhoods.
Equivariance: The TKK (1989a) marginal-density approximation is equivariant under invertible reparameterisations of $g$ , but not under nonlinear reparameterisations of $\theta$ . Parameterising $\theta$ so that $H$ is closer to quadratic typically improves accuracy.

How It Works

Univariate Laplace Approximation

For $I = \int b(\theta)\exp\{-H(\theta)\}d\theta$ : expand H around its minimiser $\hat\theta$ : $H(\theta) \approx H(\hat\theta) + \tfrac{1}{2}(\theta-\hat\theta)^2 H''(\hat\theta)$ Substituting and integrating the resulting Gaussian: $I \approx b(\hat\theta)\exp\{-H(\hat\theta)\} \left(\frac{2\pi}{H''(\hat\theta)}\right)^{1/2}$ Relative error is O(n⁻¹) when $H=nh$ and $n\to\infty$ .

Marginal Density Grid Evaluation (TKK 1989a)

To evaluate $\tilde p(y)$ on a grid $y_1,\ldots,y_m$ :

At each $y_j$ : minimise $H(\theta)$ subject to $g(\theta)=y_j$ (Newton steps starting from $\hat\theta(y_{j-1})$ ).
Compute $\Omega(y_j)=[\nabla^2H(\hat\theta(y_j))]^{-1}$ and Jacobian $Dg = \nabla g(\hat\theta(y_j))$ .
Evaluate eq. (3). Keep $b(\hat\theta(y_j))\exp\{-H(\hat\theta(y_j))\}$ from the mode.
Renormalise by numerical integration over the grid.

Why It Matters

Historical role: Laplace approximations were the primary analytic tool for Bayesian posterior computation from Tierney-Kadane (1986) until Markov Chain Monte Carlo (MCMC) became practical after Gelfand-Smith (1990). The TKK (1989a,b) papers defined the pre-MCMC state of the art.
BIC and model selection: The BIC — used throughout this wiki for VAR (vector autoregression) lag selection, variable selection, and model comparison — is a direct consequence of the Laplace approximation to the log marginal likelihood.
INLA: Integrated Nested Laplace Approximation (Rue-Martino-Chopin 2009) applies nested Laplace approximations to compute posteriors for latent Gaussian models orders of magnitude faster than MCMC. INLA is the dominant tool for Bayesian spatial and hierarchical time-series models in epidemiology.
Accuracy hierarchy: Basic Laplace O(n⁻¹) < TKK (1989b) fully exponential O(n⁻²) < Exact MCMC.

Open Questions

Accuracy degrades for skewed, multi-modal, or high-dimensional posteriors where the Gaussian approximation at the mode is poor.
For $p\gg n$ settings, the mode-based approximation breaks down; variational Bayes or MCMC is preferred.
Automatic reparameterisation to improve accuracy (e.g., log-scale for positive parameters) relies on heuristics; no systematic theory.