Tierney, Kass & Kadane (1989b) Fully Exponential Laplace Approximations to Expectations and Variances of Nonpositive Functions

Summary

Extends the "fully exponential" Laplace approximation of Tierney-Kadane (1986) — previously restricted to positive functions — to posterior expectations and variances of arbitrary functions $g(\theta)$. The key device is to approximate the moment-generating function (MGF) $M(s) = E[\exp\{sg(\theta)\}]$ via the existing fully exponential form (since $\exp\{sg\}$ is always positive), then differentiate: $\tilde E(g) = (d/ds)\log\tilde M(s)|_{s=0}$ and $\tilde V(g) = (d^2/ds^2)\log\tilde M(s)|_{s=0}$. This MGF method is shown to be arithmetically identical to both the "large constant" device and the standard second-order expansion of Lindley (1961) and Mosteller-Wallace (1964), yet requires only second derivatives of the log-posterior rather than third.

Key Claims

• Accuracy (Theorem 1): $\tilde E(g) = E(g) + O(n^{-2})$; $\tilde V(g) = V(g) + O(n^{-2})$ — one order better than the basic Laplace $O(n^{-1})$.
• Elegant form (Theorem 2): If $\hat\theta_s$ maximises $-h(\theta) + sg(\theta)/n$ with $\sigma_s = [h''(\hat\theta_s)]^{-1/2}$, then $\tilde E(g) = g(\hat\theta) + (d/ds)\log\sigma_s|_{s=0} + (d/ds)\log b(\hat\theta_s)|_{s=0}$; variance analogously via second derivatives.
• Arithmetic identity (Theorem 3): MGF method and standard form (eq. 2.6, Lindley/Mosteller-Wallace) produce the same second-order approximation.
• Large-constant equivalence (Theorem 4): $\lim_{c\to\infty} [\tilde E(c+g)-c] = \tilde E(g)$ and $\lim_{c\to\infty} \tilde V_c(g) = \tilde V(g)$.
• Normal/gamma posteriors: Approximation is exact (matches $M(s)$ exactly for all $s$).
• Student-t posterior ($k$ df): $\tilde V(X) = k(k+4)/(k+1)^2 \approx k/(k-2)$ with relative error $O(k^{-2})$; first-order approximation $k/(k+1)$ is always below 1.
• Computational efficiency: MGF method uses only second derivatives; more efficient than standard form (which needs third derivatives) when parameter dimension $m > 2$ (one Newton step) or $m > 10$ (two Newton steps for variance). Standard form is more efficient when many different $g$ functions are evaluated at the same mode.
• Parameterisation sensitivity: In the Stanford heart-transplant example (Turnbull-Brown-Hu 1974, $N=82$), MGF errors ~10–12% in original $(\tau,\lambda,p)$ parameterisation vs. ~3–4% in $(\log\tau, \log\lambda, \log p)$; direct fully exponential form errors ~4% in original scale and ~3–4% on log scale. Parameterising $\theta$ so $H$ is closer to quadratic is crucial.

Concepts Introduced or Extended

• Laplace Approximation

Entities Mentioned

• Luke Tierney
• Robert E. Kass
• Joseph B. Kadane

Quotes

"Tierney and Kadane (1986) applied the Laplace method in a special form, which we call fully exponential, that has the advantage of requiring only second derivatives of the log-likelihood function to achieve a second-order approximation to the expectation and variance of a real function $g$ of the vector parameter $\theta$, but it has the disadvantage of requiring $g$ to be positive. The purpose of this article is to extend the fully exponential method to expectations of arbitrary functions while retaining its advantageous simplicity."

"Since (3.1) does not require explicit third derivatives, it is easier to use, and this is the basis for our preference for it in our current software implementation."

My Take

The companion to TKK (1989a): together they close the gap between the basic Laplace approximation ($O(n^{-1})$, coordinate margins only) and exact Markov chain Monte Carlo (MCMC). The unification of Lindley's, Mosteller-Wallace's, and the MGF approach via Theorem 3 is elegant — it shows that the "standard" expansion was always achievable without third derivatives. The parameterisation sensitivity finding (errors 3$\times$ larger in original vs. log scale) is practically important and echoes TKK (1989a)'s equivariance caveat: the method is only as good as the Gaussian approximation to the log-posterior at the mode.