Bayesian Method of Moments

Definition

The Bayesian Method of Moments (BMOM), introduced by Arnold Zellner, is a framework for constructing post-data (posterior) densities for parameters and predictive densities for future observations without specifying a likelihood function. Instead of $p(y|\theta)$ , the analyst supplies moment constraints derived from the model — e.g. $E[y - X\beta] = 0$ , $E[(y - X\beta)'(y - X\beta)/n] = \sigma^2$ — and finds the probability distribution that satisfies these constraints while maximising entropy. The result is a fully operational Bayesian analysis that avoids "paying obeisance to a model one knows is false."

Key Ideas

Maximum entropy posterior: maximise $\int p(\theta)\log p(\theta)\,d\theta$ subject to moment constraints $\int g_j(y,\theta)\,p(\theta)\,d\theta = 0$ , $j=1,\ldots,k$ , where $g_j$ are functions summarising the information in data $y$ about $\theta$ . The solution is a posterior proportional to $\exp\!\bigl(\sum_j \lambda_j g_j(y,\theta)\bigr)$ , where $\lambda_j$ are Lagrange multipliers chosen to satisfy the constraints.
Normal regression result: under the standard regression constraints $E[\beta|D] = \hat{b}$ (Ordinary Least Squares, OLS), $\text{Cov}[\beta|D] = s^2(X'X)^{-1}$ , and treating $n-k$ degrees of freedom, the MaxEnt (maximum-entropy) posterior for $\beta$ is multivariate Normal with the OLS mean and OLS covariance. This is identical to the diffuse-prior Bayesian posterior — BMOM arrives at the same place without assuming normal errors.
Predictive density: from the posterior, BMOM generates a predictive density for future observations $y_f$ . With Normal posterior for $\beta$ , the predictive density for $y_f = W_f\beta + u_f$ is also Normal: $y_f \mid D \sim \mathcal{N}(W_f\hat{\beta},\, V_f)$ , where $V_f$ includes parameter uncertainty.
Conceptual sample: to introduce shrinkage/prior information, augment observed data with a "virtual" sample $z \sim \beta + v$ . The BMOM posterior with conceptual sample gives a weighted average of the data estimate and the prior centre, yielding Stein-type shrinkage without specifying a prior distribution; see Zellner (2002). The degree of shrinkage depends on the informativeness of the conceptual sample (g-prior interpretation: $Z_c'Z_c = g\,X'X$ ).
BMOM vs Traditional Bayes (TB): BMOM models and TB models can be compared via posterior odds: $K = p_{\text{BMOM}}(y_f|D) / p_{\text{TB}}(y_f|D)$ , using held-out data $y_f$ . When both produce Normal predictive densities, $\log K$ has a closed form involving log determinant ratios and Mahalanobis distances — related to the Jeffreys–Kullback-Leibler (KL) divergence $W = \text{CE}(f_0, f_1) + \text{CE}(f_1, f_0)$ , where $\text{CE}$ denotes cross-entropy.

How It Works

Parametric Regression

For $y = X\beta + \varepsilon$ , $n$ observations, $p$ parameters:

Compute OLS estimate $\hat{\beta} = (X'X)^{-1}X'y$ and residual variance $s^2 = (y - X\hat{\beta})'(y - X\hat{\beta})/(n-p)$ .
Impose moment constraints: the BMOM post-data mean and covariance equal the OLS mean and covariance.
MaxEnt solution: $\beta \mid D \sim \mathcal{N}(\hat{\beta},\, s^2(X'X)^{-1})$ . The marginal distribution of individual $\beta_j$ follows a $t$ -distribution with $n-p$ degrees of freedom.
Predictive density for $y_f$ : $\mathcal{N}(X_f\hat{\beta},\; s^2[I + X_f(X'X)^{-1}X_f'])$ .

Semiparametric Regression (Zellner-Tobias-Ryu 1997)

When the regression function $f(x)$ is unknown:

Approximate $f(x)$ by a series: $y = \sum_{j=1}^J \beta_j \psi_j(x) + \varepsilon$ , where $\psi_j$ are basis functions.
Apply BMOM to the resulting parametric model — the problem reduces to step 1 above.
Select $J$ $J$ (number of terms) and the basis type using:
- Estimation loss: minimise $\text{tr}(\text{Cov}[\beta|D]) = Js^2/(n-J-1)$ .
- Predictive loss ratio: $R = [(m+a_1)s_1] / [(m+a_2)s_2]$ ; favours model 1 if $R < 1$ .
- Posterior odds: $K = p(y_f \mid \text{model 1}, D) / p(y_f \mid \text{model 2}, D)$ , evaluated on hold-out data.

Available basis functions include Polynomial, Fourier (sin/cos), and Gallant's flexible functional form. The Gallant basis is the most flexible but requires more parameters; posterior odds typically favour simpler expansions unless the true function is highly non-linear.

Why It Matters

BMOM addresses genuine model fragility: standard Bayesian analysis is "brittle" (Lasky 1997) when the likelihood is uncertain — minor misspecification in the error distribution can propagate into large posterior distortions. BMOM sidesteps this by using only moment information. It produces interpretable, finite-sample posteriors and predictive densities that are automatically proper, and can be compared directly to TB models via posterior odds.

The approach has a natural connection to robust Bayesian statistics and to the information-theoretic foundations Zellner laid in his (1988) information-processing paper: the BMOM posterior is the unique minimum-information distribution consistent with the observed moment constraints.

Open Questions

BMOM does not extend easily to models with non-linear moment conditions or to time series with autocorrelated errors, where the moment constraints become significantly more complex.
The Normal BMOM result coincides with the diffuse-prior TB result, raising the question of whether BMOM provides additional robustness in practice or merely a philosophical reframing.
Extending BMOM to latent variable models (mixture models, state-space models) is non-trivial because the moment constraints become harder to derive without specifying a full distributional model.