Geweke (1996) Bayesian Inference for Linear Models Subject to Linear Inequality Constraints

Summary

Geweke (1996) develops Bayesian computational methods for the normal linear regression model $y = X\beta + \varepsilon$ subject to linear inequality constraints $a \leq D\beta \leq w$ on the coefficients. Two distinct problems are addressed: (1) evaluating the posterior probability $p_{2|1}$ that the constraints hold (or the posterior odds ratio in favor of the constrained model), handled via the Geweke-Hajivassiliou-Keane (GHK) probability simulator; and (2) computing posterior moments and probabilities conditional on the constraints, handled via a Gibbs sampler that draws each $z_j = [D(\beta-b)]_j$ from a univariate truncated normal. For the hard automobile sales example (k=11, $p_{2|1} \approx 9\times10^{-5}$) the Gibbs sampler is 5× faster than the crude accept-reject simulator at equal accuracy.

Key Claims

• Constraint probability $p_{2|1}$: under the unconstrained reference prior the posterior probability of $a \leq D\beta \leq w$ is $p_{2|1}$; for $r$ sign constraints the posterior odds ratio is $p_{2:1} = 2^r p_{2|1}$ (Section 2).
• GHK probability simulator (Section 2): let $z = D(\beta-b)$, Cholesky-factor $D(X'X)^{-1}D' = FF'$; decompose $P(a^* \leq z \leq w^*|\sigma)$ into a product of $k$ univariate conditional probabilities (eq. 2.4), each evaluated analytically from the standard normal cumulative distribution function (CDF). GHK is always more efficient than the crude frequency simulator; 265× fewer iterations needed for the automobile sales example.
• Gibbs sampler for constrained inference (Section 3): reparametrize to $z = D(\beta-b)$; then (a) $\sigma|z$: $(y-X\beta)'(y-X\beta)/\sigma^2 \sim \chi^2(T)$ (eq. 3.2); (b) $z_j|\sigma, z_{i\neq j}$: univariate truncated Normal on $[a_j^*, w_j^*]$ with mean $b_j + \sum_{i\neq j} c_{ij}z_i$ and variance $h_j^2 = (T_{jj})^{-1}$ (eq. 3.3), where $T = \sigma^2 D(X'X)^{-1}D'$. The rotation $z = D\beta$ converts joint constraints into independent coordinate-wise truncation bounds, enabling direct per-coordinate sampling.
• Performance (Tables I-IV): rent data ($k=5$, $p_{2|1} \approx 0.05$): GHK 2× more efficient than crude frequency simulator; Gibbs sampler with every-10th draw matches crude simulator quality at lower cost. Auto sales data ($k=11$, $p_{2|1} \approx 9\times10^{-5}$): crude simulator requires 4 hours for 1,000 accepted draws; Gibbs with every-100th draw achieves 4% numerical standard error (NSE) in 49 minutes.
• Extensions (Section 5): The GHK + Gibbs framework extends to (a) more than $k$ constraints via accept-reject hybrid; (b) semi-informative priors — conditional on $\sigma$ the distribution of $\beta$ remains truncated multivariate normal; (c) scale-mixture normal models, explicitly citing Geweke (1993): conditional on $\sigma$ and the latent $v_t$, $\beta$ is truncated multivariate normal and the same GHK + Gibbs applies; (d) seemingly unrelated regression (SUR), censored regression, and multinomial probit models.

Concepts Introduced or Extended

• Gibbs Sampler — coordinate-wise truncated univariate Normal draws for constrained regression; the $z = D(\beta-b)$ reparameterization converts joint constraints to per-coordinate bounds
• Multinomial Probit — explicitly named as a model where GHK + Gibbs applies for inequality constraints (Section 5)
• Mixture of Normals — Section 5 extends to scale-mixture normal regression (citing Geweke 1993)

Entities Mentioned

• John Geweke

Quotes

"The GHK probability simulator always provides a more accurate approximation to $p_{2|1}$ than does the crude frequency simulator, given the same number of iterations."

"For this problem, the Gibbs sampler is clearly the method of choice."

My Take

A cleanly written methods paper whose main contribution is showing that the change of variables $z = D(\beta-b)$ converts a jointly constrained regression problem into a sequence of univariate truncated normal draws — something the Gibbs sampler handles directly. The two-task framing (evaluate vs. condition) is useful and often conflated in practice. The detailed performance tables across three algorithms are unusually honest about when each method wins. The extension to scale-mixture models in Section 5 is brief but important: it connects directly to Geweke (1993) and anticipates the broader role of truncated-normal Gibbs blocks in models like multinomial probit. The paper's scope is narrow by design, which keeps it practical.

Originally circulated (revised June 1995) as a Federal Reserve Bank of Minneapolis / University of Minnesota working paper; published as Geweke, J. (1996), "Bayesian Inference for Linear Models Subject to Linear Inequality Constraints," in J.C. Lee, W.O. Johnson, and A. Zellner (eds.), Modelling and Prediction Honoring Seymour Geisser, New York: Springer, pp. 248–263 (the citation of record).