Percy (1992) Prediction for Seemingly Unrelated Regressions

Summary

Percy (1992) derives Bayesian predictive densities for the Seemingly Unrelated Regressions (SUR) model and demonstrates that the exact predictive density is analytically intractable in the general case — unlike the traditional multivariate regression model, where Jeffreys' prior yields a closed-form multivariate Student-t result (Zellner-Chetty 1965). Two tractable approximations are proposed and compared on simulated data: a three-block Gibbs sampler over $(y_{n+1}, \Phi, \beta)$ and a first-order approximation that integrates out $\beta$ analytically and plugs in the modal Bayes estimate of the precision matrix, yielding a closed-form multivariate normal predictive density. Both approximations closely track the exact Bayesian densities for degenerate sub-models (bivariate cases with known solutions), providing indirect validation for the general case. The paper is one of the earliest applications of Gibbs sampling to the SUR prediction problem.

Key Claims

• Analytical intractability: Under Jeffreys' invariant prior $f(\beta, \Phi) \propto |\Phi|^{-(p+1)/2}$, integrating $\Phi$ from the joint predictive density (eq. 5) leaves an integral over $\beta$ involving $|\sum_{i=1}^{n+1}(y_i - X_i\beta)(y_i - X_i\beta)^T|^{-(n+1)/2}$. Drèze (1977) showed this is intractable when the SUR design matrices $X_i$ differ across equations — the integrand does not simplify to a recognisable form. The traditional multivariate regression case ($x_{i1}=\cdots=x_{ip}$ for all $i$) is the tractable exception, giving a multivariate Student-t predictive density (Zellner-Chetty 1965).
• Gibbs sampler (Section 3.1): Three-block cycle (eqs. 6–8) with conditionals: $f(y_{n+1}|X_{n+1},\beta,\Phi,\mathbb{T}) \equiv N_p[X_{n+1}\beta, \Phi^{-1}]$; $f(\Phi|X_{n+1},y_{n+1},\beta,\mathbb{T}) \equiv W_p\!\left[n+1,\bigl(\sum_{i=1}^{n+1}(y_i-X_i\beta)(y_i-X_i\beta)^T\bigr)^{-1}\right]$; $f(\beta|X_{n+1},y_{n+1},\Phi,\mathbb{T}) \equiv N_q\!\left[(\sum X_i^T\Phi X_i)^{-1}\sum X_i^T\Phi y_i,\;(\sum X_i^T\Phi X_i)^{-1}\right]$. After $t=100$ burn-in steps, the estimated predictive density is the average of $m=200$ normal densities evaluated at sampled $(\beta^{(j)},\Phi^{(j)})$ pairs (eq. 9).
• First-order approximation (Section 3.2): Integrate $\beta$ from the predictive density analytically; replace $\Phi$ by its modal Bayes estimate $\hat\Phi$ — the maximiser of the log-integrated posterior $\ln f(\Phi|\mathbb{T}) = c + \frac{n-p-1}{2}\ln|\Phi| - \frac{1}{2}\ln|S_{XX}| - \frac{1}{2}|S_{MM}|/|S_{XX}|$ (eq. 10), computed via Newton's method. The resulting approximate predictive density is multivariate normal (eq. 11): $f(y_{n+1}|X_{n+1},\mathbb{T}) \equiv N_p[\mu,\Sigma]$ with $\mu = X_{n+1}\hat{S}_{XX}^{-1}\hat{S}_{Xy}$ and $\Sigma = \hat\Phi^{-1} + X_{n+1}\hat{S}_{XX}^{-1}X_{n+1}^T$.
• Simulation validation: Four bivariate ($p=2$) SUR cases with $n=10$, $q_1=q_2=2$: A (uncorrelated errors, shared regressors $\to$ multivariate regression); B (uncorrelated, orthogonal regressors $\to$ two independent regressions); C (correlation 0.99, shared regressors $\to$ traditional multivariate regression); D (correlation 0.99, orthogonal regressors $\to$ true SUR). Contour plots (Figs 1–4) show both approximations closely matching the exact Bayesian density for A–C; for D the two approximations agree with each other.
• Missing response extension (Section 5): When components of $y_i$ are missing, their conditional distribution given observed components and parameters is multivariate normal by standard partitioning. Adding this as a fourth Gibbs conditional extends the sampler without structural change; the first-order approximation extends by redefining $\hat\Phi^{(0)}$ on the observed training data only (eq. 13).

Concepts Introduced or Extended

• Seemingly Unrelated Regression
• Gibbs Sampler
• Bayesian Multiperiod Prediction

Entities Mentioned

• David F. Percy
• Arnold Zellner

Quotes

"The predictive density for this model cannot, in general, be evaluated analytically. Two approximations are proposed and investigated. These are Gibbs sampling and a first-order approximation based on a Bayes estimate of the precision matrix."

"For both the $L_1$- and supremum norms, Geman and Geman (1984) showed that the Gibbs algorithm converges to the actual predictive density as $t\to\infty$."

My Take

A concise methodological paper that cleanly identifies the intractability barrier in Bayesian SUR prediction and proposes two workable solutions. Its principal historical significance is as one of the first explicit applications of Gibbs sampling to SUR and to the multivariate prediction problem — appearing the same year as Gelfand and Smith's foundational Gibbs work and predating Chib-Greenberg (1995b). The first-order approximation is underappreciated: it yields a closed-form multivariate normal predictive density at far lower computational cost and performs comparably to Gibbs in all reported simulations. The paper's limitations are explicit: both approximations are unimodal regardless of the true predictive density's shape, and no theoretical accuracy guarantee exists for non-degenerate SUR data sets. The Zellner-Ando (2010) Direct Monte Carlo approach later superseded Gibbs for SUR posterior inference (convergence failures at high dimensions), but Percy's Gibbs scheme remains valid specifically for the predictive density problem.