Tiao-Zellner (1964) Bayes's Theorem and the Use of Prior Knowledge in Regression Analysis

Summary

Tiao and Zellner show that when two regression samples share the same slope vector $\beta$ but have independent unknown error variances, the Bayesian posterior of $\beta$ under noninformative priors on $\beta$ , $\log\sigma_1$ , and $\log\sigma_2$ is a product of two multivariate t-distributions — the "double-t" distribution. The normalizing constant is a $p$ -dimensional integral with no closed form, so the authors develop an asymptotic expansion in powers of $\nu_1^{-1}$ and $\nu_2^{-1}$ that converts the double-t into a multivariate normal times a computable power-series correction. The paper provides the Bayesian counterpart to Zellner (1962) for the case of diagonal, heterogeneous error covariance; the first joint work of Tiao and Zellner.

Key Claims

Noninformative-prior single-sample result (§2, eq. 2.7): Under $p(\beta) \propto 1$ and $p(\sigma) \propto 1/\sigma$ (the Jeffreys-Savage prior), the marginal posterior of $\beta$ is multivariate t with $\nu = T - p$ degrees of freedom: $p(\beta|y) \propto \left\{1 + Q(\beta,\hat\beta,Z)/(\nu s^2)\right\}^{-\frac{1}{2}(\nu+p)}$ . Result first noted by Savage (1961).
Equal-variance case (eq. 2.9): When $\sigma_1 = \sigma_2 = \sigma$ (Raiffa-Schlaifer), pooling is trivial — the posterior is still multivariate t with $\nu = T_1 + T_2 - p$ and precision matrix $Z = Z_1 + Z_2$ .
Double-t distribution (eq. 2.12): When $\sigma_1$ and $\sigma_2$ are treated as independent unknown parameters with independent log-uniform priors, $p(\beta|y_1,y_2) \propto \left\{1 + \frac{Q(\beta,\hat\beta_1,Z_1)}{\nu_1 s_1^2}\right\}^{-\frac{1}{2}(\nu_1+p)} \left\{1 + \frac{Q(\beta,\hat\beta_2,Z_2)}{\nu_2 s_2^2}\right\}^{-\frac{1}{2}(\nu_2+p)}$ , the product of two multivariate t-kernels. The normalizing constant $k$ is a $p$ -dimensional integral; for $p > 1$ it has no closed form.
K-sample generalization (eq. 2.14): $K$ regression samples with independent unknown variances $\sigma_i^2$ yield $p(\beta|y) \propto \prod_i \{1 + Q(\beta,\hat\beta_i,Z_i)/(\nu_i s_i^2)\}^{-\frac{1}{2}(\nu_i+p)}$ , a product of $K$ multivariate t-factors.
Theil (1963) connection: When $\sigma_1^2$ is known and $\nu_1 \to \infty$ (first sample treated as a degenerate normal prior), the posterior mean converges to Theil's mixed estimator $\bar\beta = D^{-1}(M_1\hat\beta_1 + M_2\hat\beta_2)$ with $D = M_1 + M_2$ , $M_i = Z_i/s_i^2$ . Precision-weighted pooling emerges naturally from the Bayesian framework.
Behrens-Fisher connection: For p = 1, the univariate double-t agrees exactly with Fisher's (1961b) fiducial solution to the Behrens-Fisher problem of comparing two normal means with unequal unknown variances.
Asymptotic expansion (§3, eq. 3.9): Expanding each t-factor in powers of $\nu_i^{-1}$ via $\{1 + Q_i/\nu_i\}^{-\frac{1}{2}(\nu_i+p)} = e^{-\frac{1}{2}Q_i}\sum_{j=0}^\infty p_j\nu_i^{-j}$ , the double-t becomes a multivariate normal times a computable power series: $p(\beta|y_1,y_2) = W^{-1}\frac{|D|^{\frac{1}{2}}}{(2\pi)^{\frac{1}{2}p}}\exp\{-\tfrac{1}{2}Q(\beta,\bar\beta,D)\}\sum_{i=0}^\infty\sum_{j=0}^\infty d_{ij}\nu_1^{-i}\nu_2^{-j}$ . The coefficients $d_{ij}$ are computed from mixed cumulants of $Q_1$ and $Q_2$ ; the normalizing constant $W$ is evaluated term by term via the moment-cumulant inversion formula (Cook 1951).
Marginal posteriors (§3.2, eq. 3.14): Integrating out $\beta_{(m)}$ from the asymptotic joint expression yields an analogous expansion for the marginal posterior of any subset $\beta_{(l)}$ , with $\omega$ -cumulants replacing $\kappa$ -cumulants.
Grunfeld investment application (§4): GE and Westinghouse, 1935–54 ( $T = 20$ , $\nu_1 = \nu_2 = 17$ ). Slope coefficients $\beta_1$ (shares value) and $\beta_2$ (capital stock) have joint posterior mode at (0.0373, 0.1446), negatively correlated, nearly bivariate normal (large $\nu$ ). Finite-sample corrections raise variance by ~6% relative to normal approximation; mean barely shifts.

Concepts Introduced or Extended

Entities Mentioned

Quotes

"The use of Bayes's theorem in statistical inference has recently been reconsidered in the works of Jeffreys (1957, 1961), Savage (1959, 1961, 1962), Raiffa & Schlaifer (1961), Box & Tiao (1962, 1964) and others." (p. 219)

"We note that the result in (2.12) is of course applicable to the problem of making inferences about a population mean when samples are drawn from two normal populations with common mean and unequal variances." (p. 221)

"It can easily be shown that the marginal distribution of a subset of the elements of $\beta$ is also in the same form as in (2.7). In particular, the marginal distribution of a single element $\beta^i$ can be expressed in terms of a univariate t-distribution with $T - p$ degrees of freedom." (p. 220)

My Take

The double-t result is conceptually important but its asymptotic approximation technique has been fully superseded by Markov chain Monte Carlo (MCMC), which makes the normalizing-constant problem trivial. The lasting contributions are: (i) the identification of the double-t as the correct Bayesian posterior under independent heterogeneous variances — a form that does not appear in the textbook conjugate framework; (ii) the demonstration that Theil's mixed estimator is a Bayesian limiting case, grounding it in a probabilistic framework; and (iii) the Grunfeld investment application, which became the canonical seemingly unrelated regression (SUR) test dataset used by Zellner (1962), this paper, and most subsequent SUR methodology papers. The paper also pioneered the Behrens-Fisher connection that recurs in later Bayesian regression literature. As the first Tiao-Zellner collaboration, it launched a research program culminating in the Box-Tiao (1973) textbook.