Zellner (1988) Optimal Information Processing and Bayes's Theorem

Summary

A 3-page American Statistician paper that derives Bayes's theorem as the unique optimal information processing rule (IPR) via a calculus-of-variations argument, independently of coherence, Dutch book, or Cox-consistency justifications. Statistical inference is reframed as an information processing problem with inputs (likelihood, prior) and outputs (postdata probability density function (pdf), predictive), and Bayes's rule is shown to be the only IPR satisfying the Information Conservation Principle — that output information exactly equals input information. The paper was published with four discussions (Jaynes, Hill, Bernardo, Kullback) and a reply.

Key Claims

Information measures (eqs. 2.2–2.5): Information in each pdf is $\int_\Theta \pi_p(\theta|D)\log[\cdot]\,d\theta$ — a posterior-weighted average of the log density. Information in the predictive $p(y|I)$ reduces to $\log p(y|I)$ .
Criterion functional (eq. 2.6): $\Delta[\pi_p] = I(\pi_p) - I(l) - I(\pi_a) + \log p(y|I)$ — the difference between output and input information. Equivalently (eq. 2.7): $\Delta[\pi_p] = 2\int\pi_p\log[\pi_p(\theta|D)/\pi_a(\theta|I) \cdot p(y|I)/f(y|\theta)]^{1/2}d\theta$ — a Kullback-Leibler (KL)–type divergence.
Optimal IPR: Minimizing $\Delta[\pi_p]$ subject to $\int\pi_p\,d\theta=1$ via Lagrange multiplier / calculus of variations yields $\pi_p^*(\theta|D) = c\,\pi_a(\theta|I)\,l(\theta|y)$ , i.e., Bayes's theorem, with $c^{-1} = p(y|I)$ .
100% efficiency: Substituting $\pi_p^*$ back into (2.6) gives $\Delta[\pi_p^*]=0$ — the Bayesian IPR conserves information exactly (ICP satisfied).
Links to maximum entropy: The negative entropy of $\pi_p$ (term $I(\pi_p)$ ) appears as part of the criterion; minimizing $\Delta[\pi_p]$ connects Bayes's theorem to max-entropy principles (Jaynes comment concurs).

Concepts Introduced or Extended

Entities Mentioned

Arnold Zellner

Quotes

"An optimal information processing rule was derived that is identical to Bayes's rule and is 100% informationally efficient."

"Jaynes mentions that the present analysis provides 'a close connection between entropy and Bayes's theorem'" — Zellner, Reply.

My Take

An elegant but narrow contribution. Kullback's comment is largely correct: once one recognizes the criterion functional (2.6) as a KL divergence between $\pi_p(\theta|D)$ and the Bayesian posterior $\pi_a(\theta|I)f(y|\theta)/p(y|I)$ , the optimality of Bayes's rule follows trivially. Zellner's defense — that the variational framework generalizes to dynamic and constrained settings where the solution is not Bayes's theorem — is the more interesting point. Bernardo's connection to proper scoring rules (his 1979 Annals of Statistics paper) and Jaynes's entropy perspective are the most useful contextualization for the wiki. Primarily of philosophical interest; does not directly inform time-series or econometric methodology.