Eckart-Young (1936) The Approximation of One Matrix by Another of Lower Rank

Summary

Eckart and Young solve the problem of approximating a given matrix by another of lower rank in the least-squares sense — the mathematical core of factor theory, where an $n\times N$ score matrix is postulated to be well approximated by a matrix of rank $r<\min(n,N)$. They show the least-squares problem cannot be attacked by naive normal equations (the entries of the approximant are not free), but becomes simple once the matrices are written in a canonic form (the singular value decomposition): the optimal rank-$r$ approximation keeps the $r$ largest canonic components and discards the rest. A solution always exists and is usually unique.

Key Claims

• Best low-rank approximation = truncated SVD. Writing $a = \sum_k s_k\, u_k v_k'$ in canonic (singular-value) form, the rank-$r$ matrix $\beta$ minimizing the sum of squared element-wise errors $\|a-\beta\|^2$ is the truncation $\beta = \sum_{k\le r} s_k\, u_k v_k'$ that retains the $r$ largest singular values.
• Existence and uniqueness. The problem always has a solution, and it is unique except in the degenerate case of tied singular values at the truncation boundary.
• Factor-theory motivation. The result formalizes the fundamental postulate of factor analysis: a data (score) matrix is adequately represented by a small number of common factors — i.e., by a low-rank matrix — with the residual treated as approximation error.
• Canonic form is the key device. Because the elements of the approximating matrix are dependent, the normal equations are not immediate; expressing the matrices in canonic (SVD) form decouples the problem and yields the closed-form solution.

Concepts Introduced or Extended

• Low-Rank Approximation
• Factor Model

Entities Mentioned

• Carl Eckart
• Gale Young

Quotes

"It is a fundamental postulate of factor theory that the resulting $n\times N$ score matrix … can be adequately approximated by another matrix … whose rank $r$ is less than the smaller of $n$ or $N$."

My Take

One of those results that quietly underlies half of applied multivariate statistics. The Eckart-Young theorem is the reason principal components, factor models, and reduced-rank regressions all "work": truncating the SVD is not an ad-hoc heuristic but the provably optimal least-squares low-rank fit. For this wiki's concerns it is the mathematical bedrock beneath factor models and dynamic factor models, PCA-based diffusion-index forecasting, and reduced-rank structures like cointegration. The 1936 framing in the language of factor theory (rather than linear algebra) is a nice reminder that the SVD entered statistics through psychometrics; Johnson (1963) later supplied the general proof the theorem had been missing.