Baldi and Hornik analyse learning in layered linear feed-forward neural networks trained by backpropagation on the usual quadratic error. Their main result is a complete description of the error landscape in terms of principal component analysis: the error has a unique minimum, corresponding to projection onto the subspace spanned by the leading principal vectors of the training-data covariance matrix, and all other critical points are saddle points (projections onto subspaces spanned by non-leading vectors). The auto-associative (autoencoder) case is treated in detail — a linear bottleneck network learns PCA — with implications for gradient-based learning.
"E has a unique minimum corresponding to the projection onto the subspace generated by the first principal vectors of a covariance matrix associated with the training patterns. All the additional critical points of E are saddle points."
A foundational bridge between neural networks and classical multivariate statistics: it says the simplest neural network — a linear autoencoder — is just PCA in disguise, and that its optimisation landscape, despite being non-convex in the weights, is benign (one global minimum up to symmetry, the rest saddles). That second point is the more prophetic one: decades later the "no bad local minima, just saddles" picture became a central theme in deep-learning theory, and this paper is its cleanest early instance. For this wiki it ties the Eckart-Young/PCA thread to the neural-network thread — the same optimal-low-rank object reached by SVD, by factor analysis, or by gradient descent on a linear autoencoder. The obvious caveat is linearity: the clean landscape result does not survive the nonlinear activations that make deep autoencoders expressive, though it remains the reference point people perturb from.