Autoencoders — Nonlinear Compression & the In-Sample Anomaly Trap

Compression Learned by Backpropagation

An autoencoder is a network trained to copy its input to its output through a bottleneck. An encoder compresses the input to a low-dimensional latent code, a decoder rebuilds the input from that code, and the network minimises reconstruction error. Because the information has to squeeze through the narrow layer, the network is forced to learn an efficient representation — dimensionality reduction, learned by backpropagation rather than solved in closed form.

The linear anchor, stated precisely

The link to PCA (principal component analysis) is exact, and worth stating precisely because two claims are easy to run together. A linear autoencoder — no activations, squared-error loss — recovers PCA's subspace: here the reconstruction MSE (mean squared error: the average squared gap between the original and its rebuild) agrees to 6×10⁻⁸ relative, and the principal angles between the encoder's subspace and PCA's top-2 subspace are 0.000°. What it does not recover is PCA's axes. The encoder's rows sit at |cos| of 0.66 and 0.75 against PC1 and PC2, and are not even orthogonal to each other. Nothing in the squared-error loss prefers one basis of the right subspace over another, so gradient descent has no reason to return the ordered, orthogonal one.

Everything an autoencoder adds beyond PCA therefore comes from nonlinearity. On Fashion-MNIST at the same 32-dimensional bottleneck, a ReLU encoder-decoder reaches 14% lower test error than PCA-32 — both fit on the training split and scored on the test split — with visibly sharper edges, because it can bend around the image manifold instead of fitting a flat subspace.

The 2-D latent space

Squeezed to a 2-D bottleneck, the latent space can be plotted directly, and "separates the classes more cleanly than PCA" is scored rather than eyeballed. Reading the held-out class label off each embedding with a 15-nearest-neighbour classifier gives 0.673 for the autoencoder against 0.525 for PCA, on ten classes where chance is 0.100 — and neither map ever saw a label, which only enters to score the result afterwards.

Reconstruction Error as an Anomaly Score

The last application is the one with a trap in it. Reconstruction error makes an unsupervised anomaly score: a network that has learned what normal looks like should rebuild normal inputs well and unusual ones badly. Applied to the cross-section of 48 stock returns, each week a 48-dimensional vector, the obvious implementation trains on every week and then scores those same weeks.

That network has 1,748 parameters and 312 weeks to fit. With capacity to spare it does not learn "normal" — it learns these observations, the extreme ones included. An outlier the network has memorised reconstructs beautifully, and therefore scores as maximally normal. The anomaly ranking inverts precisely where the method is supposed to work, and the COVID crash makes the point unmissable.

Nothing about that week changed between the two rows — only whether the model had already seen it. The two rankings correlate at Spearman 0.83 overall and share none of their top five, which is the only part of a ranking anyone uses. The signature is visible across the whole sample: the ten largest-magnitude weeks average the 44th percentile of in-sample error but the 91st out-of-sample, so memorisation bites hardest exactly where the data is most extreme — on the points the detector exists to find. Refitting under five random seeds moves the COVID week anywhere from the 0th to the 36th percentile.

What it finds once it is done properly

Scored out-of-sample the detector does work, and what it finds is worth stating plainly rather than dressing up. It agrees with the linear baseline at Spearman 0.91 against rank-4 PCA, correlates 0.71 with sheer weekly magnitude, and the weeks it flags are the March 2020 crash. On this cross-section the nonlinearity is not buying a signal a rank-4 linear projection does not already give for none of the cost. That is a useful thing to have established rather than assumed — and it is only visible once the score is computed on data the model was not fitted to.

Where this sits

The linear anchor is the subject of PCA, SVD & Factor Analysis, on the same 48-stock cross-section — an autoencoder is its nonlinear generalisation, and a linear one collapses back onto it. The encoder/decoder machinery, backpropagation and Adam come from Multilayer Perceptrons & Backpropagation. The anomaly application sits beside Realized Volatility, which measures the magnitude of market moves directly — the comparison above is what shows the two are measuring much the same thing here. The variational autoencoder would add a prior on the latent code and a full generative model, the Bayesian extension of everything on this page.

Notebook

Downloads

Fashion-MNIST is downloaded by torchvision when the notebook runs. The encoder and decoder are defined inline in the notebook rather than in a module, since the architectures differ in each section.

References

• Baldi, P. & Hornik, K. (1989). Neural networks and principal component analysis: learning from examples without local minima. Neural Networks 2(1), 53–58. — the linear autoencoder recovers PCA's subspace
• Hinton, G. E. & Salakhutdinov, R. R. (2006). Reducing the dimensionality of data with neural networks. Science 313(5786), 504–507. — deep nonlinear autoencoders
• Kingma, D. P. & Welling, M. (2014). Auto-encoding variational Bayes. ICLR. — the probabilistic extension
• Chandola, V., Banerjee, A. & Kumar, V. (2009). Anomaly detection: a survey. ACM Computing Surveys 41(3), 1–58. — reconstruction error as a novelty score, and its failure modes
• Xiao, H., Rasul, K. & Vollgraf, R. (2017). Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. arXiv:1708.07747. — the image data