← All examples

Machine Learning: Unsupervised Learning

Unsupervised methods are harder to audit than supervised ones for a simple reason: there is no held-out label to be wrong against. A clustering always returns clusters, an embedding always returns a picture, and a low-rank approximation always returns a matrix — so the question is never whether the method ran but whether the structure it reports is in the data or in the algorithm.

That makes the diagnostics the substance rather than the garnish. A rank-k reconstruction has to be compared against the right object, and against a baseline; a cluster count has to survive resampling before it means anything; an embedding's apparent groups have to be checked against what the distances actually preserve. Each example here is built around the check as much as the method.

The finance thread runs through all four: the same 48-stock cross-section and volatility series used elsewhere in the collection, so what these methods find can be set against what the supervised and Bayesian treatments found on identical data.

Two different jobs, both called unsupervised. The four examples split cleanly and the split is worth having in mind, because the methods are not alternatives to each other. The first groups observations: given 3,459 trading days, decide which of them belong together. The other three compress features: given 48 stocks measured every week, find a handful of numbers that carry most of what all 48 were doing. Grouping rows, compressing columns. A clustering answers “which days are alike?”; a decomposition answers “what are these 48 series really made of?” The reason they sit together is the shared difficulty rather than a shared technique — in both cases the method returns an answer whether or not one exists, and no held-out label is available to say which.

Grouping: centroids, mixtures, and a fitting procedure that can settle in the wrong place. k-means picks k centres, assigns every point to its nearest, moves each centre to the mean of what it caught, and repeats. Assignments are hard — a point belongs to exactly one cluster — and because distance to a centre is the only criterion, the clusters it can express are effectively spherical and similar in size. A Gaussian mixture relaxes both: each cluster gets a full mean and covariance, so it can be elongated and tilted, and assignments become soft, a probability of membership rather than a verdict. It is fitted by EM (expectation–maximisation), which alternates between computing those membership probabilities and re-estimating each cluster from the points weighted by them. EM is a local optimiser: it climbs to a nearby peak of the likelihood, and different starting points reach different peaks. That is why the from-scratch and reference implementations here agree on the likelihood to four figures and still disagree about which days belong to which cluster.

How many clusters — three answers that need not agree. Nothing in either algorithm chooses k; it is supplied. Three ways to choose it recur. The elbow plots the within-cluster spread against k and looks for where the curve stops falling steeply, the point past which extra clusters stop buying much. BIC — the Bayesian information criterion — scores a fitted model as its likelihood minus a penalty that grows with the number of parameters and with the amount of data, so it rewards fit while charging for complexity, and the k that minimises it is preferred. Both of those are computed on the same data the model was fitted to. The third is stability, which asks a different question entirely: refit the whole thing on two overlapping subsamples and check whether the days both fits saw are grouped the same way. Agreement is scored by the adjusted Rand index (ARI), which counts the pairs of points that two partitions agree to put together or apart, corrected so that chance agreement scores 0 and an identical partition scores 1. A cluster count that cannot reproduce itself on a resample is a property of the fit rather than of the data.

Compressing: components, loadings, and what is left over. PCA (principal component analysis) finds the direction along which the data varies most, then the next such direction at right angles to it, and so on — rewriting the data in a new set of axes ordered by how much variation each accounts for. That share is the explained-variance ratio, and it is what makes the compression measurable: if the first three axes carry 56% of the variation, the other 45 carry the rest between them. Each component is defined by its loadings, the weights it puts on the original variables, and those weights are readable — a component with all-positive loadings on 48 stocks is a market factor, one that is positive on energy and negative on technology is a sector bet. The computation is the SVD (singular value decomposition), a matrix factorisation that produces all of this in one step. Factor analysis looks similar and asks a different question: instead of finding directions of maximum variance, it models each variable as a few shared factors plus a term unique to that variable, which splits every stock's risk into a systematic part moving with everything else and an idiosyncratic part that diversifies away. PCA blurs that distinction; factor analysis is built on it.

Learned and nonlinear: bottlenecks and manifolds. An autoencoder is a network trained to reproduce its own input after forcing it through a narrow middle layer — the bottleneck. Whatever survives that squeeze is the latent code, and because the network is penalised for a poor reconstruction, the code has to retain whatever matters most. With no nonlinearities this is PCA in disguise, recovering the same subspace though not the same axes; with them it can follow a curved surface rather than a flat one. That curved surface is the manifold — the idea that data described by 784 pixel values may in truth vary along only a few dozen meaningful directions, which need not be straight lines. The gap between the linear and nonlinear versions is what measures whether curvature was worth the trouble.

Embeddings, and the circular scoring problem. t-SNE (t-distributed stochastic neighbour embedding) and UMAP (uniform manifold approximation and projection) exist to put high-dimensional data on a page for a human to look at. t-SNE turns distances into probabilities of being neighbours in both the original and the 2-D space, then moves the 2-D points until those two distributions match, measured by the Kullback–Leibler divergence — a non-symmetric measure of how far one probability distribution sits from another. UMAP builds a graph of nearest neighbours and lays it out to preserve that graph. Scoring the result is where the trap is. Silhouette asks how tight and well-separated the clusters look in the picture, which is precisely what both methods optimise — so it rewards the algorithm for succeeding at its own objective, whether or not the groups are real. The honest measures ask about fidelity to the original space instead: trustworthiness (are the points that ended up adjacent genuinely neighbours in the original data?), the share of each point's high-dimensional neighbours that survive the projection, and reading a held-out label off the map with a nearest-neighbour classifier. Spearman rank correlation, used here to check whether coarse arrangement is preserved, compares two orderings rather than two sets of values, so it registers whether distant pairs stayed distant without demanding the distances themselves be reproduced.