t-SNE

Definition

t-SNE (t-Distributed Stochastic Neighbor Embedding) is a nonlinear dimensionality-reduction technique for visualizing high-dimensional data by placing each point in a 2- or 3-dimensional map so that similar points are modelled by nearby map points and dissimilar points by distant ones (van der Maaten-Hinton 2008). It is a variant of Stochastic Neighbor Embedding (SNE; Hinton-Roweis 2002) that is easier to optimize and reduces the tendency to crowd points at the centre of the map.

Key Ideas

• Similarities as probabilities. In the high-dimensional space, pairwise similarity of points $i,j$ is a conditional probability $p_{j\mid i}$ from a Gaussian kernel centred on $i$; these are symmetrized to a joint distribution $p_{ij}$. The Gaussian bandwidth per point is set so the induced perplexity (a smooth measure of the effective number of neighbours) matches a user-chosen value.
• Heavy-tailed map distribution. In the low-dimensional map, similarities $q_{ij}$ use a Student-$t$ distribution with one degree of freedom (a Cauchy), $q_{ij}\propto(1+\lVert y_i-y_j\rVert^2)^{-1}$. This is the defining change from SNE.
• Cost = KL divergence. The map $\{y_i\}$ is found by minimizing the Kullback-Leibler divergence $\mathrm{KL}(P\Vert Q)=\sum_{i\ne j}p_{ij}\log\frac{p_{ij}}{q_{ij}}$ by gradient descent; the symmetric formulation gives simpler gradients.
• The crowding problem. In high dimensions the volume available at moderate distance is huge and cannot be reproduced in 2-D, so faithful methods crush moderately-distant points together. The heavy Student-$t$ tail lets moderately-distant high-D points map to larger low-D distances, relieving the crowding and easing optimization.
• Multi-scale, multi-manifold. t-SNE reveals structure at many scales in a single map — valuable when data lie on several related low-dimensional manifolds (e.g. images of objects across classes and viewpoints). Random walks on neighbourhood graphs extend it to very large data sets.

Why It Matters

• The default high-dimensional visualization. t-SNE became the standard tool for eyeballing cluster structure in embeddings — word/document vectors, single-cell genomics, neural-network representations. Domain examples abound: e.g. Cieslak et al. (2019, Marine Genomics) apply t-SNE to reduce and visualize RNA-Seq transcriptomes of copepods across developmental stages and experimental conditions — a typical "omics" use of the method outside its ML origins.
• Beats classical embeddings for visualization. On many data sets its maps are markedly clearer than Sammon mapping, Isomap, and Locally Linear Embedding, which tend to collapse or crowd multi-scale structure.
• A clean probabilistic recipe. Casting neighbour preservation as matching two similarity distributions under a KL objective is a reusable idea (later refined by UMAP and others).

Open Questions

• Not distance-preserving. Inter-cluster distances and cluster sizes in a t-SNE map are not meaningful; only local neighbourhood structure is trustworthy — easy to over-interpret.
• Sensitivity to perplexity and to the random initialization/optimization; different runs can look different.
• Primarily a visualization method (2–3 D), not general-purpose dimensionality reduction for downstream modelling; it does not provide an explicit out-of-sample mapping.

Related

• van der Maaten-Hinton (2008) Visualizing Data using t-SNE
• Laurens van der Maaten
• Geoffrey Hinton