dimensionality reduction

Dimensionality reduction is the family of techniques for projecting high-dimensional data down to a handful of dimensions — usually two or three — that a human can actually look at on a scatter plot. A BERT embedding is 768 numbers per piece of text, OpenAI's text-embedding-3-small produces 1,536, and nobody can visualize a 768-dimensional space directly; the classic demonstration dataset, MNIST's 60,000 handwritten digit images, is 784-dimensional (28 by 28 pixels flattened), and reducing it to two dimensions well enough to see ten visually distinct digit clusters is the standard benchmark every new method gets tested against. The three dominant tools for this — PCA, t-SNE, and UMAP — trade off speed, faithfulness, and what kind of structure they preserve in genuinely different ways.

PCA finds the directions of maximum variance and is the one technique here with a century-old mathematical pedigree. Karl Pearson described the underlying idea in 1901 and Harold Hotelling formalized principal component analysis in 1933, decades before any computer existed to run it — the method rotates the data onto new axes ordered by how much variance each one captures, so keeping the first handful of components keeps most of the information while discarding the rest. Because the transformation is linear and deterministic, running PCA twice on the same data gives the identical result every time, which is exactly what t-SNE and UMAP cannot promise, and why PCA is the standard first step — reducing 768 dimensions down to roughly 50 — before handing the result to a slower, non-linear method.

t-SNE optimizes for looking good locally, at the cost of meaning nothing globally. Laurens van der Maaten and Geoffrey Hinton's 2008 algorithm converts high-dimensional distances into probabilities, then arranges points in two dimensions to match those probabilities as closely as possible, which produces the famously clean, tight clusters that made t-SNE the go-to plot for "look, my embeddings separate by class." The catch is that the distance *between* two t-SNE clusters, and even their relative sizes, carry no reliable meaning — two clusters drawn close together in a t-SNE plot are not necessarily more similar than two drawn far apart — and the algorithm's default quadratic time complexity makes it impractical much past 10,000 points without subsampling.

UMAP has mostly replaced t-SNE as the default, and single-cell biology adopted it before most of machine learning did. McInnes, Healy, and Melville's 2018 paper builds a topological model of the data's neighborhood structure and optimizes a low-dimensional layout to match it, running substantially faster than t-SNE at scale and preserving more of the large-scale relationships between clusters, not just their internal tightness; this is why UMAP became the default projection method in Scanpy and other single-cell RNA sequencing toolkits years before it became the default choice for visualizing language-model embeddings, and why it now handles datasets of a million or more points where t-SNE would need to subsample first.

None of these three techniques should feed their output back into a downstream model, and that is not a minor caveat. PCA's reduced dimensions are sometimes safe to use as model inputs because the transformation is linear and well-understood, but t-SNE and UMAP projections are built purely for visualization — their 2D coordinates optimize for how a plot looks to a human eye, not for preserving the properties a classifier or regression model would need, and using them as features routinely produces models that look fine on the projection and fail on the original data. The job these tools do well is answering "what does my high-dimensional data actually look like," not "what should my model's input features be" — a distinct question that feature-engineering and feature-selection methods, not visualization projections, are built to answer.