Some Theoretical Limitations of t-SNE
Rupert Li, Elchanan Mossel · 2026
Read Some Theoretical Limitations of t-SNE by Rupert Li, Elchanan Mossel (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-13295.txt.
Introduction
Li and Mossel examine what t-SNE, a popular tool for reducing dimensions and visualizing data, necessarily loses. Every dimension-reduction method can discard important features, and the authors build a mathematical framework to pin down that loss for t-SNE specifically. Across several scenarios they prove results showing how key features of the data vanish when t-SNE compresses it. The work gives a rigorous basis for judging when t-SNE pictures can mislead and which structures they cannot be trusted to preserve.
Catalog record
| CREATOR | Rupert Li, Elchanan Mossel |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59377 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.13295
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Some Theoretical Limitations of t-SNE Rupert Li, Elchanan Mossel Abstract . t-SNE has gained popularity as a dimension reduction technique, especially for visualizing data. It is well-known that all dimension reduction techniques may lose important features of the data. We provide a mathematical framework for understanding this loss for t-SNE by establishing a number of results in different scenarios showing how important features of data are lost by using t-SNE. 1. Introduction Dimension reduction is often a central step in the visualization and analysis of high-dimensional data. In particular, our inability to visualize in more than 2 or 3 dimensions often leads researchers to present their data in 2 or 3 dimensions in order to identify qualitative structure in the data, such as clusters. Dimension reduction is also often used as part of computational pipelines in data analysis. Many dimension reduction techniques exist, but t-distributed stochastic neighborhood embedding (t-SNE), introduced by van der Maaten and Hinton [ 7 ] in 2008, has become perhaps the most popular for data visualization. t-SNE is a nonlinear algorithm, and this nonlinearity enables it to succeed in many settings where more classical, linear techniques spectacularly fail. For example, see Figure 1 for a comparison of principal component analysis (PCA) and t-SNE on data generated from a high-dimensional simplex. (a) First two principal components (b) t-SNE output using standard parameters Figure 1 . t-SNE and PCA on points clustered around the 10 vertices of a regular 9-simplex. Here, the vertices are the ten elementary basis vectors in \mathbb{R}^{10} , and each cluster has 100 points sampled i.i.d. from a spherical Gaussian with standard deviation 0.2 in each direction. We refer readers to Section 2 for a detailed description of the t-SNE algorithm. In essence, t-SNE computes a certain similarity measure between all pairs of points in the high-dimensional dataset, and then considers a candidate mapping of this dataset to points in a lower-dimensional space, typically 2- or 3-dimensional. It then computes a different similarity measure between these low-dimensional points, and attempts to arrange these points so that these two similarity measures are as close to each other as possible. It does so by minimizing the KL-divergence between these two matrices of similarity measures. We first discuss our results and then how they relate to some prior theoretical analysis of t-SNE [ 6 , 3 , 1 , 2 ] . 1.1. Our Results One of the intuitive reasons for the challenge in dimension reduction is that higher dimensions have more “space” to pack data. This may result in data points that were far away in the original data while their low dimensional projections are close together. More formally, we have the following standard result for any dimension reduction procedure: Proposition 1.1 . Let X consist of n points sampled uniformly from \mathbb{S}^{d-1} , with n\leq 2^{d} . Let g=g(n) be any sequence tending to infinity. With high probability as n,d\to\infty , for any map f:X\to\mathbb{R}^{2} with \left\lVert f(x)\right\rVert\leq B for all x\in X , for 1-O(g^{-2}) fraction of the points x\in X , there exists another point y\in X such that \left\lVert x-y\right\rVert\geq 0.2 yet \left\lVert f(x)-f(y)\right\rVert\leq 3B\frac{g}{\sqrt{n}} . We now turn our attention specifically to t-SNE. The goal of the t-SNE procedure is to map the data points to low dimension so that their normalized pairwise distances in the high dimensions P is close in KL-divergence to the normalized pairwise distances Q of the images with a different kernel. We next show that even for simple collections of points, it is impossible to find a Q such that D(P\|Q) is small. Throughout this paper unless otherwise stated, we assume the t-SNE output is two-dimensional, though all our theoretical results straightforwardly extend to any fixed output dimension. Proposition 1.2 . Let X=\left\{e_{1},\dots,e_{n},2e_{n+1},\dots,2e_{2n}\right\}\subset\mathbb{R}^{2n} be an orthogonal frame with half its points doubled in magnitude. Assume fixed bandwidths \sigma_{i}=\sigma . Then there exists a constant c>0 such that for all sufficiently large n and any embedding Y , the t-SNE objective satisfies D(P\|Q)\geq c . It is well-known that the number of equidistant points in dimension d is at most d+1 . It is natural to ask how t-SNE reduces the dimension of such collections of points. The following proposition provides an answer to this question in terms of the global optimizer of the t-SNE objective: Proposition 1.3 . Consider t-SNE on a dataset X of n points with s -dimensional output such that n>s+1 . If X is an equidistant set, i.e., all pairwise distances are equal, or the \sigma_{i} are set using a perplexity value p and we have p\geq n-1 , then any global minimizer Y of the t-SNE objective D(P\|Q) has all points y_{1},y_{2},\dots,y_{n} coinciding at the same point. Arguably the results above show a failure mode of t-SNE. However, it could be argued that it is a tailored example that does not represent the generic situation. The result below exhibits the same phenomenon in a generic setting. Theorem 1.4 . Consider an asymptotic regime as n,d\to\infty , where the bandwidths \sigma_{1},\sigma_{2},\dots,\sigma_{n} all equal some fixed constant \sigma . Fix constant \delta\in(0,1/2) and r=\omega\left(d^{-1/6+\delta/3}\right) . For some n=e^{\Theta\left(d^{2\delta}\right)} , letting x_{1},x_{2},\dots,x_{n} be uniformly random samples on the unit sphere \mathbb{S}^{d-1} , with probability 1-e^{-\Omega\left(d^{2\delta}\right)} , any global minimizer Y of the t-SNE objective D(P\|Q) has 1-r fraction of Y within some (open) ball of radius r and has all of Y contained within some ball of radius O(\sqrt{r}) . Informally, the theorem states that the t-SNE objective will be optimal only when it maps all the points to a tiny neighborhood. Here, note that larger \delta lets us have more points on the sphere, and gives us a higher probability of success; however, it gives a weaker result to the extent that r is bigger, so fewer of the points are within the radius- r ball and this ball as well as the radius- O(\sqrt{r}) ball are both bigger. This could be considered a failure mode of t-SNE: the dataset X is too high-dimensional, so that to capture the structure of many points in X being approximately equidistant to each other in different directions, the best solution for t-SNE is to put all points at essentially the same point, yielding an uninformative dimension reduction. Additionally, if the visualization is at such small scale, there may be additional concerns regarding numerical stability of the algorithm. Moreover, we can apply Proposition 1.1 to the setting of Theorem 1.4 with zoom-in. For this, let n=e^{\Theta\left(d^{2\delta}\right)} from Theorem 1.4 , which is eventually less than 2^{d} , and let B be the minimum possible radius of a closed ball containing all of Y , which by Theorem 1.4 is O\left(\sqrt{r}\right)=\omega\left(d^{-1/12+\delta/6}\right) with high probability, so B can be taken to be O\left(d^{-\varepsilon}\right) for any constant \varepsilon>0 satisfying \varepsilon<\frac{1}{12}-\frac{\delta}{6} . Let g slowly tend to infinity, say g=d . Note then that \frac{g}{\sqrt{n}}=e^{-\Theta\left(d^{2\delta}\right)} . Then Proposition 1.1 implies that with high probability, even when we zoom in to the ball of radius B , so at the proper relative scale for Y , we have 1-o(1) fraction of the points have another point within relative distance \frac{3g}{\sqrt{n}}=e^{-\Theta\left(d^{2\delta}\right)}=o(1) , despite all pairs of points in X having constant order distance,