DR-SNE: Density-Regularized Stochastic Neighbor Embedding
Maksim Kazanskii · 2026
Read DR-SNE: Density-Regularized Stochastic Neighbor Embedding by Maksim Kazanskii (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-02060.txt.
Introduction
Maksim Kazanskii extends stochastic neighbor embedding, a method that maps high-dimensional data to a low-dimensional picture while keeping neighbours close. Such maps can badly distort how dense points are in a local area. Kazanskii adds a regulariser that aligns a profile of local concentration, based on inverse neighbourhood scale, between the source and the map.
The new method, DR-SNE, keeps that relative concentration while staying unchanged under global rescaling of the map. Across real and synthetic data, it generally improved concentration preservation with controlled loss of neighbourhood fidelity. Tests show a steady trade-off as the regulariser grows stronger. The work suits cases where local density itself carries meaning.
Catalog record
| CREATOR | Maksim Kazanskii |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 75 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.02060
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DR-SNE: Density-Regularized Stochastic Neighbor Embedding Maksim Kazanskii Abstract Dimensionality reduction methods such as t-SNE are designed to preserve local neighborhood structure but do not explicitly account for how probability mass is distributed, often leading to distortions of data density. We reformulate dimensionality reduction as the joint alignment of two components: (i) conditional structure, capturing local relationships, and (ii) relative density structure, captured via local density statistics. Based on this perspective, we introduce Density-Regularized SNE (DR-SNE), which augments the stochastic neighbor embedding objective with a density regularization term derived from normalized log-density estimates. Unlike prior approaches such as DensMAP and DenSNE, which rely on local scale consistency, DR-SNE directly aligns normalized density estimates, providing a simple and scale-invariant mechanism for preserving relative density variations. Empirically, DR-SNE improves density preservation while maintaining competitive neighborhood fidelity, and yields gains on density-sensitive tasks such as anomaly detection across multiple datasets. These results suggest that incorporating density information complements geometry-focused objectives in dimensionality reduction. 1 Introduction Dimensionality reduction is typically formulated as a geometric problem: given high-dimensional data, the goal is to preserve local neighborhood relationships in a lower-dimensional embedding. Methods such as t-SNE achieve this by aligning conditional similarity distributions, effectively preserving local structure while allowing substantial freedom in global organization. However, this formulation implicitly ignores a fundamental aspect of the data: how probability mass is distributed locally across the dataset. In practice, dimensionality reduction methods severely distort density, expanding dense regions and compressing sparse ones, leading to embeddings that misrepresent the underlying data distribution. This limitation has been widely observed in practice [ 27 ] and analyzed in related theoretical work [ 10 ] , yet it is typically treated as a side effect rather than a core modeling issue. In this work, we argue that this limitation arises from an incomplete objective. We propose that dimensionality reduction should be understood as the joint alignment of two components of a data distribution: conditional structure (local relationships) and relative density structure (variation of probability mass via local density statistics). This perspective can be formalized via a decomposition of the Kullback–Leibler divergence between joint distributions into conditional and marginal components, where standard SNE objectives optimize only the conditional term. In this work, we introduce a complementary objective that aligns spatial density statistics via normalized local density estimates. Existing methods primarily optimize the first component, while leaving the second unconstrained. This leads to embeddings that are locally faithful but globally misleading in terms of density. While this decomposition is standard in information theory, its implications for dimensionality reduction have not been explicitly exploited. We use it to motivate a separation between conditional structure and marginal density. We introduce Density-Regularized SNE (DR-SNE), a simple and principled extension of the stochastic neighbor framework that explicitly incorporates a local approximation to marginal density alignment via normalized density estimates. By augmenting the standard objective with a density consistency term based on normalized log-density estimates, DR-SNE directly penalizes local volume distortion, yielding a scale-invariant and interpretable objective. Unlike prior density-aware methods such as DensMAP and DenSNE, which enforce consistency of local neighborhood scales (e.g., radii or bandwidths), the proposed approach explicitly aligns globally normalized density estimates . This formulation preserves relative density structure across the dataset via normalized local density estimates, rather than relying on indirect scale-based proxies. 2 Related Work Classical dimensionality reduction methods such as Principal Component Analysis (PCA) [ 3 ] and manifold learning approaches including Isomap [ 21 ] and Locally Linear Embedding (LLE) [ 18 ] focus on preserving global or local geometric structure, but do not explicitly address density preservation. t -distributed stochastic neighbor embedding (t-SNE) [ 22 ] is widely used due to its strong local neighborhood preservation and ability to reveal cluster structure, with several extensions improving scalability and optimization [ 9 ] . However, t-SNE is known to distort global geometry and can distort variations in data density, often producing embeddings with misleading cluster sizes and spatial relationships [ 27 , 10 ] . Uniform Manifold Approximation and Projection (UMAP) [ 12 ] provides an alternative graph-based formulation that often yields improved global organization. Other recent dimensionality reduction methods such as TriMap [ 1 ] and PaCMAP [ 26 ] aim to better preserve global structure through alternative objective formulations. Building on this, density-aware variants such as DensMAP and DenSNE [ 13 ] incorporate density information by encouraging consistency between local neighborhood scales (e.g., radii or bandwidths) across spaces. While these approaches improve qualitative density representation, they rely on scale-based proxies that are only indirectly related to local sampling density and can be sensitive to embedding scale and neighborhood definition. While kNN-based density estimates are themselves local approximations, they directly estimate relative sampling density and become scale-invariant after normalization, unlike radius-based constraints which depend on embedding scale and neighborhood definition. In contrast, we propose a formulation based on explicit alignment of density-derived local statistics . We estimate local densities using k -nearest neighbor statistics in both spaces and minimize discrepancies between normalized log-density estimates, yielding a scale-invariant objective that explicitly aligns relative density variations. This provides a more direct and interpretable mechanism for controlling density preservation than proxy-based approaches. Our formulation is also related to distribution alignment approaches such as optimal transport, which aim to match probability measures across spaces [ 16 , 15 ] . However, these methods typically focus on global distribution matching, whereas DR-SNE enforces alignment of local density statistics compatible with stochastic neighbor embedding. We note connections to recent work leveraging embedding-space density for tasks such as out-of-distribution detection [ 4 ] , where density can serve as a signal for confidence and distributional shift. While these approaches do not explicitly address density preservation in dimensionality reduction, they highlight the importance of density as a meaningful statistical quantity in learned representations. Overall, our approach provides a simple and principled mechanism for preserving relative density variations , enabling an explicit trade-off between neighborhood fidelity and density consistency. 3 Methods 3.1 Density-Regularized Stochastic Neighbor Embedding Given data X=\{x_{i}\}_{i=1}^{n}\subset\mathbb{R}^{D} , we learn an embedding Z=\{z_{i}\}_{i=1}^{n}\subset\mathbb{R}^{d} ( d\ll D ) that preserves local structure and relative density. We define high-dimensional affinities as in [ 22 ] : p_{ij}=\frac{p_{j|i}+p_{i|j}}{2n},\quad p_{j|i}\propto\exp(-\beta_{i}\|x_{i}-x_{j}\|^{2}), (1) with \beta_{i} set by perplexity. Low-dimensional similarities use a Student- t kernel: q_{ij}=\frac{(1+\|z_{i}-z_{j}\|^{2})^{-1}}{\sum_{k\neq l}(1+\|z_{k}-z_{l}\|^{2})^{-1}}. (2) The SNE objective can be interpreted as minimizing a KL divergence between joint distributions P_{ij}=P_{i}P_{j|i} and Q_{ij}=Q_{i}Q_{j|i} : \mathrm{KL}(P_{ij}\,\|\,Q_{ij})=\mathrm{KL}(P_{i}\,\|\,Q_{i})+\mathbb{E}_{i}\left[\mathrm{KL}(P_{j|i}\,\|\,Q_{j|i})\right]. (3) Since SNE fixes P_{i}=Q_{i}=1/n , the marginal term vanishes, and the objective reduces to aligning conditional distributions only. We optimize a sparse KL objective over k -nearest neighbors: \mathcal{L}_{\mathrm{KL}}=\frac{1}{n}\sum_{i=1}^{n}\sum_{j\in\mathcal{N}_{k}(i)}p_{ij}\log\frac{p_{ij}}{q_{ij}}. (4) To incorporate spatial density information not captured by the SNE objective, we estimate local densities via k -NN distances: \rho_{i}^{\mathrm{high}}=\frac{k}{\sum_{j\in\mathcal{N}_{k}(i)}\|x_{i}-x_{j}\|},\quad\rho_{i}^{\mathrm{low}}=\frac{k}{\sum_{j\in\mathcal{N}_{k}(i)}\|z_{i}-z_{j}\|}, (5) and normalize them to unit mean. We then minimize a log-density discrepancy: \mathcal{L}_{\mathrm{dens}}=\frac{1}{n}\sum_{i=1}^{n}\left(\log\rho_{i}^{\mathrm{high}}-\log\rho_{i}^{\mathrm{low}}\right)^{2}. (6) The final objective is \mathcal{L}=\begin{cases}\mathcal{L}_{\mathrm{KL}}^{\mathrm{early}},&t<T_{\mathrm{warmup}},\\ \mathcal{L}_{\mathrm{KL}}+\lambda\,\mathcal{L}_{\mathrm{dens}},&\text{otherwise},\end{cases} (7) where early exaggeration is applied during a warm-up phase. Optimization is performed with Adam and gradient clipping, with Gaussian initialization. The density regularization term is theoretically motivated via local volume distortion and marginal alignment (Appendix A ). 3.2 Comparison Methods We compare DR-SNE against widely used dimensionality reduction methods, including both classical approaches and density-aware extensions: t-SNE [ 22 ] , which preserves local neighborhood