NOMAD Projection

Brandon Duderstadt, Zach Nussbaum, Laurens van der Maaten · 2025

Read NOMAD Projection by Brandon Duderstadt, Zach Nussbaum, Laurens van der Maaten (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-15511.txt.

Introduction

Duderstadt, Nussbaum, and van der Maaten introduce NOMAD Projection in a 2025 paper. The method visualizes large unstructured data sets by nonlinear dimensionality reduction. Older tools such as t-SNE and UMAP have not kept pace with the growth of AI data sets.

NOMAD runs on multiple GPUs during training, a first for this kind of method. The authors give theory that places it as an approximate upper bound on a known t-SNE loss. Tests show better performance and speed than leading methods. As a proof of scale, they built the first complete data map of multilingual Wikipedia. The work targets exploratory analysis for AI explainability.

Catalog record

CREATORBrandon Duderstadt, Zach Nussbaum, Laurens van der Maaten
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-15511

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

NOMAD Projection Brandon Duderstadt, Zach Nussbaum, Laurens van der Maaten Abstract The rapid adoption of generative AI has driven an explosion in the size of datasets consumed and produced by AI models. Traditional methods for unstructured data visualization, such as t-SNE and UMAP, have not kept up with the pace of dataset scaling. This presents a significant challenge for AI explainability, which relies on methods such as t-SNE and UMAP for exploratory data analysis. In this paper, we introduce Negative Or Mean Affinity Discrimination (NOMAD) Projection, the first method for unstructured data visualization via nonlinear dimensionality reduction that can run on multiple GPUs at train time. We provide theory that situates NOMAD Projection as an approximate upper bound on the InfoNC-t-SNE loss, and empirical results that demonstrate NOMAD Projection’s superior performance and speed profile compared to existing state-of-the-art methods. We demonstrate the scalability of NOMAD Projection by computing the first complete data map of Multilingual Wikipedia. 1 Introduction The discovery of neural scaling laws has resulted in an explosion in the size of datasets consumed and produced by AI models ( Kaplan et al. 2020 ) Hoffmann et al. 2022 . Traditional algorithms for unstructured data visualization, such as t-SNE ( Maaten and Hinton 2008 ) and UMAP ( McInnes et al. 2020 ) , have not kept up with the pace of dataset scaling. The presents a significant challenge for data-centric AI explainability, since it relies upon methods like t-SNE and UMAP for exploratory data analysis. Traditional unstructured data visualization methods scale very poorly in both wall-clock time and memory usage. Both these scaling issues can be resolved by moving towards a massively distributed paradigm for computing the visualizations, akin to how deep learning has moved to a distributed training paradigm. Distributing traditional unstructured data visualization algorithms is a non-trivial problem since their loss functions do not factorize across data points, making multi-device orchestration challenging. In this paper, we introduce Negative Or Mean Affinity Discrimination (NOMAD) Projection, an unstructured data visualization method that enables distributed computation by approximating an upper bound on the InfoNC-t-SNE loss. We demonstrate that NOMAD Projection effectively scales to multiple GPUs, which significantly reduces wall-clock time and enables us to quickly compute visualizations of datasets that do not fit into the memory of a single GPU. 2 Background Nonlinear dimensionality reduction for use in data visualization has been a focus of the machine learning community for over a decade. There has been a particular focus on the data mapping problem, which uses nonlinear dimensionality reduction to project vector data to the 2D plane so it can be rendered on a screen for subsequent analysis. Early algorithms for solving the data mapping problem include classical multidimensional scaling ( Torgerson 1952 ) , Sammon mapping Sammon 1969 , Locally Linear Embeddings ( Roweis and Saul 2000 ) , Isomap ( Tenenbaum et al. 2000 ) , and Stochastic Neighbor Embedding (SNE; Hinton and Roweis 2002 . The first method to gain widespread adoption was t-SNE Maaten and Hinton 2008 , which solved the crowding problem present in SNE. It was discovered that data mapping algorithms scaled poorly in both wall-time and memory consumption with the size of the input data. Several methods, including Barnes-Hut t-SNE van der Maaten 2013 , LargeVis ( Tang et al. 2016 ) , UMAP ( McInnes et al. 2020 ) , and NCVis ( Artemenkov and Panov 2020 ) leverage approximations in an attempt to improve their scalability. Work by Damrich et al. 2023 has unified the design choices of t-SNE, LargeVis, UMAP, and NCVis into a cohesive theoretical framework using the language of contrastive learning. Under their contrastive framework, solving the data mapping problem is interpreted as iteratively optimizing a spring system, where positive contrastive pairs are connected by attractive spring forces and negative contrastive pairs are connected by repulsive spring forces. Naive solutions to the data mapping problem compute positive and negative forces between all pairs of points, leading to quadratic complexity and poor scalability. LargeVis, UMAP, and NCVis all use similar approximation methods to sidestep the naive quadratic complexity and improve their scalability. In particular, they employ a kNN graph to compute a linear number of positive spring forces for each point, and use negative sampling or noise contrastive estimation (NCE) ( Ma and Collins 2018 ) to compute a linear number of negative forces for each point. Damrich et al. 2023 use the connection between data mapping and contrastive learning to derive InfoNC-t-SNE, a NCE approximation to t-SNE based on the InfoNCE loss ( van den Oord et al. 2018 ) . In InfoNC-t-SNE, edges in a kNN graph constructed from a vector dataset correspond to positive contrastive pairs, and edges from a complete graph over the dataset correspond to negative contrastive pairs. During gradient based optimization of the InfoNC-t-SNE loss, positive contrastive pairs induce attractive spring forces between points, and negative contrastive pairs induce repulsive spring forces between points. Alongside algorithmic advances, implementations such as t-SNE-cuda ( Chan et al. 2018 ) and RapidsUMAP ( Nolet et al. 2021 ) have improved data mapping scalability by taking advantage of powerful GPU hardware. However, none of these implementations can effectively utilize multiple GPUs at train time. This severely limits their scalability, as low GPU vRAM caps bottleneck the size of data they can be run on. [X] C_{1} C_{2} \vdots C_{|R|} D_{1} \vdots \vdots D_{\text{rank}} D_{1} D_{\text{rank}} \vdots \vdots \mu_{D_{1}} \mu_{D_{\text{rank}}} Figure 1 : NOMAD Projection’s Distributed Training Strategy - First, input data is partitioned into clusters C_{1},C_{2},...,C_{|R|} during the creation of the ANN index. Clusters are then sharded across devices D_{1},...,D_{\text{rank}} . Since each cluster is a component of the ANN graph, no inter-device communication is required during positive spring force calculation. After every epoch, only the matrices of cluster means \mu_{D_{1}},...,\mu_{D_{\text{rank}}} are all-gathered, minimizing the inter-device communication required for negative spring force calculation. 3 NOMAD Projection In this section, we develop Negative Or Mean Affinity Discrimination (NOMAD) Projection. NOMAD Projection approximates an upper bound on the InfoNC-t-SNE loss, enabling it to easily scale to multiple GPUs at train time. 3.1 InfoNC-t-SNE InfoNC-t-SNE learns low dimensional positions for high dimensional data by framing the problem as classification. In particular, InfoNC-t-SNE learns a classifier that can discriminate between true edges in a kNN graph constructed over vector data and randomly sampled noise edges from a complete variant of the kNN graph. Importantly, the InfoNC-t-SNE classifier is parameterized by the low dimensional data positions, which encourages them to encode information about high dimensional data proximity. Let P be a distribution over directed edges in a kNN graph P , and let ij denote an edge from head node i to tail node j in the kNN graph. Let \theta be a learned matrix of low dimensional positions. Let q be the Cauchy kernel: q(\theta_{i},\theta_{j})=\frac{1}{1+||\theta_{i}-\theta_{j}||^{2}} (1) where \theta_{i} denotes the i th row of \theta . Following the notation in Damrich et al. 2023 , we will often write q(ij) as a stand in for q(\theta_{i},\theta_{j}) for notational convenience. Let \xi be a uniform noise distribution over all edges in the complete digraph over nodes in the kNN graph, from which we can draw noise samples, m\sim\xi . Note that

Read the complete text