Natively Trainable Sparse Attention for Hierarchical Point Cloud Datasets
Nicolas LapautreUniversity of Groningen [email protected], Maria MarchenkoUniversity of Amsterdam [email protected], Carlos Miguel PatiñoUniversity of Amsterdam [email protected], Xin ZhouUniversity of Amsterdam [email protected] · 2025
Read Natively Trainable Sparse Attention for Hierarchical Point Cloud Datasets by Nicolas LapautreUniversity of Groningen [email protected], Maria MarchenkoUniversity of Amsterdam [email protected], Carlos Miguel PatiñoUniversity of Amsterdam [email protected], Xin ZhouUniversity of Amsterdam [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-10758.txt.
Introduction
Nicolas Lapautre, Maria Marchenko, Carlos Miguel Patiño, and Xin Zhou adapt a leaner attention method to transformers that read point clouds, in a 2025 paper. Attention, the core of transformers, grows quadratically with input size, which chokes models of big physical systems. A recent architecture, Erwin, handles such data as trees of points. The team grafts on Native Sparse Attention, a method built for sequences, and reshapes it for unordered point sets.
The combined model was tested on three physical data sets: cosmology simulations, molecular dynamics, and air pressure fields. It matched or beat the original Erwin model on each. The team also re-ran the original Erwin experiments to check their build against the published numbers. They argue the pairing widens the reach of transformers for physics, where systems are huge and detail matters at many scales.
Catalog record
| CREATOR | Nicolas LapautreUniversity of Groningen [email protected], Maria MarchenkoUniversity of Amsterdam [email protected], Carlos Miguel PatiñoUniversity of Amsterdam [email protected], Xin ZhouUniversity of Amsterdam [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 16 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-10758
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Natively Trainable Sparse Attention for Hierarchical Point Cloud Datasets Nicolas LapautreUniversity of Groningen [email protected], Maria MarchenkoUniversity of Amsterdam [email protected], Carlos Miguel PatiñoUniversity of Amsterdam [email protected], Xin ZhouUniversity of Amsterdam [email protected] Abstract Unlocking the potential of transformers on datasets of large physical systems depends on overcoming the quadratic scaling of the attention mechanism. This work explores combining the Erwin architecture with the Native Sparse Attention (NSA) mechanism to improve the efficiency and receptive field of transformer models for large-scale physical systems, addressing the challenge of quadratic attention complexity. We adapt the NSA mechanism for non-sequential data, implement the Erwin NSA model, and evaluate it on three datasets from the physical sciences—cosmology simulations, molecular dynamics, and air pressure modeling—achieving performance that matches or exceeds that of the original Erwin model. Additionally, we reproduce the experimental results from the Erwin paper to validate their implementation. 1 Introduction Self-attention has emerged as a fundamental mechanism in deep learning, revolutionizing how models capture complex relationships within data in areas like natural language processing (NLP), computer vision (CV), and 3D point clouds. The most significant benefit of self-attention is that it enables capturing both local and long-range dependencies by allowing information flow between all tokens or nodes. However, using self-attention in scenarios that require modeling interactions between a large number of tokens or nodes scales poorly because of the quadratic complexity of the mechanism. Our work addresses the quadratic scaling in self-attention through model architecture and hardware-aware optimizations for tasks related to point clouds in the physical sciences. We build upon the hierarchical transformer proposed in Erwin [ 1 ] and enhance its efficiency through the Native Sparse Attention (NSA) mechanism [ 2 ] . Erwin is a hierarchical transformer that achieves sub-quadratic scaling while maintaining state-of-the-art (SOTA) performance in point cloud datasets that require modeling short- and long-scale interactions. However, Erwin relies on a U-Net architecture to capture long-range interactions, so the strength of the receptive field is bottlenecked by the pooling operations in the U-Net layers. NSA integrates algorithmic innovations and hardware-aligned optimizations by introducing three attention mechanisms—compressed, selection, and sliding attentions—that achieve accelerated training and inference while maintaining the performance of a complete attention mechanism. The NSA was initially proposed to capture long- and short-term relationships for sequential tasks, but we extend their idea to work on 3D point clouds. By combining Erwin’s architecture with Native Sparse Attention, an approach we refer to as Erwin NSA, we leverage the speedup observed from NSA without compromising model performance in sequential tasks. We evaluate our approach using three datasets used by Zhdanov et al. 2025 to determine how Erwin NSA compares to Erwin in model performance, runtime, and memory consumption. 2 Related work 2.1 Sub-quadratic scaling for attention Several works have addressed attention’s quadratic scaling by proposing methods such as locality-sensitivity hashing [ 3 ] , sparse attention [ 4 ] , and low-rank approximation [ 5 ] in the context of NLP, as well as non-overlapping patches [ 6 ] in the context of CV. These methods leverage the assumption that data is regularly distributed across a grid—i.e., sequences of tokens in NLP and grids of pixels in CV. This regular distribution on a grid is not proper for applications like 3D point clouds and non-uniform meshes, so other works have explored approaches like converting point clouds into sequences [ 7 ] , hierarchical attention [ 8 , 9 , 10 ] , and cluster attention [ 11 , 12 ] . Beyond model innovations, work like FlashAttention [ 13 ] speeds up model training and inference without a compromise in model performance by accounting for how GPUs handle memory operations during the attention operation. This highlights the importance of not only focusing on efficient model architectures but also identifying opportunities to incorporate implementations of existing architectures that optimize runtime and memory usage by leveraging hardware capabilities effectively. 2.2 Erwin architecture The Erwin architecture is a hierarchical transformer designed for large-scale physical systems. It leverages ball tree partitioning to organize computation on a point cloud, enabling linear-time attention by processing nodes in parallel within local balls. Erwin captures both fine-grained local details and global context through a U-net architecture which progressively coarsens and refines the ball tree structure. Additionally, cross-ball interaction is further captured by rotating the ball tree. 2.3 NSA mechanism Native Sparse Attention (NSA) is a novel attention mechanism [ 2 ] designed for long-context modeling in language models which integrates hierarchical token modeling with sparse attention. It reduces per-query computation by organizing keys and values into temporal blocks. These blocks are processed through three attention paths—compressed coarse-grained tokens, fine-grained token blocks, and sliding window attention—which are then concatenated to form the final output. The selection of fine-grained blocks is guided by importance scores derived from the compressed token attention. The final output is a learned combination of these three attention branches. NSA is shown to maintain the accuracy of Full Attention while achieving a speedup of 2x to 9x, depending on the context length, with greater speedups for longer context lengths. 3 Reproducibility To validate the Erwin Transformer’s capability to capture long-range interactions in point clouds, accelerate simulations, and enhance expressivity, we replicate three experiments from the original paper: cosmological simulations, molecular dynamics, and airflow pressure modeling. 3.1 Experiments Cosmological simulations : The cosmology dataset [ 14 ] contains large-scale point clouds representing galaxy distributions. The input is a point cloud X\in\mathbb{R}^{5000\times 3} , where each row represents a galaxy with x,y,z coordinates. The model predicts the velocity of each galaxy Y\in\mathbb{R}^{5000\times 3} . The evaluation metric is the mean squared error (MSE) between predicted and ground truth velocities. Molecular dynamics (MD) : The MD dataset [ 15 , 16 ] consists of single-chain polymers. The model input is a polymer chain of N beads, each with a specific weight and associated velocities from the previous 16 timesteps. The model predicts the mean and variance of acceleration for each bead. The results are evaluated using negative log-likelihood (NLL). Airflow pressure modeling : The ShapeNet-Car dataset [ 17 , 12 ] consists of 889 car models, each represented by 3586 3D surface points X\in\mathbb{R}^{3586\times 3} . The model predicts the pressure P\in\mathbb{R}^{3586\times 1} at each point, evaluated using mean squared error (MSE) between predicted and ground truth pressures. The three experiments are replicated with the same model configurations and hyperparameters as in the original paper, which can also be found in Appendix B , and compared to its recorded results. 3.2 Results and discussion As shown in Table 1 , the replicated results closely match the original ones. The original ShapeNet result is from 1000 epochs, not after convergence, and the difference may be due to early training fluctuations. After contacting the author for the convergence result and replicating the training, both achieved an MSE ( \times 10^{-2} ) of 0.16. Dataset Metric Original Results Replicated Results Cosmology Test MSE, \times 10^{-2} 0.60 0.60 MD Test NLL 0.71 0.70 ShapeNet Test MSE, \times 10^{-2} 1.42 1.30 Table 1: Summary of replication results compared to the original. The results align closely for Cosmology and MD datasets, while the difference in ShapeNet may be due to early training fluctuations, as the results were recorded