PointTransformerX: Portable and Efficient 3D Point Cloud Processing without Sparse Algorithms
Laurenz Reichardt, Nikolas Ebert, Oliver Wasenmüller · 2026
Read PointTransformerX: Portable and Efficient 3D Point Cloud Processing without Sparse Algorithms by Laurenz Reichardt, Nikolas Ebert, Oliver Wasenmüller (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-24169.txt.
Introduction
Reichardt, Ebert, and Wasenmüller present PointTransformerX, a fully PyTorch-native vision transformer for 3D point clouds with no custom CUDA operators. It encodes 3D spatial relations in attention with a rotary embedding that needs no neighborhood construction, and it swaps sparse convolutional patch embedding for a linear projection. On ScanNet it reaches 98.7 percent of PointTransformer V3 accuracy, with far fewer parameters and much faster execution in only 253 MB of memory. It runs natively on NVIDIA, AMD, and CPU hardware, giving a portable base for point cloud perception.
Catalog record
| CREATOR | Laurenz Reichardt, Nikolas Ebert, Oliver Wasenmüller |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38123 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.24169
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PointTransformerX: Portable and Efficient 3D Point Cloud Processing without Sparse Algorithms Laurenz Reichardt, Nikolas Ebert, Oliver Wasenmüller Abstract 3D point cloud perception remains tightly coupled to custom CUDA operators for spatial operations, limiting portability and efficiency on non-NVIDIA, AMD, and embedded hardware. We introduce PointTransformerX (PTX), a fully PyTorch-native vision transformer backbone for 3D point clouds, removing all custom CUDA operators and external libraries while retaining competitive accuracy. PTX introduces 3D-GS-RoPE, a rotary positional embedding that encodes 3D spatial relationships directly in self-attention without neighborhood construction, and further replaces sparse convolutional patch embedding with a linear projection. PTX explores inference-time scaling of attention windows to improve accuracy without retraining. With a redesigned feed-forward network, PTX achieves 98.7% of PointTransformer V3’s accuracy on ScanNet with 79.2% fewer parameters and executing 1.6 × \times faster while requiring just 253 MB memory . PTX runs natively on NVIDIA GPUs, AMD GPUs (ROCm), and CPUs, providing an efficient and portable foundation for point cloud perception. 1 Introduction Point cloud perception is fundamental to a wide range of applications, including robotics, industrial automation, autonomous driving, and augmented and virtual reality. However, point cloud data is spatially unordered, and neural networks therefore rely on explicit spatial algorithms such as neighborhood grouping, voxelization, and radius search to extract geometric features, while other approaches rely on improved data coverage 35 . Such algorithms are computationally and memory intensive, and in practice neural networks almost exclusively integrate custom CUDA operators and dedicated libraries 20 ; 4 ; 41 ; 48 ; 12 ; 52 ; 13 . This creates a structural dependency on high-end NVIDIA hardware, preventing deployment on hardware from other vendors. Furthermore this relies on a fragmented library ecosystem which is often not maintained and lags well behind the standard PyTorch framework. As a result, modern capabilities such as BF16 and FP8 precision formats, JIT compilation, and kernel fusion are poorly supported across the dominant 3D perception stack. Recent point cloud transformers have begun reducing reliance on custom algorithms to better scale self-attention 43 , though this trend has not directly targeted the broader portability problem. Because attending to an entire point cloud is computationally intractable, initial point transformers 52 ; 47 ; 26 ; 29 ; 20 ; 19 form local attention windows through neighborhood grouping ( e.g . , K-Nearest Neighbors) or volumetric grouping ( e.g . , radius search, voxel windows). Neighborhood grouping is memory-inefficient, while volumetric grouping produces windows of unequal point density, leading to irregular parallelization and poor hardware utilization. Fixed-sequence transformers 22 ; 46 ; 44 address these issues with the goal of better model scaling by sorting or unfolding point clouds into equal-length patches, trading precise spatial locality for computational regularity and scaling efficiency. This allows self-attention to be implemented without external libraries and allows optimizations such as FlashAttention 8 . However, these architectures remain hybrid, retaining sparse 3D convolution 13 and dedicated scatter/gather libraries 12 that limit portability to embedded and non-NVIDIA hardware. We introduce PointTransformerX (PTX), a PyTorch-native point cloud transformer that eliminates all reliance on custom CUDA operators and external libraries, establishing a fully portable transformer backbone for 3D point clouds across diverse hardware targets. PTX builds on the serialization-based architecture of PointTransformer V3 (PTv3) 46 , but takes the approach further by completely removing sparse algorithms, resulting in a lightweight network with only 20.8% of PTv3’s parameter count. To maintain competitive accuracy, we make three targeted contributions. First, we replace sparse convolution with 3D-GS-RoPE and a standard linear projection for patch embedding. 3D-GS-RoPE is a novel Rotary Position Embedding 40 that avoids materializing neighborhood matrices and allows each attention head to independently learn an orthonormal coordinate rotation, enabling cross-axis positional relationships without custom operators. Second, we show that inference-time attention window scaling is beneficial in conjunction with 3D-GS-RoPE, improving accuracy while reducing the window size during training. Third, we redesign the feed-forward network (FFN) within the transformer block to further improve efficiency. Together, these contributions yield a model that achieves 98.7\% of PTv3’s accuracy on ScanNet 7 at just 9.6M parameters and 253 MB memory (BF16+FP8), effectively matching PTv3 performance without hard dependencies on specialized libraries or CUDA kernels. PTX is implemented fully in the standard PyTorch ecosystem and runs natively on NVIDIA, AMD, and CPU hardware. 2 Related Work 2.1 Point Cloud Backbones Point cloud backbones fall into three main families: point-based, voxel-based, and projection-based methods 34 . Point-based methods such as PointNet++ 33 , KPConv 42 , and Edge Convolution 45 operate on unstructured point sets and rely on kNN or radius search for local aggregation. These approaches require O(N^{2}) pairwise distance computations, which makes them prohibitively expensive at high resolutions. Voxel-based methods face a similar limitation since naive convolution or attention on dense voxel grids of resolution R scales as O(R^{3}) . Because most voxels are empty, methods such as SECOND 48 , MinkUNet 4 , and StratifiedTransformer 20 employ sparse algorithms that operate only on occupied voxels to reduce computational and memory costs. Projection-based methods such as BEV pillar encodings 21 use standard 2D convolution but incur irreversible geometric information loss. Furthermore, projection and unprojection as well as pooling and interpolation often rely on custom scatter and gather operators for efficiency 28 . Across all families, spatial algorithms heavily depend on custom CUDA operators across a fragmented ecosystem with limited support for modern compute and memory optimizations available in PyTorch. 2.2 Point Cloud Transformers As in other spaces, transformer based backbones see widespread adoption for point cloud perception. Early point cloud transformers 29 ; 52 ; 47 ; 20 ; 19 ; 26 used kNN or volumetric grouping for local window attention, producing irregular workloads incompatible with efficient attention implementations and limiting the receptive field. More recently, fixed-sequence transformers have emerged as a scalable alternative. These methods impose a one-dimensional ordering on the point cloud and divide the resulting sequence into fixed-length patches. OctFormer 44 adopts the z-order traversal used in octree construction for one-dimensional sequence ordering. FlatFormer 22 unfolds BEV pillars, while PointTransformer V3 (PTv3) 46 serializes voxels in three-dimensional space using z-order 27 and Hilbert 17 space-filling curves, partitioning the sorted voxels into fixed-size patches processed with window attention. PTv3 and FlatFormer further apply shuffle-order strategies to improve inter-patch connectivity across layers, whereas OctFormer uses dilated attention. Since no total ordering can preserve exact three-dimensional neighborhood relationships, this design trades precise spatial locality for computational efficiency. In PTv3, this increases the receptive field from 16 to 1024 points while achieving 3\times higher throughput and 10\times lower memory usage compared to its kNN-based predecessor PointTransformer V2 (PTv2) 47 . OctFormer, FlatFormer, and PTv3 therefore exhibit a consistent reduction in custom operators along the attention pathway. However, OctFormer and PTv3 still rely on sparse convolution for positional encoding and patch embedding, which results in a hybrid pipeline dependent on external libraries, while FlatFormer remains limited to two-dimensional projected point clouds. OctFormer is notable in that it provides PyTorch fallbacks to their CUDA kernels, allowing for portability, at the cost of efficiency. 2.3 Coordinate Position Embedding Point cloud transformers typically use relative position embedding (RPE) based on pairwise distances between points 20 ; 52 ; 49 . However while spatially precise, the O(N^{2})