URoPE : Universal Relative Position Embedding across Geometric Spaces

Yichen Xie, Depu Meng, Chensheng Peng, Yihan Hu, Quentin Herau, Masayoshi Tomizuka, Wei Zhan · 2026

Read URoPE : Universal Relative Position Embedding across Geometric Spaces by Yichen Xie, Depu Meng, Chensheng Peng, Yihan Hu, Quentin Herau, Masayoshi Tomizuka, Wei Zhan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-18747.txt.

Introduction

Xie and six colleagues propose URoPE, a general form of rotary position embedding that works across camera views and across dimensions. For each image patch, the method samples 3D points along the camera ray at preset depth anchors and maps them onto the query image plane, where standard 2D rotary embedding then applies. It adds no new parameters, uses camera intrinsics, and works with existing rotary-optimized attention kernels. In tests on view synthesis, 3D detection, tracking, and depth estimation, it lifts transformer models across the board.

Catalog record

CREATORYichen Xie, Depu Meng, Chensheng Peng, Yihan Hu, Quentin Herau, Masayoshi Tomizuka, Wei Zhan
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41621
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.18747

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

URoPE : Universal Relative Position Embedding across Geometric Spaces Yichen Xie, Depu Meng, Chensheng Peng, Yihan Hu, Quentin Herau, Masayoshi Tomizuka, Wei Zhan Abstract Relative position embedding has become a standard mechanism for encoding positional information in Transformers. However, existing formulations are typically limited to a fixed geometric space, namely 1D sequences or regular 2D/3D grids, which restricts their applicability to many computer vision tasks that require geometric reasoning across camera views or between 2D and 3D spaces. To address this limitation, we propose URoPE , a universal extension of Rotary Position Embedding (RoPE) to cross-view or cross-dimensional geometric spaces. For each key/value image patch, URoPE samples 3D points along the corresponding camera ray at predefined depth anchors and projects them into the query image plane. Standard 2D RoPE can then be applied using the projected pixel coordinates. URoPE is a parameter-free and intrinsics-aware relative position embedding that is invariant to the choice of global coordinate systems, while remaining fully compatible with existing RoPE-optimized attention kernels. We evaluate URoPE as a plug-in positional encoding for transformer architectures across a diverse set of tasks, including novel view synthesis, 3D object detection, object tracking, and depth estimation, covering 2D–2D, 2D–3D, and temporal scenarios. Experiments show that URoPE consistently improves the performance of transformer-based models across all tasks, demonstrating its effectiveness and generality for geometric reasoning. Our project website is: https://urope-pe.github.io/ . 1 Introduction Transformers [ 31 ] have become the dominant architecture in the field of computer vision including multi-view perception and generation tasks, from novel view synthesis [ 13 , 23 ] and stereo matching [ 41 , 14 ] , to 2D/3D object detection [ 2 , 20 , 19 , 34 ] . A critical challenge in applying Transformers to geometric tasks is how to encode the spatial relationships between tokens that originate from different viewpoints, coordinate systems, or even different geometric modalities (2D images and 3D points). As a standard solution, position embeddings inject positional information into the permutation invariant Transformer architecture. Compared with absolute position embeddings [ 31 ] , relative formulations [ 27 , 24 , 3 , 22 ] offer improved generalization and length extrapolation, making them a mainstream choice in modern Transformers, especially for geometric tasks. A particularly important development is Rotary Position Embedding (RoPE) [ 27 ] , which encodes relative rather than absolute positions through rotation matrices applied to query/key pairs. Standard RoPE operates within a single flat coordinate space, assigning positions by sequence indices (1D) or image grid locations (2D), which is fundamentally inadequate for cross-view geometric reasoning: pixels from two camera views may be close in 3D space yet far apart in their respective 2D grids. Figure 1 : URoPE for Relative Position Embedding across Geometric Spaces. Previous relative position embeddings can only handle a shared geometric space (top), while URoPE focuses on the relative position embedding across geometric spaces (bottom). Recent efforts have begun to incorporate camera geometry into attention, but often rely on matrix multiplication rather than extending RoPE itself to cross-view settings. Plücker ray embeddings [ 25 ] concatenate ray origin and direction as input features. This is an absolute encoding that lacks the relative bias properties crucial for attention generalization, while its relative variant [ 38 ] still relies on the definition of a global coordinate system. Recent work [ 21 , 15 ] combines camera parameters and standard 2D RoPE in a channel-wise split. However, the inter-camera geometry and intra-image spatial position are encoded disjointly with no interactions, and the inter-camera geometry is only considered for each camera rather than the local patch. Overall, it remains an open question to model the relative position across camera views and wider geometric spaces. We observe that the fundamental question in cross-view relative position is: where does a key token’s 3D content appear in the query token’s image? Instead of encoding abstract ray coordinates, URoPE uses explicit projective geometry to express cross-view correspondences directly in the query image plane , so the relative position can be modeled in a single shared coordinate system. Specifically, for each key token from a source view, URoPE starts from its camera ray and samples 3D points along the ray at a set of fixed depth anchors. Each sampled 3D point is then projected into the query camera using the relative camera transform and the query intrinsics, yielding a depth-conditioned pixel coordinate in the query image plane. Finally, we apply standard 2D RoPE between the query location and the projected key location, producing a geometry-aware relative positional bias that is consistent across camera views. A central challenge is that cross-view projection is inherently depth ambiguous: a source pixel corresponds to an epipolar line in the query view. We address this with depth-anchored multi-head attention : different attention heads (or head groups) are assigned with different fixed depth anchors, so each head encodes one depth hypothesis and multi-head attention jointly covers near- to far-field correspondences. In each head, we split the per-head channels across the horizontal and vertical axes ( d_{h}/2 for u and d_{h}/2 for v ) and apply standard 2D RoPE in the query image plane, keeping URoPE natively compatible with FlashAttention [ 4 ] and other RoPE-optimized kernels. A key contribution of URoPE is its universality. We demonstrate that the same projective RoPE formulation without task-specific modifications, serves as a plug-in position encoding across diverse geometric tasks: • Novel view synthesis : 2D \rightarrow 2D cross-view attention with known camera poses on Objaverse [ 5 ] and RealEstate10k [ 43 ] . • 3D object detection and tracking : 2D \rightarrow 3D attention between image features and 3D query positions on nuScenes [ 1 ] . • Stereo depth estimation : 2D \rightarrow 2D cross-view matching for depth prediction on RGBD [ 26 ] , Scenes11 [ 30 ] , and SUN3D [ 37 ] . We show consistent improvements across all benchmarks, establishing URoPE as a general-purpose geometric position encoding for Transformers. 2 Related Work Table 1 : Comparison of Cross-View Position Embedding. Mechanism: how geometry modulates attention: concatenation with input tokens (Concat.), matrix multiplication on Q/K/V features (MatMul), sinusoidal rotation on Q/K (RoPE), or a combination. Per-patch Geo.: whether the camera geometry is encoded at the individual patch level. SE(3) Inv.: invariance to rigid transformations of the global coordinate system. Param-Free: no learnable parameters introduced. Method Mechanism Geometric Info Per-patch Geo. SE(3) Inv. Param-Free Plücker [ 25 ] Concat. 6D Ray ✓ ✗ ✓ Relative Ray [ 38 ] RoPE 6D Ray ✓ ✗ ✓ GTA [ 21 ] MatMul Extrinsics ✗ ✓ ✓ PRoPE [ 15 ] RoPE + MatMul Proj. matrix + grid ✗ ✓ ✓ RayRoPE [ 36 ] RoPE + Linear Ray + learned depth ✓ ✓ ✗ URoPE RoPE Proj. coords ✓ ✓ ✓ Position Encoding in Transformers. Since sinusoidal positional encodings are introduced for sequence modeling [ 31 ] , position embeddings have become standard in transformers [ 6 , 2 ] for multiple tasks. As an important milestone, Rotary Position Embedding (RoPE) [ 27 ] encodes relative positions through rotation matrices applied to query and key vectors, enabling relative position bias without explicit bias terms. RoPE has become the default position encoding

Read the complete text