FishRoPE: Projective Rotary Position Embeddings for Omnidirectional Visual Perception
Rahul Ahuja, Mudit Jain, Bala Murali Manoghar Sai Sudhakar, Venkatraman Narayanan, [2pt] Pratik Likhar, Varun Ravi Kumar, Senthil Yogamani, [6pt] Automated Driving, Qualcomm Technologies, Inc · 2026
Read FishRoPE: Projective Rotary Position Embeddings for Omnidirectional Visual Perception by Rahul Ahuja, Mudit Jain, Bala Murali Manoghar Sai Sudhakar, Venkatraman Narayanan, [2pt] Pratik Likhar, Varun Ravi Kumar, Senthil Yogamani, [6pt] Automated Driving, Qualcomm Technologies, Inc (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-10391.txt.
Introduction
Ahuja and six colleagues adapt frozen vision models to fisheye cameras on self driving cars. Fisheye lenses warp straight lines, so tools built for plain pinhole cameras misread their views. Their FishRoPE method recasts attention in sphere based angles instead of flat pixel gaps. A tuned DINOv2 core with a small low rank adapter carries strong self taught features to fisheye views. It sets new best scores on WoodScape detection and SynWoodScapes scene parsing, with 54.3 mAP on detection.
Catalog record
| CREATOR | Rahul Ahuja, Mudit Jain, Bala Murali Manoghar Sai Sudhakar, Venkatraman Narayanan, [2pt] Pratik Likhar, Varun Ravi Kumar, Senthil Yogamani, [6pt] Automated Driving, Qualcomm Technologies, Inc |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28837 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.10391
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FishRoPE: Projective Rotary Position Embeddings for Omnidirectional Visual Perception Rahul Ahuja, Mudit Jain, Bala Murali Manoghar Sai Sudhakar, Venkatraman Narayanan, [2pt] Pratik Likhar, Varun Ravi Kumar, Senthil Yogamani, [6pt] Automated Driving, Qualcomm Technologies, Inc Abstract Vision foundation models (VFMs) and Bird’s Eye View (BEV) representation have advanced visual perception substantially, yet their internal spatial representations assume the rectilinear geometry of pinhole cameras. Fisheye cameras, widely deployed on production autonomous vehicles for their surround-view coverage, exhibit severe radial distortion that renders these representations geometrically inconsistent. At the same time, the scarcity of large-scale fisheye annotations makes retraining foundation models from scratch impractical. We present FishRoPE, a lightweight framework that adapts frozen VFMs to fisheye geometry through two components: a frozen DINOv2 backbone with Low-Rank Adaptation (LoRA) that transfers rich self-supervised features to fisheye without task-specific pretraining, and Fisheye Rotary Position Embedding (FishRoPE), which reparameterizes the attention mechanism in the spherical coordinates of the fisheye projection so that both self-attention and cross-attention operate on angular separation rather than pixel distance. FishRoPE is architecture-agnostic, introduces negligible computational overhead, and naturally reduces to the standard formulation under pinhole geometry. We evaluate FishRoPE on WoodScape 2D detection (54.3 mAP) and SynWoodScapes BEV segmentation (65.1 mIoU), where it achieves state-of-the-art results on both benchmarks. 1 Introduction Vision foundation models (VFMs) and standardized perception pipelines from DINOv2 16 backbones to BEVFormer-style lifting 11 have dramatically advanced autonomous driving perception. Yet this progress is fundamentally pinhole-centric : position encodings assume a uniform Cartesian grid, cross-attention lifting operates in rectilinear pixel space, and pretraining data is overwhelmingly perspective imagery. Fisheye cameras, which equip virtually every production vehicle with full 360 ∘ surround coverage using as few as four sensors ( > 190 ∘ FoV each) 27 ; 10 , are largely locked out of these advances. The barrier is twofold. First , large-scale fisheye annotations are orders of magnitude scarce than perspective benchmarks like nuScenes or KITTI, making it impractical to retrain foundation models from scratch. Existing fisheye methods 18 ; 9 ; 26 ; 20 therefore rely on ImageNet-pretrained ResNet, forfeiting the generalizable, distortion-robust representations that VFMs provide. Second , the spatial reasoning built into modern architectures is geometrically wrong for fisheye: a fixed pixel offset at the image center subtends 3 – 5\times more angular extent than the same offset at the periphery. Position encodings which are learned, sinusoidal, or even 2D rotary 5 encode pixel distances that misrepresent true spatial relationships under radial distortion. The same mismatch corrupts BEVFormer-style cross-attention, where Cartesian reference points fail to capture the non-linear image-to-ground mapping of fisheye projection. What the community needs is not another fisheye-specific architecture, but a geometric adapter that makes the existing VFM ecosystem and established lifting paradigms natively compatible with non-pinhole geometries. We present FishRoPE, a lightweight framework that serves exactly this role: • VFM backbone with geometry-aware adaptation. We adopt a frozen DINOv2 16 encoder (ViT-B/14) with lightweight LoRA 6 adaptation ( {\sim} 3M trainable parameters). While concurrent work (FishBEV 1 ) also explores VFM backbones for fisheye BEV, our contribution is the combination of VFM features with a geometry-aware position encoding that jointly addresses both the feature quality and spatial reasoning gaps. • Fisheye Rotary Position Embedding (FishRoPE). We propose a projective RoPE variant that encodes spatial positions in the spherical coordinate system (\theta,\phi) of the fisheye lens rather than the Cartesian pixel grid. FishRoPE preserves the relative-position property of RoPE 24 in the geometrically meaningful angular coordinate system, and applies uniformly to both encoder self-attention and BEVFormer-style cross-attention lifting. It adds negligible overhead, is architecture-agnostic, and degenerates gracefully to standard 2D RoPE for pinhole cameras. • Multi-task evaluation on established benchmarks. We evaluate on WoodScape 27 2D detection and SynWoodScapes 22 BEV segmentation, achieving competitive or state-of-the-art results on both tasks against published baselines. 2 Related Work Fisheye Object Detection. Standard bounding boxes are a poor representation for fisheye images due to radial distortion. FisheyeYOLO 18 and FisheyeDetNet 23 explored oriented boxes, ellipses, and polygon representations on the WoodScape dataset 27 ; 25 , establishing baselines for surround-view fisheye detection. OmniDet 10 demonstrated multi-task perception (detection, segmentation, depth) on fisheye imagery. These methods use conventional backbones (ResNet-18) and standard Cartesian position encodings, leaving distortion-aware feature extraction and geometrically faithful spatial reasoning unexplored. BEV Perception from Perspective Cameras. Generating bird’s-eye-view representations from camera images is a central problem in autonomous driving, with two dominant paradigms. Depth-based lifting methods 17 ; 3 predict a per-pixel depth distribution and use it to splat image features into a 3D voxel grid before collapsing to BEV. Lift-Splat-Shoot (LSS) 17 is foundational here, demonstrating that learning an implicit depth distribution in an end-to-end manner is both effective and scalable. Query-based lifting methods 11 ; 14 instead place BEV queries in ego space and gather image evidence via cross-attention. BEVFormer 11 is the seminal work in this direction: a grid of BEV queries attends to image features at projected reference points via deformable spatial cross-attention 31 , additionally leveraging temporal self-attention across frames to propagate history. PETR 14 encodes 3D camera-ray positions directly into image features as 3D positional embeddings before applying DETR-style cross-attention, avoiding explicit projection at query time. All of these methods assume pinhole cameras with Cartesian position encodings; applying them directly to fisheye cameras introduces geometric inconsistencies because equal pixel separations correspond to vastly different angular extents. Our BEV lifting module builds on BEVFormer’s spatial cross-attention paradigm and identifies position encoding as the key gap: we replace Cartesian query–key position encodings with FishRoPE, which operates in the spherical coordinate system of the fisheye lens. Fisheye BEV Segmentation. F2BEV 20 introduced the first BEV segmentation pipeline from surround-view fisheye cameras, adapting BEVFormer-style spatial cross-attention with Kannala–Brandt reference point projection while retaining standard learned position embeddings. FisheyeBEVSeg 29 proposed distortion-aware BEV pooling with explicit occlusion reasoning. DaF-BEVSeg 28 extended this with an occlusion-aware training objective. BEVCar 21 explored fisheye camera–radar fusion for BEV map segmentation in the WoodScape domain, highlighting the difficulty of lifting from severely distorted peripheral regions. FishBEV 1 recently demonstrated that DINOv2 features substantially improve fisheye BEV segmentation, reaching 42.1 mIoU on WoodScape with ViT-L. A key insight from FishBEV is that richer image features matter—but it retains standard 2D positional embeddings in the cross-attention, leaving the geometric mismatch between Cartesian encodings and fisheye projection unaddressed. Our work targets precisely this gap: FishRoPE encodes angular position in the cross-attention so that BEV queries correctly identify image evidence under non-linear fisheye distortion. Vision Foundation Models. DINOv2 16 produces rich visual features via self-supervised pretraining on 142M diverse, uncurated images. Its self-distillation objective yields features that are geometrically robust and generalize across large distribution shifts, making it a natural candidate for fisheye imagery where ImageNet-pretrained backbones degrade. Recent work has applied DINOv2 to autonomous driving via BEV feature distillation 19 and as a frozen backbone for downstream perception tasks. Parameter-efficient adaptation via LoRA 6 enables fine-tuning with only a small fraction of new parameters—critical when the base model is frozen. We are the first to combine a frozen DINOv2 backbone with geometry-aware position encodings designed specifically for fisheye cameras. Rotary Position Embeddings. RoPE 24 encodes relative position by applying frequency-dependent rotation matrices to query–key pairs in self-attention, so