Open-Vocabulary BEV Segmentation with 3D-Aware Geometric Constraints

Hojun Choi ∗, Seulbin Hwang, Dae Jung Kim, Kisung Kim, Hyunjung Shim †, Jinhan Lee † · 2026

Read Open-Vocabulary BEV Segmentation with 3D-Aware Geometric Constraints by Hojun Choi ∗, Seulbin Hwang, Dae Jung Kim, Kisung Kim, Hyunjung Shim †, Jinhan Lee † (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-24353.txt.

Introduction

Choi and colleagues open the wordlist of the top-down maps self-driving cars navigate by. Bird's-eye-view perception fuses camera images into a unified overhead picture. Production systems recognise only a fixed category list, though. They stumble on the wild real world. Their 2026 task, open-vocabulary BEV segmentation, borrows vision-language models to recognise categories never seen in training.

Lifting two-dimensional semantics into three dimensions is ill-posed in geometry. The framework, OVBEVSeg, constrains it in three growing stages. Reliable three-dimensional projection generates pseudo-labels for general use. Joint per-scene tuning enforces structural match between the flat and overhead views. A distillation step restores online speed. On the nuScenes benchmark the system sets the state of the art. It beats closed-set methods by a wide margin on unseen categories. It performs remarkably even with no ground truth for the novel classes.

Catalog record

CREATORHojun Choi ∗, Seulbin Hwang, Dae Jung Kim, Kisung Kim, Hyunjung Shim †, Jinhan Lee †
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE66 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.24353

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Open-Vocabulary BEV Segmentation with 3D-Aware Geometric Constraints Hojun Choi ∗, Seulbin Hwang, Dae Jung Kim, Kisung Kim, Hyunjung Shim †, Jinhan Lee † Abstract Bird’s-eye view (BEV) perception fuses multi-camera images into a unified top-down representation for autonomous driving. Despite recent progress, state-of-the-art methods remain confined to closed-set scenarios, making them vulnerable to unpredictable real-world environments. In this work, we introduce open-vocabulary BEV segmentation (OVBS), which leverages vision-language models (VLMs) to recognize categories beyond the training set while maintaining precise BEV perception and real-time efficiency. A key challenge in OVBS lies in the 3D geometric inconsistency inherent in the ill-posed lifting of 2D VLM semantics into BEV. To address this, we propose OVBEVSeg, a geometry-aware OVBS framework that enhances efficient Gaussian splatting (GS)-based unprojection by leveraging robust 3D geometric constraints across three progressive stages: (1) 2D-to-BEV pseudo-labeling via reliable 3D projection for OV generalization; (2) joint 2D–BEV per-scene optimization with BEV structural constraints for 3D geometric consistency; and (3) 3D geometric distillation for online efficiency. On the nuScenes dataset, OVBEVSeg achieves state-of-the-art performance, outperforming closed-set methods by 15.3 mIoU on unseen categories. Remarkably, even with no novel-class ground-truth labels, it remains competitive with self- and semi-supervised baselines trained with up to 40% of ground-truth annotations. Furthermore, it achieves 2.5 × \times faster inference with only 0.22 × \times the memory consumption of projection-based methods. † † footnotetext: ∗ Work done during an internship at NAVER LABS. † † footnotetext: † Co-corresponding authors 1 Introduction Bird’s-eye view (BEV) perception has emerged as a foundational paradigm in autonomous driving (AD), providing a unified ego-centric top-down representation for effective multi-sensor fusion. It now underpins a broad spectrum of 3D scene understanding tasks, including object detection [ 29 , 33 , 53 ] , semantic segmentation [ 5 , 18 , 20 , 41 , 61 ] , and trajectory planning [ 20 , 22 , 21 ] . A central challenge in BEV perception is how to establish spatial correspondence between 2D camera features and the BEV representation. This process is fueled by advancements in depth-based [ 20 , 23 , 41 , 28 ] , projection-based [ 5 , 18 ] , attention-based [ 29 , 61 ] , and Gaussian-based [ 4 , 36 ] architectures, leading to successful real-time, scalable inference BEV perception critical for modern autonomous platforms. Figure 1 : (c,d) Ground-truth or pseudo BEV annotations. (a,e) 3D and BEV renderings with plain 3D Gaussian splatting, showing dispersed splats beyond object regions. (b,f) 3D and BEV renderings with our BAGS, with splats concentrated along object boundaries. (g,h) Online BEV predictions: the baseline misses novel classes ( e.g . , “truck”) and exhibits boundary distortions, which are successfully resolved by our method. Despite strict demands for efficient deployment, a crucial yet overlooked assumption persists: existing BEV perception pipelines are fundamentally framed as closed-set systems. That is, both training and inference assume the same, finite set of semantic classes. In real-world AD, this assumption is neither realistic nor safe. In Fig. 1 (g) and 4 , autonomous vehicles continuously encounter previously unseen objects— e.g . , “truck,” or atypical movers like “stroller” or “wheelchair.” These novel classes, while present in sensor data, lack both human-annotated and ground-truth labels during training and are therefore silently ignored during inference, creating semantically blind regions in the BEV map. This discrepancy is more than a technical shortcoming: it poses a direct threat to downstream tasks such as path planning and collision avoidance [ 22 , 24 ] . We argue that reframing BEV perception for open-world environments is not optional but essential for ensuring semantic completeness and safety in real-world autonomy. To this end, we introduce open-vocabulary BEV segmentation (OVBS), a new task that extends OV recognition from 2D vision to the BEV domain, leveraging vision-language models (VLMs) to recognize previously unseen categories. The objective of OVBS is to simultaneously achieve three goals: OV generalization to novel classes, precise BEV perception, and inference-time efficiency. A natural extension to OVBS is to graft OV visual recognition [ 50 , 51 ] onto the core logic of unprojection-based BEV pipelines. Concretely, one could attach a 2D OV detector [ 32 ] to each camera view, lift the predicted 2D regions into 3D space via depth estimation, and project them into BEV—often used as a fair OV baseline. However, this seemingly straightforward “2D-OV-then-BEV” solution remains vulnerable to a longstanding challenge in AD: it relies on direct 2D-to-3D lifting, which is fundamentally brittle in large-scale outdoor scenes with sparse views. Small errors in 2D localization or semantic classification are amplified by the ill-posed lifting process [ 41 , 28 ] —a structural flaw inherent to all unprojection-based methods. In Fig. 1 , this direct mapping fails on two fronts: (L1) (c) compounded errors hinder the recognition of novel classes in BEV; and (L2) (g) inherent geometric instability results in geometrically inconsistent or distorted BEV boxes. In short, naive designs built upon 2D unprojection break down due to the lack of consistent multi-view 3D geometry—a prerequisite for both OV-to-BEV extensions and reliable unprojection-based BEV perception. This inherent vulnerability persists even in Gaussian splatting (GS)-based unprojection—arguably the most promising candidate due to its superior efficiency and scalability—as it remains susceptible to the same 2D-to-3D lifting errors (L1, L2). Specifically, it amplifies these biases by directly lifting 2D features into 3D Gaussians for BEV splatting. To address these limitations, our key insight is that the direction of geometric reasoning must be reversed: rather than lifting noisy 2D predictions into 3D (unprojection), we should project reliable 3D structure down into 2D and BEV (projection). Following this projective shift, we propose OVBEVSeg, a novel geometry-aware OVBS framework that leverages robust 3D geometric constraints across three progressive stages: (1) generating BEV pseudo-labels by establishing consistent 2D–BEV correspondences and routing 2D VLM semantics via reliable 3D projection of class-agnostic 3D detections [ 11 ] , thereby bypassing erroneous 2D unprojection and resolving (L1); (2) conducting joint 2D–BEV per-scene optimization constrained by BEV spatial layouts as strong geometric constraints, thereby enabling robust 3D geometric reconstruction from sparse viewpoints; and (3) distilling the recovered geometry into a student model, mitigating (L2) while preserving inference efficiency. Crucially, these stages are not independent modules but successive instantiations of a single principle: leveraging 3D geometric constraints to bypass the ill-posed 2D unprojection—first for semantics, geometry, and efficient knowledge transfer. Extensive experiments show that OVBEVSeg sets a new state of the art, surpassing prior work [ 5 ] by 15.3 mIoU on novel classes while running 2.5 \times faster with 0.22 \times memory usage. Without novel-class ground-truth labels, it remains comparable to semi-/self-supervised baselines using up to 40% annotations. 2 Related Work Bird’s-Eye View (BEV) Perception. BEV perception maps multi-sensor data into a unified top-down space for 3D scene understanding. Existing methods fall into four paradigms: (1) depth-based methods [ 23 , 28 , 41 ] explicitly splat image features with per-pixel depth distributions into a 3D grid, yielding geometrically grounded BEV; (2) projection-based schemes [ 5 , 18 ] query camera features at predefined 3D points; (3) attention-based transformers [ 29 , 31 , 33 , 48 , 61 ] leverage cross-attention to fuse image and BEV tokens; and (4) feed-forward

Read the complete text