Unifying 2D and 3D Vision-Language Understanding
Ayush Jain, Alexander Swerdlow, Yuzhou Wang, Sergio Arnaud, Ada Martin, Alexander Sax, Franziska Meier, Katerina Fragkiadaki · 2025
Read Unifying 2D and 3D Vision-Language Understanding by Ayush Jain, Alexander Swerdlow, Yuzhou Wang, Sergio Arnaud, Ada Martin, Alexander Sax, Franziska Meier, Katerina Fragkiadaki (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-10745.txt.
Introduction
Ayush Jain and co-authors present a 2025 arXiv paper on one model that understands language about both flat images and 3D scenes. Three-dimensional training data are scarce, so their system, UniVLG, starts from pretrained 2D weights and trains on a blend of 2D and 3D material. A shared mask decoder, guided by language, picks out referred objects in either kind of input.
Lifting tricks convert 2D data into 3D help, narrowing the domain gap. The model set new best scores across several 3D grounding tasks, and joint training lifted both modes without hurting 2D skill. It needs no mesh reconstruction or pre-given object proposals.
Catalog record
| CREATOR | Ayush Jain, Alexander Swerdlow, Yuzhou Wang, Sergio Arnaud, Ada Martin, Alexander Sax, Franziska Meier, Katerina Fragkiadaki |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 64 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-10745
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Unifying 2D and 3D Vision-Language Understanding Ayush Jain, Alexander Swerdlow, Yuzhou Wang, Sergio Arnaud, Ada Martin, Alexander Sax, Franziska Meier, Katerina Fragkiadaki Abstract Progress in 3D vision-language learning has been hindered by the scarcity of large-scale 3D datasets. We introduce UniVLG, a unified architecture for 2D and 3D vision-language understanding that bridges the gap between existing 2D-centric models and the rich 3D sensory data available in embodied systems. Our approach initializes most model weights from pre-trained 2D models and trains on both 2D and 3D vision-language data. We propose a novel language-conditioned mask decoder shared across 2D and 3D modalities to ground objects effectively in both RGB and RGB-D images, outperforming box-based approaches. To further reduce the domain gap between 2D and 3D, we incorporate 2D-to-3D lifting strategies, enabling UniVLG to utilize 2D data to enhance 3D performance. With these innovations, our model achieves state-of-the-art performance across multiple 3D vision-language grounding tasks, demonstrating the potential of transferring advances from 2D vision-language learning to the data-constrained 3D domain. Furthermore, co-training on both 2D and 3D data enhances performance across modalities without sacrificing 2D capabilities. By removing the reliance on 3D mesh reconstruction and ground-truth object proposals, UniVLG sets a new standard for realistic, embodied-aligned evaluation. Code and additional visualizations are available at univlg.github.io . † † affiliationnotice: Equal contribution Figure 1 : (A) UniVLG achieves state-of-the-art performance performance across a range of referential grounding, question answering, and instance segmentation benchmarks. (B) UniVLG is a unified model which accepts posed RGB-D sequences, or monocular 2D images which are then lifted to 3D pointmaps. (C) UniVLG significantly benefits from joint 2D-3D training, further boosted when all parameters are shared between modalities by using 2D-3D lifting. (D) Example task inputs/outputs for UniVLG. 1 Introduction Today’s real-world embodied systems rely on depth sensors and egocentric, calibrated camera setups for navigation and interaction with their surroundings ( Ahn et al. 2022 ; Chiang et al. 2024 ) . However, despite having access to rich 3D information, these systems predominantly use 2D vision-language models to interpret their sensory video input, rather than leveraging 3D models that incorporate depth and egomotion. At first glance, this reliance on 2D models appears counterintuitive, as prior research has consistently shown that 3D models outperform their 2D counterparts when trained on comparable amounts of data ( Siddiqui et al. 2023 ; Kundu et al. 2020a ; Fang et al. 2021 ; Rukhovich et al. 2022 ) . The key limitation, however, is dataset availability: while 2D datasets are vast and well-curated, 3D datasets remain scarce and expensive to annotate ( Dai et al. 2017 ; Yeshwanth et al. 2023 ) . As a result, there are currently no high-performing, pre-trained 3D encoders capable of processing 3D inputs at the same level as CLIP ( Radford et al. 2021 ) does for 2D images. This data imbalance has led to a significant performance gap, ultimately slowing the widespread adoption of 3D models in embodied systems. Given these challenges, is scaling 3D training data the only viable path to bridging this gap, or are there alternative strategies for making 3D models more effective? In this paper, we introduce UniVLG, a unified 2D-3D vision-language model designed to improve 3D understanding by leveraging large-scale 2D data and pre-trained 2D models. UniVLG is trained on both 2D and 3D vision-language tasks, including referential grounding, object detection, and question answering in images and 3D scenes. Unlike models that operate directly on 3D point clouds, UniVLG processes RGB and RGB-D images—natural sensory inputs for embodied agents—and supports both single-view RGB images or multi-view posed RGB-D images. UniVLG processes each image with strong pre-trained 2D backbones, which also constitute the majority of its parameters, and fully leverages their representational power. It discriminates between 2D and 3D purely through the positional encodings of 2D image patch features, which represent the 2D pixel grid locations in images and the 3D (X,Y,Z) coordinates in scenes, similar to ( Jain et al. 2024 ) . When training on 2D RGB images, we consider both 2D and 3D processing pathways within UniVLG, by using predicted 3D pointmaps ( Wang et al. 2024 ) , which further narrows the domain gap between 2D and 3D input. We further introduce a novel language-conditioned mask decoder, shared across both 2D and 3D input, which predicts segmentation masks by conditioning on both visual features and language instructions to ground objects mentioned in the language input. Segmentation masks serve as a unifying output representation because they involve per-patch predictions, where each patch corresponds to either a 2D pixel or a 3D point. In our experiments, we show that besides unifying the output space, decoding to 3D masks results in significantly more precise predictions, challenging the established paradigm which decode bounding boxes or rely on object proposals. Our model is designed with the goal of sharing all weights across RGB image and RGB-D image sequence processing. We test UniVLG on established 2D and 3D vision language benchmarks ( Achlioptas et al. 2020 ; Chen et al. 2020a ) . We find that when trained exclusively on 3D data, UniVLG achieves state-of-the-art performance across all established benchmarks, outperforming prior methods in comparable settings by more than 15%. Furthermore, co-training UniVLG with 2D data enhances its 3D performance even further, both on in-domain and out-of-domain benchmarks. Notably, this improvement does not come at the expense of 2D tasks—UniVLG retains strong performance on 2D referential grounding datasets ( Kazemzadeh et al. 2014 ) compared to its version which is only trained on 2D referential grounding data. UniVLG directly uses sensor point clouds without any mesh pre-processing of the RGB-D input and without relying on ground-truth bounding box proposals, typically used in existing works and benchmarks ( Achlioptas et al. 2020 ) . By benchmarking in these more realistic settings, we hope to encourage future research that aligns more closely with the goals of embodied vision and promotes the progress of 3D vision in practical, real-world scenarios. In summary, our contributions are: • Unified 2D-3D Visual Grounding : We propose a model that can consume and benefit from both 2D and 3D vision-language data. • State-of-the-Art Performance: UniVLG achieves state-of-the-art performance on in-domain 3D referential grounding benchmarks, including ReferIt3D (SR3D, NR3D) and ScanRefer, outperforming prior methods by a significant margin, while also excelling in out-of-domain 3D referential grounding datasets. • Language-Conditioned 3D Mask Decoder: We propose a novel language-conditioned mask decoder head for 3D referential grounding and show its superior performance over bounding box decoders. • Realistic Evaluation Settings: We benchmark prior methods and UniVLG in realistic, embodied-aligned settings by using sensor-generated instead of mesh-reconstructed point clouds and without relying on ground-truth object proposals. We make our code publicly available at univlg.github.io . 2 Related Work Use of 2D data for 3D Visual Language Understanding Tasks Most 3D Visual Language models directly operate over the provided 3D point clouds without using any 2D pre-trained features. SAT-2D ( Yang et al. 2021a ) is one of the first 3D visual grounding model which used 2D visual features during training for aligning 2D and 3D visual features and show significant boost over its versions that do not use 2D features. Recent methods in 3D Question Answering like