From Thousands to Billions: 3D Visual Language Grounding via Render-Supervised Distillation from 2D VLMs
Ang Cao, Sergio Arnaud, Oleksandr Maksymets, Jianing Yang, Ayush Jain, Sriram Yenamandra, Ada Martin, Vincent-Pierre Berges, Paul McVay, Ruslan Partsey, Aravind Rajeswaran, Franziska Meier, Justin Johnson, Jeong Joon Park, Alexander Sax · 2025
Read From Thousands to Billions: 3D Visual Language Grounding via Render-Supervised Distillation from 2D VLMs by Ang Cao, Sergio Arnaud, Oleksandr Maksymets, Jianing Yang, Ayush Jain, Sriram Yenamandra, Ada Martin, Vincent-Pierre Berges, Paul McVay, Ruslan Partsey, Aravind Rajeswaran, Franziska Meier, Justin Johnson, Jeong Joon Park, Alexander Sax (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-20389.txt.
Introduction
Ang Cao and fourteen co-authors teach 3D scene models to understand language by borrowing supervision from 2D image models. Their 2025 paper faces a data gap: 2D models train on billions of images, while labeled 3D scenes number only in the thousands. That gap caps how well robots and AR systems can find things named in words.
Their method renders a predicted 3D scene back into 2D views during training. Big 2D models then grade those views, and no hand-labeled 3D data is needed at all. Scores on open-vocabulary 3D sorting jumped well past the prior best. Referential grounding tasks gained ten to thirty percent. Pretraining also doubled the value of later fine-tuning data, a sign of how starved the field is.
Catalog record
| CREATOR | Ang Cao, Sergio Arnaud, Oleksandr Maksymets, Jianing Yang, Ayush Jain, Sriram Yenamandra, Ada Martin, Vincent-Pierre Berges, Paul McVay, Ruslan Partsey, Aravind Rajeswaran, Franziska Meier, Justin Johnson, Jeong Joon Park, Alexander Sax |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-20389
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
From Thousands to Billions: 3D Visual Language Grounding via Render-Supervised Distillation from 2D VLMs Ang Cao, Sergio Arnaud, Oleksandr Maksymets, Jianing Yang, Ayush Jain, Sriram Yenamandra, Ada Martin, Vincent-Pierre Berges, Paul McVay, Ruslan Partsey, Aravind Rajeswaran, Franziska Meier, Justin Johnson, Jeong Joon Park, Alexander Sax Abstract 3D vision-language grounding faces a fundamental data bottleneck: while 2D models train on billions of images, 3D models have access to only thousands of labeled scenes–a six-order-of-magnitude gap that severely limits performance. We introduce LIFT-GS , a practical distillation technique that overcomes this limitation by using differentiable rendering to bridge 3D and 2D supervision. LIFT-GS predicts 3D Gaussian representations from point clouds and uses them to render predicted language-conditioned 3D masks into 2D views, enabling supervision from 2D foundation models (SAM, CLIP, LLaMA) without requiring any 3D annotations. This render-supervised formulation enables end-to-end training of complete encoder-decoder architectures and is inherently model-agnostic. LIFT-GS achieves state-of-the-art results with 25.7% mAP on open-vocabulary instance segmentation (vs. 20.2% prior SOTA) and consistent 10-30% improvements on referential grounding tasks. Remarkably, pretraining effectively multiplies fine-tuning datasets by 2×, demonstrating strong scaling properties that suggest 3D VLG currently operates in a severely data-scarce regime. Project page: https://liftgs.github.io . † † affiliationnotice: This work was partially done during the internship of Ang Cao at Meta. 1 Introduction When a user mentions the keys by the door or the blue mug on the table , they use language to indicate a specific set of objects and 3D locations in space. Such 3D language grounding provides a particularly natural interface for people to communicate about their surroundings. For AI systems operating in physical spaces, identifying the 3D masks or bounding boxes indexed by language queries represents a core functionality, with applications across autonomous navigation, robotic manipulation, and AR/VR. Figure 1 : LIFT-GS Overview. We train a powerful 3D vision language grounding model ( i.e. , 3D mask decoder) with point clouds and language as inputs by learning from 2D VLM foundation models without any 3D supervision. Yet despite its importance, 3D vision-language grounding (3D VLG) faces a fundamental bottleneck: data scarcity. While 2D vision-language models are trained on billions of labeled images and masks ( Achiam et al., 2023 ; Touvron et al., 2023 ; Radford et al., 2021 ; Labs, 2023 ) , existing 3D VLG models have access to only thousands of labeled 3D scenes and masks. This six-order-of-magnitude gap in data availability severely limits the capabilities of current 3D grounding systems, creating one of the most significant challenges in embodied AI. A common workaround to this scarcity constructs 3D feature fields from 2D features ( e.g. , CLIP embeddings) and performs text queries via dot products between the text and 3D embeddings. Although this provides good generalization, performance degrades with more detailed descriptions typical of real-world queries, as illustrated in Figure 3 . From this perspective, the dual-encoder approach falls short of 3D grounding as it contradicts a core grounding requirement. In this paper, we ask: can we combine the best part of both pipelines, i.e. , training a powerful grounding model while still overcoming the data scarcity by learning from powerful 2D models? The key insight is that differentiable rendering provides a natural bridge between 3D and 2D. If we can predict 3D masks and render them into 2D views, we can supervise them using 2D foundation models that have been trained on internet-scale data. This approach could enable training 3D models without any 3D mask annotations. We introduce Language-Indexed Field Transfer with Gaussian Splatting (LIFT-GS), which implements this idea as a practical training pipeline. Given a point cloud and language query, LIFT-GS predicts 3D Gaussian representations that can be rendered into multiple 2D views. These rendered masks are then supervised using pseudo-labels from 2D foundation models. We show results where SAM provides mask supervision ( Kirillov et al., 2023 ) , and CLIP or LLaMa provide language understanding ( Radford et al., 2021 ; Meta AI, 2024 ) . The approach effectively distills internet-scale 2D knowledge into 3D understanding. This render-supervised formulation offers several key advantages. First, it is inherently architecture-agnostic; specifying only the outputs leaves flexibility in underlying model design. Second, this allows us to overcome fundamental scaling limitations by training a large transformer decoder instead of previous dual-encoder approaches (as shown in Fig 3 ) ( Zhu et al., 2023b ; Gu et al., 2024 ) . Third, the approach is highly practical: LIFT-GS operates directly on raw point clouds from sensors, such as the outputs from SLAM or SfM systems, eliminating the preprocessing and feature fusion required by methods like ConceptFusion ( Jatavallabhula et al., 2023b ; Arnaud et al., 2025 ) , reducing inference time from 60 seconds to just 1 second. Our experiments validate the effectiveness of this approach. LIFT-GS achieves state-of-the-art performance on standard 3D VLG benchmarks, with 25.7% mAP on open-vocabulary instance segmentation (vs. 20.2% previous SOTA) and consistent 10-30% relative improvements on referential grounding tasks. More importantly, we observe that across data scales for SFT, pretraining effectively ”multiplies” the fine-tuning dataset by approximately a constant factor (2x). That is, a pretrained model with 50% of fine-tuning data matches the performance of training from scratch with 100% data. This somewhat counterintuitive observation indeed matches empirical data scaling laws for pretraining in other modalities ( Hernandez et al., 2021 ) , and the fact that this scaling coefficient remains constant without diminishing returns across data scales adds an empirical data point that 3D VLG currently operates in the very low-data regime 1 1 1 ( Hernandez et al., 2021 ) define a “low-data regime as having 10% or less of the amount of data it would take to get to 99% of the performance that infinite data would yield.” . The implications extend beyond 3D grounding. Render supervision is powerful, but we demonstrate that it can serve as a bridge for large-scale knowledge transfer from 2D foundation models to 3D models. Any 3D/4D task with renderable outputs can potentially leverage 2D supervision. As 2D foundation models continue to improve and scale, 3D models trained using render-supervised distillation are positioned to benefit. This opens the possibility of training 3D understanding models at the scale of 2D datasets–which would represent a fundamental shift from the current paradigm of limited 3D annotations. To summarize, our contributions are: • A render-supervised training pipeline for 3D vision-language grounding that requires only 2D supervision. We show how differentiable rendering enables training 3D models with 2D losses, eliminating dependence on scarce 3D annotations. • Demonstrating a pseudo-labeling strategy for distilling 2D foundation models into 3D. LIFT-GS shows using SAM, CLIP, and LLMs to generate 2D supervision. • State-of-the-art performance in realistic evaluations. LIFT-GS achieves SOTA results using sensor point clouds common in embodied settings, with detailed ablations revealing scaling properties. Figure 2 : 3D Referential Grounding. For each mentioned instance in a text description, predict a 3D mask and map it to corresponding text tokens. 2 Related Work 2.1 The Data Scarcity Challenge in 3D VLG 3D Vision-Language Grounding (3D VLG) maps language descriptions to corresponding 3D masks or bounding boxes in observed scenes ( Yuan et al., 2021 ; Roh et al., 2021 ; Yang et al., 2021