LEO-VL: Towards 3D Vision-Language Generalists via Data Scaling with Efficient Representation

Jiangyong Huang Xiaojian Ma Xiongkun Linghu Yue Fan Junchao He Wenxin Tan, Qing Li, Song-Chun Zhu, Yixin Chen, Baoxiong Jia, Siyuan Huang · 2025

Read LEO-VL: Towards 3D Vision-Language Generalists via Data Scaling with Efficient Representation by Jiangyong Huang Xiaojian Ma Xiongkun Linghu Yue Fan Junchao He Wenxin Tan, Qing Li, Song-Chun Zhu, Yixin Chen, Baoxiong Jia, Siyuan Huang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-09935.txt.

Introduction

Jiangyong Huang and co-authors present a model that understands three-dimensional rooms from words and images, in this 2025 paper. The head titles the model LEO-VL and calls it an efficient route to scalable 3D vision-language learning. Such models still reason poorly about space, held back by heavy scene codes, thin training data, and shaky robustness.

Their fix starts with a condensed feature grid, a scene code that uses far fewer tokens while keeping detail. On top of it they trained LEO-VL on over 700,000 examples across four indoor settings and five tasks, from captioning to dialogue. A post-training step, SceneDPO, sharpens answers by contrasting scenes as well as replies. The model topped standard 3D benchmarks. Tests showed data quality and task variety mattered more than raw scale, a useful lesson for the field.

Catalog record

CREATORJiangyong Huang Xiaojian Ma Xiongkun Linghu Yue Fan Junchao He Wenxin Tan, Qing Li, Song-Chun Zhu, Yixin Chen, Baoxiong Jia, Siyuan Huang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-09935

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LEO-VL: Towards 3D Vision-Language Generalists via Data Scaling with Efficient Representation Jiangyong Huang Xiaojian Ma Xiongkun Linghu Yue Fan Junchao He Wenxin Tan, Qing Li, Song-Chun Zhu, Yixin Chen, Baoxiong Jia, Siyuan Huang Abstract Developing 3dvl generalists capable of understanding 3D scenes and following natural language instructions to perform a wide range of tasks has been a longstanding goal in the 3dvl community. Despite recent progress, 3dvl models still lag behind their 2D counterparts in capability and robustness, falling short of the generalist standard. A key obstacle to developing 3dvl generalists lies in data scalability, hindered by the lack of an efficient scene representation. We propose LEO-VL , a 3dvl model built upon cfg ( cfg ), an efficient scene representation that bridges 2D perception and 3D spatial structure while significantly reducing token overhead. This efficiency unlocks large-scale training towards 3dvl generalist, for which we curate over 700k high-quality 3dvl data spanning four domains of real-world indoor scenes and five tasks such as captioning and dialogue. LEO-VL achieves state-of-the-art performance on a variety of 3D QA benchmarks, including SQA3D, MSQA, and Beacon3D. Ablation studies confirm the efficiency of our representation, the importance of task and scene diversity, and the validity of our data curation principle. Furthermore, we introduce SceneDPO, a novel post-training objective that enhances the robustness of 3dvl models. We hope our findings contribute to the advancement of scalable and robust 3dvl generalists. Figure 1: An overview of LEO-VL , a 3dvl generalist. LEO-VL features an efficient scene representation with strong perception capability and low representation costs. We curate a comprehensive 3dvl dataset across diverse scene domains and tasks for the instruction tuning of LEO-VL . 1 Introduction With the crucial role of 3D scene understanding in human intelligence, building generalist models that can understand 3D environments and communicate through natural language [ 34 , 97 , 78 , 55 ] has become a central goal of the 3dvl ( 3dvl ) community. Recent progress has advanced capabilities across 3dvl tasks such as grounding [ 117 , 118 , 39 ] , captioning [ 53 , 93 ] , reasoning [ 32 , 46 ] , and dialogue [ 34 , 95 ] . Nonetheless, current 3dvl models fall significantly short of the generalist capabilities demonstrated by their 2D counterparts [ 60 , 82 , 88 , 13 , 42 ] . We posit that a key obstacle to developing 3dvl generalists lies in data scalability, hindered by the absence of an efficient scene representation. Therefore, establishing an efficient and scalable scene representation is a critical first step towards building 3dvl generalists. Existing scene representations in 3dvl models can be categorized into two approaches. The first adopts 3D modality inputs such as point clouds, but requires complex pre-processing pipelines such as 3D reconstruction and instance segmentation [ 117 , 39 , 11 ] . And they confront the inherent challenge of direct 3D perception, which is exacerbated by the scarcity of 3dvl data. The other approach leverages 2dvl ( 2dvl ) models for perception [ 115 , 111 , 71 ] , fueled by the advancement of 2D perception. However, this approach suffers from poor data scalability, as representing a 3D scene with multi-view 2D features incurs significant computational overhead. Additionally, both approaches typically use rudimentary 3D position embeddings for 3D spatial modeling, which fall short in capturing 3D spatial structures effectively. We propose LEO-VL , a 3dvl model built on cfg ( cfg ), an efficient scene representation that bridges 2D perception and 3D spatial structure, significantly improving efficiency while preserving the strength of 2D perception. Specifically, we employ a pretrained 2D visual encoder to extract image features, which are then back-projected into 3D space for voxelization. We utilize rope ( rope ) [ 79 ] to encode the height of voxels and condense voxel features vertically to form a cfg . For horizontal spatial modeling, we integrate 2D Fourier features [ 80 ] into the cfg . In addition to efficiency, our cfg simplifies 3D spatial modeling by disentangling vertical and horizontal dimensions. The scene tokens of cfg are further connected to a llm ( llm ) for auto-regressive language modeling. For the training of LEO-VL , we incorporate five 3dvl tasks, including captioning, qa ( qa ), planning, and dialogue, spanning four domains of real-world indoor scenes: ScanNet [ 19 ] , 3RScan [ 86 ] , MultiScan [ 58 ] , and ARKitScenes [ 5 ] . Within each domain, we curate existing datasets and newly generated data while prioritizing data quality over scale. This could avoid model performance degradation from naive scaling with low-quality sources. With our curation principle, we collect a balanced and diverse corpus of over 700k high-quality 3dvl data samples for the instruction tuning of LEO-VL . We evaluate LEO-VL on a variety of 3D QA benchmarks, demonstrating state-of-the-art performance with significantly higher efficiency than prior methods. Ablation studies exhibit advantages of the cfg in efficiency and 3D spatial modeling. Ablations on various training data configurations reveal the benefits of tasks such as captioning and dialogue, and the importance of scene diversity for strong cross-domain performance. We further show that scaling with simplistic QA data leads to performance degradation, confirming our data curation principle, while our curated data yields consistent scaling effects. Additionally, we introduce SceneDPO, a novel post-training objective that incorporates contrasts across answers and scenes, and demonstrate its effectiveness in enhancing model robustness compared to conventional sft ( sft ). In summary, our contributions are as follows: 1. We propose LEO-VL , a 3dvl generalist built upon cfg ( cfg ), an efficient scene representation that enables strong capability in 3D scene perception with significantly improved efficiency and simplified 3D spatial modeling. 2. We construct a comprehensive 3dvl instruction-tuning dataset spanning four scene domains and five tasks. Our dataset comprises over 700k high-quality samples, striking a balance between scale and quality, and serving as a reliable foundation for the training of 3dvl generalists. 3. We demonstrate LEO-VL ’s state-of-the-art performance across 3D QA benchmarks such as SQA3D. Ablation studies validate the efficiency of our representation, the benefits of task and domain diversity, and confirm our data curation principle. We also showcase the superiority of our post-training objective in enhancing model robustness compared to conventional sft . 2 Related Work 3D Vision-Language Models. Early work in 3dvl understanding develops models that process 3D representations such as point clouds [ 67 , 69 , 65 , 45 , 91 , 68 , 108 , 37 , 100 ] and voxels [ 27 , 59 , 74 , 81 , 28 , 18 , 15 , 76 ] . With the maturation of pretraining and instruction tuning techniques, 3dvl models have demonstrated significant performance improvements [ 109 , 36 , 1 , 10 , 36 , 49 , 94 , 116 , 117 , 114 , 90 ] and the capability of generalist models [ 29 , 93 , 34 , 95 , 11 , 70 , 118 , 26 , 107 , 16 , 32 , 21 ] . Nonetheless, 3D perception remains challenging due to the data scarcity and inherent complexity of 3D representation space. In contrast, an alternative line

Read the complete text