City-VLM: Towards Multidomain Perception Scene Understanding via Multimodal Incomplete Learning DOI: XXXXXXX.XXXXXXX ISBN: 978-1-4503-XXXX-X/18/06 CCS: Computing methodologies Natural language processing CCS: Computing methodologies Scene understanding
Penglei Sun, Yaoxian Song, Xiangru Zhu, Xiang Liu, Qiang Wang, Yue Liu, Changqun Xia, Tiefeng Li, Yang Yang, Xiaowen Chu · 2025
Read City-VLM: Towards Multidomain Perception Scene Understanding via Multimodal Incomplete Learning DOI: XXXXXXX.XXXXXXX ISBN: 978-1-4503-XXXX-X/18/06 CCS: Computing methodologies Natural language processing CCS: Computing methodologies Scene understanding by Penglei Sun, Yaoxian Song, Xiangru Zhu, Xiang Liu, Qiang Wang, Yue Liu, Changqun Xia, Tiefeng Li, Yang Yang, Xiaowen Chu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-12795.txt.
Introduction
Vision-language models ace indoor scenes but stumble outdoors, where a city sprawls across satellite, drone, and street views at once. Penglei Sun and nine co-authors built the first data set for that multidomain task in 2025 and a model trained to learn even when one sense is missing.
The data set, SVM-City, packs 420 thousand images and billions of 3D points with over half a million question-answer pairs, gathered from cars, low and high aircraft, and satellites. Their learning method fuses 2D and 3D streams and copes when a modality drops out. The authors present the pair, data set plus model, as a base for machines tasked with reading whole cities, not just rooms.
Catalog record
| CREATOR | Penglei Sun, Yaoxian Song, Xiangru Zhu, Xiang Liu, Qiang Wang, Yue Liu, Changqun Xia, Tiefeng Li, Yang Yang, Xiaowen Chu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-12795
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
City-VLM: Towards Multidomain Perception Scene Understanding via Multimodal Incomplete Learning DOI: XXXXXXX.XXXXXXX ISBN: 978-1-4503-XXXX-X/18/06 CCS: Computing methodologies Natural language processing CCS: Computing methodologies Scene understanding Penglei Sun, Yaoxian Song, Xiangru Zhu, Xiang Liu, Qiang Wang, Yue Liu, Changqun Xia, Tiefeng Li, Yang Yang, Xiaowen Chu Abstract . Scene understanding enables intelligent agents to interpret and comprehend their environment. While existing large vision-language models (LVLMs) for scene understanding have primarily focused on indoor household tasks, they face two significant limitations when applied to outdoor large-scale scene understanding. First, outdoor scenarios typically encompass larger-scale environments observed through various sensors from multiple viewpoints (e.g., bird view and terrestrial view), while existing indoor LVLMs mainly analyze single visual modalities within building-scale contexts from humanoid viewpoints. Second, existing LVLMs suffer from missing multidomain perception outdoor data and struggle to effectively integrate 2D and 3D visual information. To address the aforementioned limitations, we build the first multidomain perception outdoor scene understanding dataset, named SVM-City , deriving from multi S cale scenarios with multi V iew and multi M odal instruction tuning data. It contains 420 420 k images and 4,811 4,811 M point clouds with 567 567 k question-answering pairs from vehicles, low-altitude drones, high-altitude aerial planes, and satellite. To effectively fuse the multimodal data in the absence of one modality, we introduce incomplete multimodal learning to model outdoor scene understanding and design the LVLM named City-VLM . Multimodal fusion is realized by constructing a joint probabilistic distribution space rather than implementing directly explicit fusion operations (e.g., concatenation). Experimental results on three typical outdoor scene understanding tasks show City-VLM achieves 18.14 % 18.14\% performance surpassing existing LVLMs in question-answering tasks averagely. Our method demonstrates pragmatic and generalization performance across multiple outdoor scenes. Our project is available on our website 1 1 1 https://sites.google.com/view/cityvlm/ . 1. Introduction Scene understanding involves enabling agents to recognize and interpret the semantic information of objects within their surrounding environment ( Chen et al. 2024b ) , which is a fundamental task for autonomous navigation ( Deruyttere et al. 2019 ; Yu et al. 2023 ; Song et al. 2024 ) , robot manipulation ( Song et al. 2022 ; Sarch et al. 2023 ; Sun et al. 2024b ) , digital city ( Xiang et al. 2020 ) , etc. Technically, it usually involves space-sky-land multidomain perception data in multimodal (e.g., image, point cloud) gathered from multiview observation (e.g., humanoid view, terrestrial view, and bird view) to profile cities at multiple scales ( Sola et al. 2020 ) . Currently, large vision language models (LVLMs) are used to model scene understanding problems popularly, which are fed with visual-texture information and generate text descriptions of a situated environment for an agent ( Fei et al. 2022 ) . The existing research mainly investigates indoor scene understanding while LVLMs in outdoor scene understanding have not been explored systematically. In indoor environments, as shown in Figure 1 (a), LVLMs integrate vision and language representations to respond to the context of indoor scenes ( Hong et al. 2024 ; Chen et al. 2024a ; Fu et al. 2024 ; Li et al. 2024 ; Huang et al. 2023 ) . The datasets in these studies, such as ScanNet ( Dai et al. 2017 ; Armeni et al. 2016 ) and Matterport3D ( Chang et al. 2017 ) , are primarily collected using portable devices equipped with scanning sensors or stereo-vision cameras. These LVLMs are often trained on downstream tasks like question-answering (QA) related to household activities at the building scale from the humanoid viewpoint, where they generally process a single visual modality (e.g., 2D or 3D data) at a time. In contrast, outdoor scenes are usually constructed through space-sky-land multidomain perception data collected from sources including terrestrial vehicle cameras ( Caesar et al. 2020 ; Geiger et al. 2012 ) , low-altitude drones ( Yang et al. 2023b ; Hu et al. 2022 ) , and high-altitude aircraft or satellites ( Wang et al. 2021 ) . However, existing LVLM research for the outdoors has not fully integrated multiscale, multiview, and multimodal visual data, nor does it effectively handle the simultaneous processing of these diverse data ( Yang et al. 2023a ; Cao et al. 2024 ) . Figure 1. Various large vision-language models (LVLMs) in scene understanding. (a) Existing works focus on indoor scenes with single-scale visual data collected by limited stereo-vision cameras or scanning sensors from a humanoid view. They process a single visual modality (e.g., 2D or 3D data) at a time. (b) Our work City-VLM studies multi- \{ Scale, View, Modal \} scene understanding outdoors . City-VLM employs the Incomplete Multimodal Fusion Module (IMF-Module) to model the incomplete visual perception (e.g., the 2D data or the 3D data is absent). Figure 2. The overview of SVM-City: (a) the visual semantic from SVM-City and (b) the data annotation process applied to the SVM-City dataset. To address these challenges, we explore the LVLMs in outdoor scene understanding by constructing a novel dataset SVM-City and the first outdoor LVLM model City-VLM, as shown in Figure 1 (b). For dataset design , we propose the instruction tuning datasets from the multi S cale outdoor city-level scene based on multi V iew observation and multi M odal data, called SVM-City . We collect multiscale visual data including high-altitude satellite remote sensing (RS) images, high-altitude aerial orthophotos, low-altitude drone point clouds, and terrestrial vehicle camera and lidar points. We design a question taxonomy that includes five question types to address common queries related to outdoor scene understanding based on the spatial question categories in cognitive science research ( Hayward and Tarr 1995 ) . Then we propose the automatic data annotation based on the ChatGPT and existing segmentation method. SVM-City contains 420 k images and 4,811 M point clouds with 567 k QA pairs. For model design, we propose an LVLM, named City-VLM , using multimodal data from multiview observation for the multiscale outdoor city-level scene (SVM-City) based on incomplete multimodal learning inspired by Wei et al. 2024 . In contrast to conventional multimodal fusion methods ( Ngiam et al. 2011 ) , our City-VLM attempts to construct a joint probabilistic distribution space for the multimodal input through the VAE-based ( Kingma 2013 ) Incomplete Multimodal Fusion Module (IMF Module), as shown in Figure 1 (b). Specifically, it is common that the partial input visual modalities are missing in practical outdoor environments. For example, in high-altitude situations, only 2D remote sensing images are available while 3D data is absent. To address it, a shared visual representation of incomplete visual perception is obtained by sampling from a probabilistic distribution based on the only available 2D image data. Experimental results indicate that our method outperforms the existing LVLMs 18.14\% averagely in three outdoor question-answering (QA) tasks. Our main contributions can be summarized as follows: \bullet We are the first to elaborately investigate outdoor scene understanding on multiscale, multiview, and multimodal city-level using LVLM, making it perform robustly in real scenarios. \bullet We propose SVM-City , the first outdoor city-level multi S cale, multi V iew, and multi M odal instruction tuning dataset. It consists of 420 thousand images and 4810.8 million point clouds with 567 thousand question-answering (QA) pairs.