OpenCity3D: What do Vision-Language Models know about Urban Environments?
2025
Read OpenCity3D: What do Vision-Language Models know about Urban Environments? (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-16776.txt.
Introduction
Valentin Bieri and co-authors asked in 2025 what vision-language models know about whole cities when shown three-dimensional reconstructions built from aerial photos. Such models usually work indoors or on roads. Their OpenCity3D approach points them at urban scale and asks high-level questions instead of tracing outlines.
The models estimated population density, guessed building ages, predicted property prices, and assessed crime and noise, all with little or no task-specific training. Zero-shot and few-shot results impressed the team. The authors claim a new paradigm: language-driven urban analytics, where a planner can interrogate a city model in words. Planning, policy, and environmental monitoring are the named beneficiaries. The project page hosts the materials for others to try.
Catalog record
| YEAR | 2025 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-16776
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
OpenCity3D: What do Vision-Language Models know about Urban Environments? Abstract Vision-language models (VLMs) show great promise for 3D scene understanding but are mainly applied to indoor spaces or autonomous driving, focusing on low-level tasks like segmentation. This work expands their use to urban-scale environments by leveraging 3D reconstructions from multi-view aerial imagery. We propose OpenCity3D, an approach that addresses high-level tasks, such as population density estimation, building age classification, property price prediction, crime rate assessment, and noise pollution evaluation. Our findings highlight OpenCity3D’s impressive zero-shot and few-shot capabilities, showcasing adaptability to new contexts. This research establishes a new paradigm for language-driven urban analytics, enabling applications in planning, policy, and environmental monitoring. See our project page: opencity3d.github.io Valentin Bieri 1 , Marco Zamboni 1 , Nicolas S. Blumer 1,2 , Qingxuan Chen 1,2 , Francis Engelmann 1,3 1 ETH Zürich 2 University of Zurich 3 Stanford University Figure 1 : OpenCity3D is a method for zero-shot urban 3D scene understanding, enabling insights into higher-level attributes such as crime rates, population density, housing prices, and local landmarks. For each text prompt, we visualize a response heatmap, where areas of higher relevance are highlighted in yellow, transitioning to blue for lower relevance. 1 Introduction Recent advancements in vision-language models (VLMs) and neural rendering techniques, such as Neural Radiance Fields (NeRFs) [ 21 ] and Gaussian Splatting (GS) [ 14 ] , have unlocked a wide range of applications in open-vocabulary 3D scene understanding. These approaches go beyond traditional 3D scene understanding capabilities by recognizing arbitrary object classes and concepts without requiring task-specific annotated training data. Instead, they leverage the generalization power of VLMs pre-trained on internet-scale image-text pairs, which encapsulate a vast array of human knowledge and concepts. A key application of these methods is open-vocabulary 3D segmentation, where user-provided queries – expressed as natural language or images – guide the recognition of regions within a 3D scene that correspond to the query. Methods such as OpenScene [ 25 ] and LeRF [ 15 ] have demonstrated this paradigm for 3D semantic segmentation on polygon meshes and neural fields, respectively. Similarly, OpenMask3D [ 36 ] and Open3DIS [ 22 ] address 3D instance segmentation. However, these methods have been predominantly evaluated in constrained domains, such as indoor environments and autonomous driving scenarios. This work explores, for the first time, the application of VLMs to city-scale urban 3D environments. Understanding urban-scale properties – ranging from building age to population density and crime rates – is vital for urban planning and sustainable development. Urban settings, however, introduce unique challenges, including large-scale spatial complexity and heterogeneous scene composition, which pose significant limitations for existing methods. Despite these challenges, leveraging VLMs in urban environments has the potential to provide actionable insights, offering a pathway to improved urban analysis and sustainability. To address these challenges, we propose OpenCity3D, a framework for open-vocabulary city-scale 3D scene understanding. OpenCity3D generates a language-enriched point cloud by processing RGB-D images rendered from aerial mesh reconstructions. Through the integration of language encoders, OpenCity3D enables querying of this enriched point cloud to analyze features of urban objects ( e.g . , buildings) and properties ( e.g . , population density, crime rates). Our findings demonstrate significant promise for urban scene understanding, particularly in tasks such as identifying building ages, estimating housing prices, and analyzing population density, while preliminary results for higher-order tasks like crime rate prediction and noise emission analysis remain less conclusive. In summary, our contributions are as follows: • We introduce OpenCity3D, a novel framework for open-vocabulary city-scale 3D scene understanding. • We present the first systematic analysis of VLM applicability to urban-scale question answering and scene analysis. • We establish an initial benchmark for open-vocabulary urban scene understanding, providing a foundation for future research in this domain. Figure 2 : The OpenCity3D model . Multi-view RGB-D images are rendered from aerial 3D reconstructions, followed by extracting pixel-wise hierarchical visual-language features. These features are mapped back to the 3D mesh, enabling language-based queries. 2 Related Work Open-Vocabulary 3D Segmentation Recent work has introduced advanced methods for 3D scene understanding, including a large variety of tasks including 3D segmentation [ 12 , 25 , 36 , 39 , 40 ] , human segmentation [ 37 ] , affordances [ 8 , 42 ] , localization [ 20 ] and robot applications [ 18 , 44 ] . Peng et al . [ 25 ] proposed OpenScene, which computes per-point features and fuses them with multi-view CLIP [ 29 ] embeddings to support open-vocabulary queries. However, OpenScene struggles with producing sharp segmentation masks and lacks instance-level discrimination. OpenMask3D [ 36 ] addresses open-vocabulary 3D instance segmentation by building on Mask3D [ 31 ] , using CLIP embeddings derived from SAM [ 16 ] masks on posed RGB-D images. These embeddings are matched to Mask3D instance masks, allowing comparison with text queries. A key advantage of OpenMask3D is its mask-level reasoning, which improves scalability in terms of computation and storage—crucial for city-scale 3D environments. However, as Mask3D is trained on indoor datasets, it does not generalize well to urban-scale scenes. Similarly, class-agnostic models like Segment3D [ 7 ] also underperform on city-scale data due to their indoor training regime. This concept is extended in Search3D to towards hierarchical 3D scenes. LERF [ 15 ] and OpenNeRF [ 9 ] extend NeRF [ 21 ] to language-driven queries by learning language fields from 2D CLIP features, enabling similarity-based rendering. LangSplat [ 28 ] instead combines 3D Gaussian Splatting with CLIP and SAM features, compressing VLM embeddings via an autoencoder and optimizing them through rendered-view comparisons with CLIP. While LangSplat shows promise, scaling Gaussian Splatting to large urban environments remains challenging. Moreover, its reliance on feature compression limits open-vocabulary performance. Our approach removes this bottleneck by adapting LangSplat’s hierarchical feature extraction to a sparse point cloud representation inspired by OpenScene. By forgoing Gaussian Splatting in favor of point clouds, we retain full, uncompressed VLM features—sacrificing some geometric fidelity, which is less critical for the high-level urban analytics tasks we target. Large-Scale 3D Scene Representations Existing methods for 3D scene understanding use diverse scene representations, including polygon meshes [ 25 , 36 ] , point clouds [ 11 , 35 ] , scene graphs [ 17 ] , NeRFs [ 9 ] , and Gaussian splats [ 28 ] . Urban-scale environments pose unique challenges, requiring representations that scale efficiently in both memory and computation. Implicit models such as ImpliCity [ 34 ] and Block-NeRF [ 38 ] offer strong denoising and compression but incur high optimization cost and require rendering for querying. In contrast, our method adopts an explicit point cloud representation to enable direct and efficient access to scenes. 3 Method 3D City Representation. Fig. 2 illustrates our OpenCity3D model. The goal is to create a 3D city representation augmented with VLM features for answering high-level socio-economical questions, such as population density, housing costs, and noise pollution. This 3D city representation is computed in three stages: rendering RGB-D images from a textured 3D mesh, extracting multi-scale pixel-aligned VLM features, and backprojecting these features onto the mesh vertices. The input consists of a 3D polygon mesh derived from an aerial 3D reconstruction, such as Google Maps 3D