Learning Street View Representations with Spatiotemporal Contrast Journal: Elesevier
Yong Li, Yingjing Huang, Gengchen Mai, Fan Zhang · 2025
Read Learning Street View Representations with Spatiotemporal Contrast Journal: Elesevier by Yong Li, Yingjing Huang, Gengchen Mai, Fan Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-04638.txt.
Introduction
Yong Li and co-authors teach machines to read street level photos of cities in a 2025 paper. Street photos capture moving life, built form, and neighborhood feel at once. Older encodings blur those layers together for city tasks. Their method uses time and place links between photos as free lessons.
Photos of one spot across years teach what stays fixed in built form. Nearby photos from one moment teach the feel of a neighborhood. The learned codes beat older methods on place finding and social estimates. Different lesson types shaped the codes in measurably different ways.
Catalog record
| CREATOR | Yong Li, Yingjing Huang, Gengchen Mai, Fan Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-04638
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Learning Street View Representations with Spatiotemporal Contrast Journal: Elesevier Yong Li, Yingjing Huang, Gengchen Mai, Fan Zhang Abstract Street view imagery is extensively utilized in representation learning for urban visual environments, supporting various sustainable development tasks such as environmental perception and socio-economic assessment. However, it is challenging for existing image representations to specifically encode the dynamic urban environment (such as pedestrians, vehicles, and vegetation), the built environment (including buildings, roads, and urban infrastructure), and the environmental ambiance (such as the cultural and socioeconomic atmosphere) depicted in street view imagery to address downstream tasks related to the city. In this work, we propose an innovative self-supervised learning framework that leverages temporal and spatial attributes of street view imagery to learn image representations of the dynamic urban environment for diverse downstream tasks. By employing street view images captured at the same location over time and spatially nearby views at the same time, we construct contrastive learning tasks designed to learn the temporal-invariant characteristics of the built environment and the spatial-invariant neighborhood ambiance. Our approach significantly outperforms traditional supervised and unsupervised methods in tasks such as visual place recognition, socioeconomic estimation, and human-environment perception. Moreover, we demonstrate the varying behaviors of image representations learned through different contrastive learning objectives across various downstream tasks. This study systematically discusses representation learning strategies for urban studies based on street view images, providing a benchmark that enhances the applicability of visual data in urban science. The code is available at https://github.com/yonglleee/UrbanSTCL . 1 Introduction In recent years, unsupervised learning has demonstrated outstanding performance on various downstream tasks. By leveraging methods such as contrastive learning ( He et al. 2020 ; Chen et al. 2020 ; Chen et al. 2021 ) and masked image modeling ( He et al. 2022 ; Xie et al. 2022 ) , it has achieved efficient image representation and exhibited excellence in classical computer vision tasks such as image classification ( Radford et al. 2021 ) , object detection ( He et al. 2022 ) , and semantic segmentation ( Wang et al. 2020a ) , surpassing the vast majority of traditional supervised learning only approaches. However, current unsupervised learning and self-supervised learning aim to encode as much semantic and structural information of objects and environments in a scene as possible ( Park et al. 2023 ; Huang et al. 2024 ) . This is not suitable for all downstream tasks in tasks such as street view-based urban environment understanding. For instance, in place recognition tasks ( Lowry et al. 2015 ) , the features are expected to focus only on place-invariant information, such as buildings and roads, filtering out dynamic information like lighting conditions, pedestrians, vehicles, and vegetation. In contrast, in tasks related to human perception of places ( Dubey et al. 2016 ; Zhang et al. 2018 ) , these dynamic elements are important. Moreover, tasks like socioeconomic prediction ( Wang et al. 2020b ) emphasize the spatially consistent expression of neighboring scenes. In image representation learning, selectively encoding dynamic and static information in urban environments and the ambiance they create is highly important but inherently challenging ( Cordts et al. 2016 ) . Achieving precise encoding of such information typically requires separately labeling dynamic and static elements and using specific training strategies (e.g., masking dynamic elements when encoding static ones ( Cheng et al. 2017 ; Wang et al. 2019 ) ). However, both the labeling and training processes are fraught with difficulties. Factors such as lighting conditions, vegetation appearance, and ground litter are challenging to label objectively and consistently. This makes it nearly impossible to accurately represent these complex environmental factors using traditional datasets (e.g., ImageNet ( Deng et al. 2009 ) , Places ( Zhou et al. 2017 ) ) and classical methods (supervised or unsupervised). Figure 1 : Spatial and temporal contrastive learning with street view images. Using street view images captured at the same location over time, contrastive learning tasks are designed to learn the temporal-invariant characteristics of the built environment; Using spatially proximate street view images from the same period, learning tasks are crafted to learn the spatial-invariant neighborhood ambiance, such as socioeconomic atmosphere. Unlike existing large-scale datasets, street view imagery, as a high-resolution urban visual dataset, possesses unique spatiotemporal attributes that can capture both dynamic and static information in urban environments and the ambiance they form ( Biljecki and Ito 2021 ; Zhang et al. 2024 ) . Therefore, in this work, we leverage these spatiotemporal attributes of street view imagery to propose a self-supervised urban visual representation framework based on street view imagery (see Figure 1 ). This framework aims to selectively extract and encode dynamic and static elements and their ambiance in urban environments according to the requirements of different downstream tasks, achieving precise representations of urban environments. Specifically, the framework is based on the following three hypotheses: 1. Temporal Invariance Representation : At the same location, static elements such as buildings and streets do not change in images taken at different times, whereas dynamic elements like lighting conditions, pedestrians, vehicles, and vegetation present randomness in images taken at different times. Learning temporal invariant representations can retain the encoding of static elements while automatically filtering out information about dynamic elements. To capture this temporal invariance, we utilize the temporal attributes of street view imagery to construct positive sample pairs from historical street view images taken at different times at the same location. We expect that, after pre-training, the temporal encoder can learn stable features of the built environment. This method is suitable for tasks that rely on temporal stability, such as visual place recognition. 2. Spatial Invariance Representation : At the same time, urban spaces at nearby locations usually exhibit similarity; the architectural styles and urban functions in adjacent areas are relatively consistent, while specific visual elements in images of nearby locations present randomness. Learning spatial invariant representations can encode the overall neighborhood ambiance within a specific spatial range while avoiding focusing on any specific elements. To capture this spatial invariance, we leverage the spatial attributes of street view imagery to construct positive sample pairs from street view images taken in adjacent areas at the same time. We expect that, after pre-training, the spatial encoder can learn spatially invariant neighborhood ambiance. This method is suitable for tasks that require spatial consistency, such as socioeconomic prediction. 3. Global Information Representation : Besides temporal and spatial invariance, there are elements in urban environments that require holistic perception; these global features are vital for tasks involving human perception. To capture these characteristics, we construct positive sample pairs by applying data augmentation to the same street view image. We expect that, after pre-training, the model can retain the key elements of the scene and comprehensively capture the global information of the image. We validate the effectiveness of these hypotheses across multiple urban downstream tasks. Experimental results demonstrate that different contrastive learning objectives can learn different types of features that are more suitable for their respective downstream tasks. We also conduct an in-depth analysis of the reasons behind the performance of different contrastive methods, further underscoring the importance of targeted learning strategies. This study systematically explores representation learning strategies in urban studies based on street view images,