CITYREP: A Unified Benchmark for Urban Representations Across Cities, Tasks, and Modalities
Junyuan Liu 1 Xinglei Wang 1 Zichao Zeng 1,2 Jiazhuang Feng 1 Quan Qin 1,3 Ilya Ilyankou 1 Guangsheng Dong 1,4 Tao Cheng 1,† 1 SpaceTimeLab, University College London, UK 2 3DIMPact, University College London, UK 3 School of Resource and Environmental Sciences, Wuhan University, China 4 State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, China † Corresponding author: [email protected] · 2026
Read CITYREP: A Unified Benchmark for Urban Representations Across Cities, Tasks, and Modalities by Junyuan Liu 1 Xinglei Wang 1 Zichao Zeng 1,2 Jiazhuang Feng 1 Quan Qin 1,3 Ilya Ilyankou 1 Guangsheng Dong 1,4 Tao Cheng 1,† 1 SpaceTimeLab, University College London, UK 2 3DIMPact, University College London, UK 3 School of Resource and Environmental Sciences, Wuhan University, China 4 State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, China † Corresponding author: [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26036.txt.
Introduction
Liu and seven colleagues offer CityRep, a shared yardstick for urban representation learning across cities, tasks and data types. City encodings turn complex city life into general vectors for downstream jobs, but current tests cover one or two cities and lean on random splits. Random splits leak nearby data across train and test, inflating scores. CityRep instead uses block-based spatial splits and a unit-agnostic alignment module for varied encodings. Its unified protocol should give fairer cross-city comparisons and firmer support for urban foundation models.
Catalog record
| CREATOR | Junyuan Liu 1 Xinglei Wang 1 Zichao Zeng 1,2 Jiazhuang Feng 1 Quan Qin 1,3 Ilya Ilyankou 1 Guangsheng Dong 1,4 Tao Cheng 1,† 1 SpaceTimeLab, University College London, UK 2 3DIMPact, University College London, UK 3 School of Resource and Environmental Sciences, Wuhan University, China 4 State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, China † Corresponding author: [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58372 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.26036
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CITYREP: A Unified Benchmark for Urban Representations Across Cities, Tasks, and Modalities Junyuan Liu 1 Xinglei Wang 1 Zichao Zeng 1,2 Jiazhuang Feng 1 Quan Qin 1,3 Ilya Ilyankou 1 Guangsheng Dong 1,4 Tao Cheng 1,† 1 SpaceTimeLab, University College London, UK 2 3DIMPact, University College London, UK 3 School of Resource and Environmental Sciences, Wuhan University, China 4 State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, China † Corresponding author: [email protected] Abstract Urban representation learning encodes complex urban environments into general-purpose embeddings for diverse downstream tasks and emerging urban foundation models. However, current evaluations are limited, typically focusing on one or two cities and tasks and relying on random splits that introduce spatial leakage, leading to inflated performance and weak support for cross-location generalization and fair comparison. To address this, we propose CityRep , a unified benchmark that evaluates urban representations across data modalities, cities, and tasks using spatially structured splits. CityRep consists of three key components: (1) a spatial unit-agnostic evaluation framework that supports heterogeneous urban representations through a standardized alignment module; (2) a unified evaluation protocol using block-based spatial splits to mitigate spatial leakage and enable rigorous model comparison; and (3) an extensible multi-city, multi-task benchmark suite spanning 8 cities and 8 tasks across regression, classification, and distribution prediction. We evaluate 11 representative urban representation models. Results show that performance is highly sensitive to the split protocol, with random splits inflating scores and altering model rankings. We also observe substantial variability across cities and tasks, underscoring the need for generalization-aware evaluation. CityRep is released as a reproducible benchmark with datasets, evaluation pipelines, and diagnostic tools to facilitate fair comparison and support future research in urban representation learning towards urban foundation models. Code: https://github.com/inwind0212/CityRep . 1 Introduction Figure 1: Framework of CityRep Benchmark. CityRep standardizes the evaluation of heterogeneous urban representations by aligning different spatial supports to common downstream task units, evaluating them across eight cities and eight tasks, and using spatial block splits to mitigate leakage. Urban representation learning seeks to turn heterogeneous observations of cities into reusable spatial embeddings. Recent models draw on remote sensing [ 16 , 12 , 7 ] , street-view imagery [ 42 , 47 ] , and points of interest [ 46 , 14 , 41 , 3 , 20 ] to encode geographic entities, regions, or locations. This motivation parallels the broader shift toward foundation models: representations learned from broad urban data are expected to transfer across tasks, locations, and domains. Yet the evaluation of these representations remains much less unified. Reported results are often tied to a particular model interface or a particular format of downstream task, making it difficult to assess whether an embedding is broadly useful or only effective under a narrow evaluation setup. Existing evaluations are also too narrow to support claims about general-purpose urban representations. Many studies evaluate on one or two cities, a small number of tasks, or a single label type. Such experiments are valuable for demonstrating a specific application, but they do not reveal whether a representation transfers across urban contexts or across qualitatively different prediction problems. This limitation is especially important for the emerging urban foundation models, whose value depends on broad reuse. A benchmark should therefore cover multiple cities, multiple urban domains, and multiple task types, while remaining extensible to incorporate new cities, tasks, and models. Finally, evaluation must explicitly account for spatial dependence. In urban representation learning, we expect models to leverage information from observed regions to make predictions in unseen areas, rather than merely interpolate among nearby samples. While this challenge has been well recognized in spatial validation studies [ 34 , 38 , 26 ] , most existing urban representation models are still evaluated using random splits that ignore spatial structure. We address this gap by establishing a unified spatially structured evaluation protocol. Our benchmark provides empirical evidence that random splits can substantially inflate performance and lead to over-optimistic conclusions about model generalization. We introduce CityRep , a unified benchmark for urban representations across modalities, tasks, and cities, as shown in Figure 1 . The central goal of CityRep is to move urban representation evaluation beyond narrow, single-setting comparisons. Rather than assessing an embedding on one city, one task, or one type of urban label, CityRep asks whether a representation remains useful across a sufficiently broad set of urban phenomena and geographic contexts. To operationalize this goal, we construct downstream evaluation data from four key dimensions of urban systems: morphology, demographics, economy, and environment. These dimensions are instantiated as eight tasks across eight cities, covering classification, regression, and distribution prediction. CityRep pairs this broad task suite with a common evaluation protocol for heterogeneous representations. Models based on rasters, regions, entities, or coordinates can be evaluated under the same downstream interface, while spatially structured splits are used to reduce leakage between nearby training and test samples. CityRep therefore makes it possible to examine not only average performance, but also how representation quality changes across urban contexts, task domains, label types, and split protocols. This provides a basis for assessing whether urban representations are genuinely general-purpose, rather than effective only under narrow evaluation settings. In summary, our contributions are: • We introduce CityRep , a unified and extensible benchmark for urban representation learning that supports heterogeneous representation types across data modalities, cities, and downstream urban tasks. • We design a spatially structured evaluation methodology, including spatial-unit alignment and block-based spatial splits, to enable fair comparison across heterogeneous urban representations while mitigating spatial leakage. • We conduct a large-scale empirical study of eleven representative urban and geospatial representation models across eight cities and eight tasks, showing that benchmark conclusions are highly sensitive to the evaluation protocol, task domain, and urban context. • We publicly release datasets, evaluation pipelines, processed benchmarks, model manifests, and diagnostic tools to support reproducible research on urban representation learning and urban foundation models. 2 Related Work Urban Representation Learning Existing urban representation learning methods are highly heterogeneous, differing in the data they utilise, the spatial units they operate upon, and the urban signals they encode. Following geographic information systems (GIS) taxonomy, this heterogeneity is largely shaped by whether models ingest vector data, such as points, polylines, and polygons, or raster data, such as satellite and street-view imagery. Vector-based methods often rely on POI data but produce different outputs: category embedding methods, including Place2Vec [ 46 ] , POI2Vec [ 11 ] , and SPPE [ 14 ] , capture spatial co-occurrence patterns of POI types and require aggregation to represent urban spaces, whereas entity embedding models such as Urban2Vec [ 42 ] , HGI [ 15 ] , and CityFM [ 3 ] directly encode regions, buildings, or roads. Raster-based methods, by contrast, naturally produce grid-cell embeddings, with earth-observation foundation models such as AlphaEarth Foundation [ 7 ] and TESSERA [ 12 ] enabling dense large-scale representations, and AETHER further incorporating POI semantics into raster foundations [ 19 ] . A related line of coordinate-based encoders, including Space2Vec [ 22 ] , SatCLIP [ 16 ] , and CaLLiPer [ 41 ] , learns representations for continuous locations from POIs, imagery, or language supervision. Consequently, the resulting embeddings operate over