A Modality-Tailored Graph Modeling Framework for Urban Region Representation via Contrastive Learning

Yaya Zhao, Kaiqi Zhao, Zixuan Tang, Zhiyuan Liu, Xiaoling Lu, Yalei Du · 2025

Read A Modality-Tailored Graph Modeling Framework for Urban Region Representation via Contrastive Learning by Yaya Zhao, Kaiqi Zhao, Zixuan Tang, Zhiyuan Liu, Xiaoling Lu, Yalei Du (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-23772.txt.

Introduction

A city district can be described by shops, taxi flows, land use, roads, and pictures, but one graph model rarely fits them all. Yaya Zhao and 5 co-authors built MTGRR, a framework that tailors graph modeling to each data type. Sources split into aggregated-level and point-level groups. Expert graph networks handle each aggregated source, while a dual-level network draws fine visual features from street-level data.

Fusion weights then vary by region, respecting spatial differences instead of assuming one mix everywhere. Contrastive learning ties the levels together. On two real data sets, six data types, and three tasks, the method beat leading baselines throughout.

Catalog record

CREATORYaya Zhao, Kaiqi Zhao, Zixuan Tang, Zhiyuan Liu, Xiaoling Lu, Yalei Du
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-23772

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Modality-Tailored Graph Modeling Framework for Urban Region Representation via Contrastive Learning Yaya Zhao, Kaiqi Zhao, Zixuan Tang, Zhiyuan Liu, Xiaoling Lu, Yalei Du Abstract Graph-based models have emerged as a powerful paradigm for modeling multimodal urban data and learning region representations for various downstream tasks. However, existing approaches face two major limitations. (1) They typically employ identical graph neural network architectures across all modalities, failing to capture modality-specific structures and characteristics. (2) During the fusion stage, they often neglect spatial heterogeneity by assuming that the aggregation weights of different modalities remain invariant across regions, resulting in suboptimal representations. To address these issues, we propose MTGRR, a modality-tailored graph modeling framework for urban region representation, built upon a multimodal dataset comprising point of interest (POI), taxi mobility, land use, road element, remote sensing, and street view images. (1) MTGRR categorizes modalities into two groups based on spatial density and data characteristics: aggregated-level and point-level modalities. For aggregated-level modalities, MTGRR employs a mixture-of-experts (MoE) graph architecture, where each modality is processed by a dedicated expert GNN to capture distinct modality-specific characteristics. For the point-level modality, a dual-level GNN is constructed to extract fine-grained visual semantic features. (2) To obtain effective region representations under spatial heterogeneity, a spatially-aware multimodal fusion mechanism is designed to dynamically infer region-specific modality fusion weights. Building on this graph modeling framework, MTGRR further employs a joint contrastive learning strategy that integrates region aggregated-level, point-level, and fusion-level objectives to optimize region representations. Experiments on two real-world datasets across six modalities and three tasks demonstrate that MTGRR consistently outperforms state-of-the-art baselines, validating its effectiveness. † † paperid: 7857 1 Introduction Graph-based models have demonstrated a strong ability to capture complex spatial dependencies and relational structures among urban regions. Recent works ( Mandal and O’Connor 2024 ; Zhang et al. 2023a ; Zhou et al. 2023 ) have leveraged this capability to learn comprehensive urban region representations from diverse data modalities, including points of interest (POIs), taxi trajectories, land use, road elements, remote sensing imagery, and street-view images. By integrating these multimodal data sources, the resulting region representations capture the distinct semantic and structural characteristics of each modality. These representations can be applied to various downstream tasks, including carbon emission estimation ( Dai et al. 2023 ) , GDP prediction ( Perera and Fernando 2024 ) , and population forecasting ( Chen et al. 2022 ) . Figure 1: Comparison between existing methods and MTGRR . Despite their effectiveness, existing graph-based models for urban region representation face two key limitations, as shown in Figure 1 (left). Limitation 1: Most approaches use identical graph neural network architectures across all modalities, applying uniform processing pipelines to heterogeneous data. For example, models such as ( Xu and Zhou 2024 ; Zhou et al. 2023 ) integrate POI distributions and taxi mobility patterns, which represent different aspects of urban dynamics, yet they still process them using the same GCN architecture ( Kipf and Welling 2017 ) . This modality-agnostic design leads to under-representative embeddings that fail to capture modality-specific structures and characteristics. Limitation 2: Existing modality fusion strategies often overlook spatial heterogeneity across regions, assuming that the contribution of each modality remains invariant across the city. For example, ( Zhang et al. 2023a ; Zhang et al. 2023b ) directly average modality outputs to obtain region representations, and the fusion weights are shared across regions, failing to capture region-specific modality importance. This results in suboptimal representations that fail to reflect spatially varying urban characteristics. To address these issues, we propose MTGRR , a modality-tailored graph framework for urban region representation (Figure 1 , right). It leverages a comprehensive urban dataset comprising six modalities, categorized into two groups based on data characteristics and spatial density. (1) Aggregated-level modalities include POI, taxi mobility, land use, road element, and remote sensing imagery. Each POI is labeled with a functional category (e.g., restaurant, school); each taxi record is associated with a source region ID indicating inflow; each land parcel has a land-use label (e.g., residential, industrial); and each road segment is assigned a category label (e.g., living street, trunk link). For the first four modalities, modeling each point individually results in redundant information, as points with the same label exhibit highly similar features. Hence, they are better analyzed through label-based aggregation and treated as aggregated-level modalities. Although remote sensing imagery lacks explicit labels, it is considered aggregated-level because each region is associated with a single image that reflects region-level semantics. (2) Point-level modality refers to street-view imagery, where each region contains numerous images capturing fine-grained visual semantics at specific spatial locations. Due to significant variation among individual images and the absence of well-defined predefined labels for grouping, these data are modeled at the instance level and treated as a point-level modality. Based on this modality classification, MTGRR constructs modality-tailored graph neural networks for the two distinct types of urban modalities. For aggregated-level modalities, MTGRR adopts a mixture-of-experts (MoE) graph architecture, which consists of a global heterogeneous graph integrating all modalities, a set of expert GNNs each dedicated to a specific modality, and a modality-aware gating mechanism that adaptively fuses expert outputs into distinct modality-specific representations. For the point-level modality (i.e., street-view imagery), MTGRR employs a dual-level graph neural network to extract fine-grained visual semantic features. All images are treated as first-level nodes, and a second-level virtual node is introduced per region to aggregate local features. Intra-region edges connect all first-level image nodes to their corresponding second-level virtual node, while inter-region connections are only established between second-level nodes of adjacent regions. This hierarchical design enhances localized visual semantic representations. Furthermore, to obtain effective region representations under spatial heterogeneity, MTGRR introduces a spatially-aware multimodal fusion mechanism that dynamically learns region-specific fusion weights, allowing the model to adaptively adjust modality contributions across diverse regions and enhance the fused representations. Building on this framework, MTGRR adopts a joint contrastive learning strategy that integrates aggregated-level, point-level, and fusion-level objectives to optimize urban region representations. • We propose MTGRR , a modality-tailored graph modeling framework for urban region representation, which constructs a mixture-of-experts (MoE) graph architecture for aggregated-level modalities and a dual-level graph neural network for the point-level modality, enabling effective extraction of modality-specific representations incorporating distinct modality characteristics. • We introduce a spatially-aware multimodal fusion module that dynamically learns region-specific modality weights, enabling effective region representations that account for spatial heterogeneity. • We design a joint contrastive learning strategy that integrates region aggregated-level, point-level, and fusion-level contrastive objectives, jointly optimizing multimodal region representations. • We conduct extensive experiments on two real-world datasets covering six modalities and demonstrate that MTGRR consistently outperforms state-of-the-art methods across three downstream tasks, validating its effectiveness in diverse scenarios. 2 Related Work Graph Neural Networks for Multimodal Urban Data. Graph Neural Networks (GNNs) have shown strong potential in modeling multimodal urban data ( Schlichtkrull et al. 2018 ) , where heterogeneous sources such as POIs, taxi mobility, remote sensing, and street-view imagery are naturally structured as graphs. Recent works leverage GNNs to jointly encode semantic and spatial information across modalities. Early approaches often relied on single-modality data, such as taxi mobility, to define urban graphs ( Wang and Li 2017 ; Wu et al. 2022 ; Yao et al.

Read the complete text