Multimodal Graph Representation Learning with Dynamic Information Pathways

Xiaobin Hong, Mingkai Lin, Xiaoli Wang, Chaoqun Wang, Wenzhong Li · 2026

Read Multimodal Graph Representation Learning with Dynamic Information Pathways by Xiaobin Hong, Mingkai Lin, Xiaoli Wang, Chaoqun Wang, Wenzhong Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-09258.txt.

Introduction

Xiaobin Hong and colleagues propose DiP, a framework for learning on multimodal graphs whose nodes mix features such as images and text. Older methods lean on static structures or dense attention, limiting flexible message flow. DiP adds modality-specific pseudo nodes that route messages within each mode by proximity, while shared state-space pathways carry information across modes. The sparse design keeps linear complexity, and tests on link prediction and node classification beat baselines across benchmarks.

Catalog record

CREATORXiaobin Hong, Mingkai Lin, Xiaoli Wang, Chaoqun Wang, Wenzhong Li
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47470
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.09258

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Multimodal Graph Representation Learning with Dynamic Information Pathways Xiaobin Hong, Mingkai Lin, Xiaoli Wang, Chaoqun Wang, Wenzhong Li Abstract Multimodal graphs, where nodes contain heterogeneous features such as images and text, are increasingly common in real-world applications. Effectively learning on such graphs requires both adaptive intra-modal message passing and efficient inter-modal aggregation. However, most existing approaches to multimodal graph learning are typically extended from conventional graph neural networks and rely on static structures or dense attention, which limit flexibility and expressive node embedding learning. In this paper, we propose a novel multimodal graph representation learning framework with Dynamic information Pathways (DiP). By introducing modality-specific pseudo nodes, DiP enables dynamic message routing within each modality via proximity-guided pseudo-node interactions and captures inter-modality dependence through efficient information pathways in a shared state space. This design achieves adaptive, expressive, and sparse message propagation across modalities with linear complexity. We conduct the link prediction and node classification tasks to evaluate performance and carry out full experimental analyses. Extensive experiments across multiple benchmarks demonstrate that DiP consistently outperforms baselines. Introduction Graphs are powerful abstractions for modeling relational and structural dependencies across a wide range of domains, including social networks ( 54 ; 35 ) , recommendation systems ( 50 ; 45 ) , and biological interactions ( 40 ; 27 ) . In many real-world applications, graph nodes are enriched with multimodal attributes, such as textual descriptions and visual content, leading to the emergence of Multimodal Graphs (MMGs) ( 49 ; 57 ) . As illustrated in Figure 1 , a typical MMG from a recommendation system represents items as nodes annotated with both images and textual metadata, while edges encode complex semantic or behavioral relationships among them. The inherent heterogeneity of MMGs, characterized by diverse feature modalities and intricate inter-node dependencies, makes them a compelling representation for downstream tasks. such as recommendation ( 48 ) , knowledge discovery ( 53 ; 55 ) , and scene understanding ( 22 ; 46 ) . To effectively leverage such heterogeneous information, models must not only propagate messages within each modality in a context-aware manner but also enable efficient and meaningful communication across modalities. Figure 1: A multimodal ego-graph example from a recommendation system, where its node is attributed with multimodal raw data (i.e., image and text), and the link indicates the complex relations between nodes. While multimodal learning ( 59 ) has garnered significant attention in recent years, research on multimodal graph representation learning remains relatively nascent. Most existing approaches to multimodal graph learning typically extend conventional graph neural networks (GNNs, like GCN ( 21 ) and GAT ( 42 ) ) by stacking modality-specific encoders with static graph convolutions ( 48 ) or cross-modal attention mechanisms ( 38 ) . While such architectures offer a straightforward extension of unimodal GNNs, they face significant limitations when applied to realistic, large-scale, and semantically complex multimodal graphs. First, there exists a fundamental misalignment in information granularity across modalities . Visual data often encodes fine-grained, instance-level cues such as spatial layout or object parts, whereas textual descriptions tend to abstract high-level semantic concepts. This granularity gap complicates alignment, as a direct fusion of such heterogeneous features often leads to semantic dilution or misinterpretation, especially when applied uniformly across all nodes. Second, most prior methods rely on static structure-based aggregation , either predefined by heuristics or constructed from fixed similarity measures. Such static topology fails to capture dynamic and context-aware dependencies between nodes, and cannot adapt to task-specific signals. Consequently, these rigid structures often lead to known issues such as over-smoothing ( 29 ) and over-squashing ( 5 ; 6 ) . Third, previous strategies exploit modal-agnostic fusion , such as feature concatenation or shared cross-modal attention after modal-independent learning, overlook the complementary nature of different modalities during both local and global aggregation. Without explicitly modeling modality-aware interactions and routing dynamics, these methods fail to fully exploit the expressiveness of MMGs. To address these challenges, we propose a novel multimodal graph representation learning method with Dynamic information Pathways (termed DiP). DiP introduces modality-specific pseudo nodes as lightweight, dynamic intermediaries enabling flexible, efficient, and scalable multimodal graph learning. The core idea is to decouple the complexity of node-level interactions by introducing two information pathways: (i) Intra-modal diffusion pathway : Each modal is equipped with a set of learnable pseudo nodes that mediate the message diffusion through a shared proximity-based attention mechanism. This allows the model to construct flexible message passing pathways that adapt to task-specific context, overcoming the rigidity of static graph structures. (ii) Inter-modal aggregation pathway : Instead of directly modeling dense cross-modal node interactions, DiP restricts inter-modality communication to pseudo-to-pseudo interactions. Pseudo nodes from different modalities interact in a shared state space using dynamic proximity, enabling expressive and complementary information fusion at significantly reduced computational cost. This design enables adaptive routing, heterogeneity-aware fusion, and sparse computation in a unified framework. The overall complexity scales linearly with node number, ensuring high scalability. We instantiate DiP with a recurrent architecture and shared message update function, improving parameter efficiency and generalizability. As a result, DiP supports expressive multimodal reasoning while maintaining linear complexity in graph size, making it suitable for large-scale applications. Experiments on various multimodal graph tasks (i.e., link prediction and node classification) show that DiP consistently outperforms existing MMG methods, particularly when modality relationships are complex or dynamically shifting. Our main contributions can be summarized as follows: • We propose DiP, a novel framework for multimodal graph representation learning, which enables adaptive, efficient, and scalable message propagation via learnable dynamic information pathways. • We design a multimodal message passing system that constructs dynamic intra- and inter-modality pathways, leading to expressive and context-aware node embeddings. • We conduct extensive experiments on multiple downstream tasks and provide a comprehensive analysis to demonstrate the effectiveness and efficiency of DiP. Related Work Multimodal Graph Learning Multimodal Graphs (MMGs) indicate that the nodes are associated with multimodal attributes, such as texts and images, which have been widely featured in real-life scenarios. Prior research in multimodal graph learning has largely focused on domain-specific applications such as knowledge graphs ( 3 ; 52 ) , molecular structures ( 20 ) , and brain networks ( 47 ) . These models are often tightly coupled with particular tasks and rely on handcrafted designs or domain expertise, which limits their ability to generalize across different graph structures, modalities, or learning objectives. More recently, MMGL ( 51 ) attempts to bridge this gap by employing foundation models from multiple modalities on multimodal graphs. However, its focus remains constrained to generative settings, leaving the challenge of building transferable, task-agnostic representations for multimodal graphs largely unaddressed. UniGraph2 ( 14 ) performs large-scale pretraining on visually grounded heterogeneous graphs using modality-specific encoders and MoE fusion. While these methods provide valuable insights into multimodal fusion, they typically use static intra-modality structures or dense fusion strategies, lacking the adaptability and efficiency necessary for dynamic multimodal graphs. Graph Message Passing Message passing is the fundamental paradigm of GNNs, which aggregates the information from neighbors and updates node states ( 21 ; 13 ; 15 ) . Many graph learning methods enhance expressiveness by approximating filters with parameterized polynomials ( 4 ; 9 ) , but are

Read the complete text