TMTE: Effective Multimodal Graph Learning with Task-aware Modality and Topology Co-evolution 7509 CCS: Computing methodologies Semi-supervised learning settings
Yinlin Zhu, Xunkai Li, Di Wu, Wang Luo, Miao Hu, Guocong Quan · 2026
Read TMTE: Effective Multimodal Graph Learning with Task-aware Modality and Topology Co-evolution 7509 CCS: Computing methodologies Semi-supervised learning settings by Yinlin Zhu, Xunkai Li, Di Wu, Wang Luo, Miao Hu, Guocong Quan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-27723.txt.
Introduction
Zhu and co-authors let graphs and modalities improve each other over time. Multimodal graphs often carry noisy or missing links. One fixed graph rarely suits every task. TMTE evolves topology and features in a closed loop. Metric learning over modality embeddings drives link updates. Smooth fusion with cross-modal alignment updates features. An anchor trick keeps the math cheap. Task focus sharpens both sides as rounds proceed.
Catalog record
| CREATOR | Yinlin Zhu, Xunkai Li, Di Wu, Wang Luo, Miao Hu, Guocong Quan |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 81710 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.27723
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TMTE: Effective Multimodal Graph Learning with Task-aware Modality and Topology Co-evolution 7509 CCS: Computing methodologies Semi-supervised learning settings Yinlin Zhu, Xunkai Li, Di Wu, Wang Luo, Miao Hu, Guocong Quan Abstract . Multimodal-attributed graphs (MAGs) are a fundamental data structure for multimodal graph learning (MGL), enabling both graph-centric and modality-centric tasks. However, our empirical analysis reveals inherent topology quality limitations in real-world MAGs, including noisy interactions, missing connections, and task-agnostic relational structures. A single graph derived from generic relationships is therefore unlikely to be universally optimal for diverse downstream tasks. To address this challenge, we propose T ask-aware M odality and T opology co- E volution (TMTE), a novel MGL framework that jointly and iteratively optimizes graph topology and multimodal representations toward the target task. TMTE is motivated by the bidirectional coupling between modality and topology: multimodal attributes induce relational structures, while graph topology shapes modality representations. Concretely, TMTE casts topology evolution as multi-perspective metric learning over modality embeddings with an anchor-based approximation, and formulates modality evolution as smoothness-regularized fusion with cross-modal alignment, yielding a closed-loop task-aware co-evolution process. Extensive experiments on 9 MAG datasets and 1 non-graph multimodal dataset across 6 graph-centric and modality-centric tasks show that TMTE consistently achieves state-of-the-art performance. Our code is available at this repository . 1. Introduction Multimodal-attributed graphs (MAGs) represent entities as nodes with multimodal attributes (e.g., images and texts) and encode their relational dependencies as edges, which have been widely adopted in various real-world applications, such as finance ( 50 ) , biochemistry ( 25 ) , and recommendation systems ( 15 ) . Building upon this expressive data structure, multimodal graph learning (MGL) has attracted growing attention in recent years. On the one hand, MGL enhances graph-centric tasks (e.g., node classification ( 17 ) and link prediction ( 26 ) ) by leveraging rich multimodal attributes; on the other hand, it benefits modality-centric tasks (e.g., graph-to-text ( 12 ) and graph-to-image ( 23 ) generation) by exploiting graph topology for contextual information ( 30 ) . Despite their notable advances, most existing MGL methods rely on a perfect topology assumption , where the real-world graphs are presumed to be sufficiently well-structured for downstream tasks ( 7 ) . However, this assumption often does not hold in practice, which can be summarized from three perspectives: (1) Noisy Interactions. Due to inherent modality noise in the MAGs or imperfections in the collection process, the graph topology may contain spurious connections ( 54 ) . (2) Missing Interactions. The collected MAGs may overlook potentially informative connections. Moreover, in some practical applications with sequential entities such as audiovisual speech recognition ( 20 ) or isolated multimodal entities like image-text matching ( 36 ) , no explicit graph priors are available to characterize the inter-entity relationships. (3) Task-agnostic Interactions. Both intuitive reasoning and our empirical study (Sec. 3 ) suggest that a single graph structure derived from task-agnostic relationships (e.g., co-occurrence patterns) is unlikely to be universally optimal for diverse downstream tasks, since different tasks often benefit from distinct contextual signals. For example, classification tasks emphasize semantic category consistency, while generation tasks may require semantic diversity. Notably, existing methods ( 54 ) prune spurious links based on low cross-modal similarity, thereby targeting only noisy interactions while overlooking the other challenges. Consequently, existing MGL methods inevitably depend on multimodal contexts induced by suboptimal topologies, which compromises downstream task performance. How can we adaptively learn graph structures that faithfully capture inter-entity relationships while simultaneously providing task-specific contextual cues for diverse downstream tasks, thereby enabling effective MGL? In this work, we address this question by proposing a novel MGL framework, termed T ask-aware M odality and T opology co- E volution (TMTE). TMTE jointly and iteratively optimizes graph topology and multimodal representations toward the target task. Our key insight lies in the intrinsic coupling between modality and topology in MGL: (1) Modality \rightarrow Topology. Multimodal descriptions of entities naturally induce relational structures; for instance, user attributes shape social connections, and molecular properties govern bonding patterns. (2) Topology \rightarrow Modality. Conversely, graph topology influences modality representations via message passing or cross-modal alignment, thereby shaping the latent node embeddings used for downstream tasks. Concretely, TMTE casts topology evolution as multi-perspective metric learning over modality embeddings with an anchor-based approximation, and formulates modality evolution as smoothness-regularized fusion with cross-modal alignment. The refined topology provides improved contextual signals tailored to each node with respect to the downstream objective and induces enhanced modality representations through message passing or graph contrastive alignment. This procedure iterates across training epochs and forms a task-aware modality-topology co-evolution process, converging when the learned structure sufficiently approximates a topology optimized for the target task. Our Contributions: (1) In-depth Investigation. We provide an in-depth empirical analysis to demonstrate the inherent topology quality limitations of MAGs, as different downstream tasks often require distinct graph topologies to provide appropriate contextual signals. (2) Novel Method. We propose TMTE, a novel MGL framework that jointly and evolutionarily optimizes the graph topology and modality attributes toward the downstream task. (3) State-of-the-art Performance. Extensive experimental results on 9 MAG datasets and 1 non-graph multimodal dataset demonstrate that TMTE consistently outperforms the state-of-the-art baselines on 3 graph-centric and 3 modality-centric downstream tasks. 2. Problem Formulation A MAG is denoted as \mathcal{G}=(\mathcal{V},\mathcal{E},\{\mathbf{X}^{(m)}\}_{m\in\mathcal{M}}) , where \mathcal{V} is the set of nodes with N=|\mathcal{V}| , \mathcal{E} is the set of edges, and \mathcal{M} indexes the available modalities (e.g., text and images). For each node v_{i} under modality m , we associate a modality-specific feature vector \mathbf{x}_{i}^{(m)}\in\mathbb{R}^{d_{m}} , obtained by encoding the raw modality data with a pre-trained modality encoder (e.g., Sentence-BERT ( 38 ) and T5 ( 37 ) for text; ViT ( 45 ) and DINOv2 ( 34 ) for images). For each modality m , the embeddings of all nodes are organized into a feature matrix \mathbf{X}^{(m)}\in\mathbb{R}^{N\times d_{m}} . All modalities share a common relational structure represented by the adjacency matrix \mathbf{A}\in\mathbb{R}^{N\times N} , with the corresponding degree matrix \mathbf{D} , where \mathbf{D}_{ii}=\sum_{j}\mathbf{A}_{ij} and \mathbf{D}_{ij}=0 for i\neq j . To capture structural smoothness, the symmetrically normalized adjacency matrix and Laplacian of \mathcal{G} are represented by \tilde{\mathbf{A}}=\mathbf{D}^{-1/2}\mathbf{A}\mathbf{D}^{-1/2} and \mathbf{L}=\mathbf{I}-\tilde{\mathbf{A}} , respectively. In this paper, we consider three Graph-centric Tasks , including (1) Node Classification : assigning category labels to unlabeled nodes; (2) Link Prediction : estimating whether an edge (u,v) should exist in \mathcal{E} ; (3) Node Clustering : grouping nodes into clusters by applying standard clustering algorithms to the learned node embeddings; and three Modality-centric Tasks , including (1) Modality Retrieval : given a query from one modality (e.g., text), retrieving its corresponding representation in another modality; (2) Graph-to-Text Generation (G2Text) : producing textual outputs conditioned on a target node along with task instructions and its graph neighborhood; (3) Graph-to-Image Generation (G2Image) : synthesizing images using diffusion models conditioned on the node’s textual descriptions and the associated graph context. Further discussion of these tasks is presented in Appendix D . 3. Empirical Investigation In this section, we conduct a comprehensive analysis of the topology quality limitations of MAGs (as discussed in Sec. 1 ). Our goal is to systematically examine the following research questions, including Q1 : Are the original MAGs optimal for downstream tasks? Q2 : Do different