Beyond the Aggregation Dilemma: Prior-Retaining Decoupled Learning for Multimodal Graphs
Hao Yan 1 , Xuanru Wang 1 , Jun Yin, Shirui Pan, , Senzhang Wang 2 , and Chengqi Zhang 2 · 2026
Read Beyond the Aggregation Dilemma: Prior-Retaining Decoupled Learning for Multimodal Graphs by Hao Yan 1 , Xuanru Wang 1 , Jun Yin, Shirui Pan, , Senzhang Wang 2 , and Chengqi Zhang 2 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-24684.txt.
Introduction
Hao Yan and others diagnose why graph learning can lose to plain MLPs once encoders grow large. Under pretrained priors, forced aggregation mixes topology noise into node features, drowning the signal. It also throttles gradient flow while a shared loss lets dominant modalities crush weaker ones. Their SUPRA design decouples the paths. Topology-free MLPs handle modality features while a light shared graph network captures structural synergy, with extra supervision fighting starvation. Tests reach top results with 3.5 times lower peak GPU memory and much faster training than graph transformers.
Catalog record
| CREATOR | Hao Yan 1 , Xuanru Wang 1 , Jun Yin, Shirui Pan, , Senzhang Wang 2 , and Chengqi Zhang 2 |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 70740 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.24684
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Beyond the Aggregation Dilemma: Prior-Retaining Decoupled Learning for Multimodal Graphs Hao Yan is with the School of Computer Science and Engineering, Central South University, Changsha, China, and also with the Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University, Hong Kong SAR, China. This work was conducted during his visit to The Hong Kong Polytechnic University. Email: [email protected], [email protected] Xuanru Wang and Senzhang Wang are with the School of Computer Science and Engineering, Central South University, Changsha, China. Email: [email protected], [email protected] Jun Yin and Chengqi Zhang are with the Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University, Hong Kong SAR, China. Email: [email protected], [email protected] Shirui Pan is with the School of Information and Communication Technology, Griffith University, Brisbane, Australia. Email: [email protected] * Equal Contribution. † Corresponding Authors. Hao Yan 1 , Xuanru Wang 1 , Jun Yin, Shirui Pan, , Senzhang Wang 2 , and Chengqi Zhang 2 Abstract Multimodal Attributed Graph Learning (MAGL) integrates intrinsic node attributes with structural topology via graph aggregation. However, as pretrained encoders evolve into Large Foundation Models (LFMs), the landscape of MAGL fundamentally shifts: under high-confidence LFM priors, mandatory aggregation introduces topological noise that overwhelms discriminative signals, triggering a counter-intuitive performance inversion where sophisticated MAGL architectures underperform simple topology-agnostic MLPs. Through systematic empirical and theoretical analysis, we identify that this inversion stems from a fundamental aggregation dilemma characterized by two concurrent pathologies: (1) Representational Pathology (SNR Degradation)—mandatory aggregation dilutes robust intrinsic features with topological noise, causing the noise penalty to outweigh its collaborative benefit; and (2) Optimization Pathology (Gradient Starvation)—topological aggregation attenuates gradient flow, while a shared task loss causes dominant modalities to prematurely suppress weaker ones. To resolve this dilemma, we propose SUPRA (Shared-Unique Prior-Retaining Architecture), a decoupled dual-pathway paradigm. SUPRA processes modality-specific features through topology-agnostic MLPs while capturing structural synergy via a lightweight shared GNN, with auxiliary deep supervision counteracting gradient starvation. Extensive evaluations demonstrate that SUPRA achieves state-of-the-art performance while requiring 3.5 × \times lower peak GPU memory and up to 4.4 × \times faster training time than Multimodal Graph Transformers. I Introduction Graphs are ubiquitous in the real world, where nodes correspond to entities and edges capture their relationships [ 10 , 23 ] . In many practical scenarios [ 29 , 6 , 3 ] , nodes are associated with multimodal attributes (e.g., text and images), giving rise to Multimodal Attributed Graphs (MAGs) [ 28 , 16 ] . Effective learning on MAGs requires characterizing nodes from two complementary perspectives: the node-level intrinsic multimodal attributes and the structural context provided by the graph topology. A fundamental challenge lies in integrating these information sources. To address this challenge, Multimodal Attributed Graph Learning (MAGL) paradigms have evolved from early Multimodal GNNs (e.g., MMGCN [ 24 ] and MGAT [ 21 ] ) to advanced Multimodal Graph Transformers (e.g., MIG-GT [ 9 ] , NTSFormer [ 8 ] ). Historically, to compensate for the limited expressiveness of initial node representations, these architectures heavily relied on complex aggregation mechanisms to deeply fuse intrinsic semantics with structural topology [ 15 ] . Concurrently, the rapid advancement of Foundation Models (FMs) has transformed representation learning [ 20 , 2 , 27 ] . As feature extractors evolve from classic specialized FMs (e.g., RoBERTa [ 12 ] and CLIP [ 19 ] ) to advanced Large Foundation Models (LFMs, e.g., Llama-3.2-11B-Vision [ 22 ] ), the quality of initial node attributes fundamentally shifts from noisy inputs to highly discriminative semantic priors. This paradigm shift raises a critical question: in the era of powerful Foundation Models, is structural context still necessary for multimodal graphs, and if so, can existing MAGL architectures effectively integrate topology with strong semantic priors? (a) Overall Performance Comparison. (b) Semantic Attribution Analysis. Fig. 1 : Performance inversion and semantic attribution. (a) Accuracy comparison under classic FMs versus high-confidence LFMs, demonstrating the unexpected superiority of topology-agnostic MLPs. (b) Semantic attribution on Reddit-M isolating intrinsic node features ( Unique / Shared ) from structural context ( Synergy ). Motivated by this question, we conduct a pilot study comparing a topology-agnostic Multimodal MLP against representative MAGL models under the two feature regimes (Figure 1 a). Under the classic FM regime (RoBERTa+CLIP), structural context is clearly beneficial, with MAGL models generally outperforming the MLP. However, upgrading to LLaMA-Vision features triggers a counter-intuitive performance inversion . While most models improve, existing graph architectures fail to meaningfully capitalize on topology. Specifically, the topology-free MLP surpasses traditional multimodal GNNs across the board, notably achieving the highest overall accuracy on the Movies dataset. On Reddit-M, only the state-of-the-art NTSFormer manages to outperform the MLP, but it relies on a heavily parameterized architecture that incurs a prohibitive computational cost. This indicates that complex aggregation struggles to justify its severe overhead under high-confidence priors. To uncover the root cause, our semantic attribution analysis (Figure 1 b) explicitly breaks down the prediction accuracy into two fundamental sources: the baseline performance derived from the node’s own intrinsic features (comprising modality- Unique and Shared semantics), and the collaborative gain extracted from the graph topology ( Synergy ). The results reveal that existing paradigms are trapped in a fundamental aggregation dilemma : while traditional Multimodal GNNs employ rigid aggregation that dilutes robust intrinsic features with topological noise, Graph Transformers such as MIG-GT better preserve these features but employ weak aggregation that starves cross-modal synergy. As semantic priors become dominant, the noise penalty of aggregation begins to outweigh its collaborative benefit, turning this inherent dilemma into a critical bottleneck. Furthermore, while state-of-the-art models like NTSFormer attempt to bypass this by feeding explicitly concatenated multi-hop features into a Transformer, this brute-force dimension expansion incurs prohibitive overhead and still fails to escape the intrinsic topological noise, posing a severe risk of overfitting, notably failing to outperform the topology-agnostic MLP on the Movies dataset (Figure 1 a). We theoretically formalize this architectural dilemma by analyzing learning dynamics through the lens of Signal-to-Noise Ratio (SNR) [ 13 , 25 ] and optimization pathology (§ III ). We demonstrate that mandatory topological aggregation intrinsically degrades the SNR of high-confidence semantic priors (§ III-A ). Furthermore, we identify a dual-bottleneck optimization pathology termed Gradient Starvation [ 18 , 17 ] , where topological aggregation attenuates gradient flow, and a shared task loss allows dominant modalities to prematurely starve weaker ones (§ III-B ). Guided by these theoretical insights, we propose SUPRA ( S hared- U nique P rior- R etaining A rchitecture), a straightforward yet effective decoupled paradigm for multimodal graph learning. Diverging from the trend of increasing architectural complexity, SUPRA advocates selective propagation to explicitly model the two complementary perspectives of MAGs defined at the outset: node-level intrinsic multimodal attributes and structural context. Specifically, SUPRA establishes a decoupled dual-pathway architecture: it employs topology-agnostic MLPs as modality projectors to process and compress modality-unique features into specificity streams , which strictly bypass the GNN to preserve high-confidence multimodal attributes. Concurrently, a lightweight synergy stream (e.g., employing a foundational GNN encoder [ 10 ] ) jointly processes these features to capture structural context by extracting structural consensus. To counteract the Gradient Starvation triggered by topology-diluted gradients, we apply auxiliary deep supervision directly to the specificity branches, ensuring they receive