OptiMAG: Structure-Semantic Alignment via Unbalanced Optimal Transport

Yilong Zuo, Xunkai Li, Zhihan Zhang, Qiangqiang Dai, Ronghua Li, Guoren Wang · 2026

Read OptiMAG: Structure-Semantic Alignment via Unbalanced Optimal Transport by Yilong Zuo, Xunkai Li, Zhihan Zhang, Qiangqiang Dai, Ronghua Li, Guoren Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-22856.txt.

Introduction

Yilong Zuo and five co-authors work on graphs whose nodes carry both text and images. Such multimodal graphs model many real systems, from products to papers. A problem arises when the meaning implied by node features clashes with the graph's own links.

Their method, OptiMAG, aligns the two views using a form of optimal transport that allows uneven matching. When features and links agree, tasks like labeling nodes work far better. The approach fits social, shopping, and science graphs where text and images meet. When structure and meaning line up, downstream tasks gain from both the graph links and the rich node content.

Catalog record

CREATORYilong Zuo, Xunkai Li, Zhihan Zhang, Qiangqiang Dai, Ronghua Li, Guoren Wang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-22856

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

OptiMAG: Structure-Semantic Alignment via Unbalanced Optimal Transport Yilong Zuo, Xunkai Li, Zhihan Zhang, Qiangqiang Dai, Ronghua Li, Guoren Wang Abstract Multimodal Attributed Graphs (MAGs) have been widely adopted for modeling complex systems by integrating multi-modal information, such as text and images, on nodes. However, we identify a discrepancy between the implicit semantic structure induced by different modality embeddings and the explicit graph structure. For instance, neighbors in the explicit graph structure may be close in one modality but distant in another. Since existing methods typically perform message passing over the fixed explicit graph structure, they inadvertently aggregate dissimilar features, introducing modality-specific noise and impeding effective node representation learning. To address this, we propose OptiMAG , an Unbalanced Optimal Transport-based regularization framework. OptiMAG employs the Fused Gromov-Wasserstein distance to explicitly guide cross-modal structural consistency within local neighborhoods, effectively mitigating structural-semantic conflicts. Moreover, a KL divergence penalty enables adaptive handling of cross-modal inconsistencies. This framework can be seamlessly integrated into existing multimodal graph models, acting as an effective drop-in regularizer. Experiments demonstrate that OptiMAG consistently outperforms baselines across multiple tasks, ranging from graph-centric tasks (e.g., node classification, link prediction) to multimodal-centric generation tasks (e.g., graph2text, graph2image). The source code will be available upon acceptance. Figure 1 : Cross-modal inconsistency in real MAG edges. Top (ele-fashion): two connected product nodes share the “Dickies Men’s” brand (textually similar) yet differ visually (T-shirt vs. cap). Bottom (Reddit-S): two connected posts show similar squirrel-with-nut imagery but diverge in text content. 1 Introduction Multimodal Attributed Graphs (MAGs), which integrate graph topology with rich multimodal node features, have gained traction in various domains including e-commerce recommendation and social network analysis ( Yan et al. 2024 ; Zhu et al. 2025 ) . Unlike Text-Attributed Graphs (TAGs) that carry only textual features, each node in a MAG is associated with multiple modalities (e.g., text and images). This enriches node semantics yet introduces challenges in learning coherent representations from such heterogeneous inputs. Prevailing approaches ( Fang et al. 2025 ; He et al. 2025 ) usually follow a two-stage pipeline: ( i ) project features of each modality into a shared embedding space and optionally fusing them; ( ii ) propagate information from neighbors via Graph Neural Networks (GNNs). This paradigm implicitly assumes high homophily , i.e., that adjacent nodes share similar features for all modalities. However, this assumption often breaks down in MAGs. As Figure 1 shows, two adjacent nodes can be highly similar in one modality yet dissimilar in another. This cross-modal inconsistency has been theoretically characterized through selection and perturbation biases in multimodal representation learning ( Cai et al. 2025 ) . Our empirical study across multiple MAG datasets ( Appendix B ) also confirms that such cross-modal inconsistency is pervasive: edges frequently connect nodes that share features in only a subset of modalities. We attribute this to the nature of MAG edge formation: an edge typically reflects similarity in some modality, without requiring agreement across all modalities. To characterize this phenomenon, we distinguish two topologies: the implicit semantic structure , an ideal graph where edges link only feature-similar nodes within a given modality, and the observed explicit graph structure , which is shaped by multi-modal factors and thus includes edges that are homophilous for some modalities but heterophilous for others. We term this mismatch structural-semantic conflict . This conflict affects GNN message passing in nuanced ways. On one hand, heterophilous edges can propagate features that are inconsistent within a given modality, injecting modality-specific noise and degrading learned representations. On the other hand, not every such edge is detrimental: when two nodes are semantically related yet visually distinct (e.g., “book” and “bookshelf”), aggregating their divergent visual features can enrich the representation by capturing complementary cues ( Li et al. 2024a ) . The challenge is that current multimodal graph models ( Hu et al. 2021 ; Tao et al. 2020 ) apply uniform aggregation, conflating beneficial complementarity with harmful conflict and thereby admitting destructive noise. Meanwhile, heterophilous-graph techniques—including spectral filtering ( Zhu et al. 2020 ) , graph rewiring ( Li et al. 2024b ) , and non-local aggregation (cf. ( Zheng et al. 2024 ) for a survey)—target unimodal settings; naively decoupling modalities forfeits cross-modal guidance and integrates poorly with existing pipelines. What is needed, therefore, is an alignment mechanism that (i) bridges each modality’s implicit semantic structure with the explicit graph, (ii) distinguishes beneficial complementarity from harmful conflict rather than enforcing rigid alignment, and (iii) operates as a plug-and-play regularizer for seamless integration with state-of-the-art multimodal graph models. We term this goal structural-semantic alignment . To meet these requirements, we present OptiMAG ( Figure 2 ), a regularization framework based on Unbalanced Optimal Transport (UOT). OptiMAG recasts structural-semantic alignment as a distributional transport problem and employs a hybrid strategy: the Fused Gromov-Wasserstein (FGW) distance ( Vayer et al. 2019b ) aligns cross-domain structures (addressing i ), while a KL divergence penalty enables adaptive rejection of harmful edges (addressing ii ). As a drop-in regularizer (addressing iii ), OptiMAG integrates seamlessly with existing backbones. Because computing GW scales as O(N^{3}) , we accelerate it via subgraph sampling and entropic Sinkhorn iterations, making OptiMAG practical for large-scale graphs. We evaluate OptiMAG on six MAG benchmarks covering both graph-centric (node classification, link prediction, clustering) and generation tasks (graph-to-text, graph-to-image). OptiMAG yields consistent gains across diverse backbones, improving node classification accuracy by up to 4.6% and captioning CIDEr by over 4 points. The improvement is most pronounced for pretrained encoders like UniGraph2, where OptiMAG’s structured regularization steers large models toward finer-grained graph adaptation. Our Contributions. (1) Problem Formalization. We formalize structural-semantic conflict and provide empirical evidence for its prevalence in MAGs and its adverse effect on GNN aggregation. (2) Method. We introduce OptiMAG, an Unbalanced Optimal Transport-based regularizer that mitigates aggregation noise stemming from cross-modal structural misalignment. (3) Validation. We validate OptiMAG on diverse backbones across graph-centric (node classification, link prediction, clustering) and multimodal-centric (graph-to-text, graph-to-image) tasks, showing consistent gains over strong baselines. 2 Preliminaries 2.1 Multimodal Attributed Graphs A Multimodal Attributed Graph is defined as \mathcal{G}=(\mathcal{V},\mathcal{E},\mathcal{M}) ( Yan et al. 2024 ; Zhu et al. 2025 ) , where \mathcal{V} is a set of N nodes, \mathcal{E} the edge set, and \mathcal{M}=\{T,I\} the modality set (text and image). For each node v_{i}\in\mathcal{V} , let \mathbf{x}_{i}^{T} and \mathbf{x}_{i}^{I} denote the raw textual and visual inputs. Modality-specific encoders f_{\theta_{T}} and f_{\theta_{I}} map these inputs to latent representations: \mathbf{h}_{i}^{T}=f_{\theta_{T}}(\mathbf{x}_{i}^{T})\in\mathbb{R}^{d_{T}},\quad\mathbf{h}_{i}^{I}=f_{\theta_{I}}(\mathbf{x}_{i}^{I})\in\mathbb{R}^{d_{I}} (1) where d_{T} and d_{I} are the respective embedding dimensions. Typically d_{T}\neq d_{I} , so the two modality spaces are not naturally aligned. 2.2 Optimal Transport Optimal Transport (OT) formalizes the comparison of probability distributions as a constrained mass transfer problem ( Peyré & Cuturi 2019 ) . Given two discrete distributions \boldsymbol{\mu}\in\Sigma_{n} and \boldsymbol{\nu}\in\Sigma_{m} over source domain \mathcal{X} and target domain \mathcal{Y} , where \Sigma_{n}=\{\mathbf{a}\in\mathbb{R}_{+}^{n}|\sum_{i}a_{i}=1\} denotes the probability simplex, the Kantorovich formulation seeks a transport plan \boldsymbol{\pi}\in\mathbb{R}_{+}^{n\times m} that minimizes the total cost of moving mass from \boldsymbol{\mu} to \boldsymbol{\nu} : \displaystyle\mathcal{W}(\boldsymbol{\mu},\boldsymbol{\nu}) \displaystyle=\min_{\boldsymbol{\pi}\in\Pi(\boldsymbol{\mu},\boldsymbol{\nu})}\sum_{i=1}^{n}\sum_{j=1}^{m}\mathbf{M}_{ij}\boldsymbol{\pi}_{ij} (2) \displaystyle=\min_{\boldsymbol{\pi}\in\Pi(\boldsymbol{\mu},\boldsymbol{\nu})}\langle\boldsymbol{\pi},\mathbf{M}\rangle_{F} Here, \Pi(\boldsymbol{\mu},\boldsymbol{\nu})=\{\boldsymbol{\pi}\in\mathbb{R}_{+}^{n\times m}|\boldsymbol{\pi}\mathbf{1}_{m}=\boldsymbol{\mu},\boldsymbol{\pi}^{\top}\mathbf{1}_{n}=\boldsymbol{\nu}\} defines the set of couplings satisfying marginal constraints. The ground cost matrix \mathbf{M}\in\mathbb{R}_{+}^{n\times m} encodes pairwise transport costs, with \mathbf{M}_{ij}=\|x_{i}-y_{j}\|^{2} being a common choice. When \mathbf{M} equals the p -th power of a

Read the complete text