ML a GA : Multimodal Large Language and Graph Assistant
Dongzhe Fan, Yi Fang, Jiajin Liu, Djellel Difallah, Qiaoyu Tan · 2025
Read ML a GA : Multimodal Large Language and Graph Assistant by Dongzhe Fan, Yi Fang, Jiajin Liu, Djellel Difallah, Qiaoyu Tan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-02568.txt.
Introduction
Dongzhe Fan and colleagues built an assistant in 2025 for graphs whose nodes carry both text and images. Social and product networks often mix written profiles with photos, yet language-model tools for graphs mostly handle text-only nodes, leaving image clues unused. Their model, MLaGA, aims to reason over structure and mixed attributes together.
A structure-aware encoder first aligns text and image features in one space, trained with a joint graph objective. Light projectors then feed those features, plus graph structure, into the language model through multimodal instruction tuning. Across several datasets the assistant beat leading baselines on varied graph tasks. It also transferred to new tasks without full retraining, a sign the alignment generalizes.
Catalog record
| CREATOR | Dongzhe Fan, Yi Fang, Jiajin Liu, Djellel Difallah, Qiaoyu Tan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-02568
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ML a GA : Multimodal Large Language and Graph Assistant Dongzhe Fan, Yi Fang, Jiajin Liu, Djellel Difallah, Qiaoyu Tan Abstract Large Language Models (LLMs) have demonstrated substantial efficacy in advancing graph-structured data analysis. Prevailing LLM-based graph methods excel in adapting LLMs to text-rich graphs, wherein node attributes are text descriptions. However, their applications to multimodal graphs–where nodes are associated with diverse attribute types, such as texts and images–remain underexplored, despite their ubiquity in real-world scenarios. To bridge the gap, we introduce the M ultimodal Large L anguage a nd G raph A ssistant ( MLaGA ), an innovative model that adeptly extends LLM capabilities to facilitate reasoning over complex graph structures and multimodal attributes. We first design a structure-aware multimodal encoder to align textual and visual attributes within a unified space through a joint graph pre-training objective. Subsequently, we implement a multimodal instruction-tuning approach to seamlessly integrate multimodal features and graph structures into the LLM through lightweight projectors. Extensive experiments across multiple datasets demonstrate the effectiveness of MLaGA compared to leading baseline methods, achieving superior performance in diverse graph learning tasks under both supervised and transfer learning scenarios. 1 Introduction Graphs are ubiquitous and versatile data structures that serve as powerful tools for modeling intricate relationships and interactions among entities in various domains, including social networks ( Wu et al. 2020 ) , molecular graphs ( Zhou et al. 2020 ) , and recommendation systems ( Wu et al. 2022 ) . Graph neural networks (GNNs) ( Veličković et al. 2018 ; Hamilton et al. 2018 ) are extensively utilized to integrate information from node attributes and graph structures through message passing and aggregation mechanisms. Despite their widespread application, GNNs are limited to processing numerical features as inputs ( Chen et al. 2024b ) , which constrains their ability to generalize across diverse graphs and tasks ( Jin et al. 2024 ) . As a remedy, recent research has shifted from GNNs to graph LLMs ( Liu et al. 2024b ; Zhang et al. 2024 ; Fang et al. 2024a ; Liu et al. 2024a ; Sun et al. 2025 ; Zhao et al. 2023 ) , which exploit LLMs’ powerful generalization abilities for graph learning tasks by processing nodes’ raw textual attributes ( He et al. 2023 ) . However, integrating graph structures into queries remains a key hurdle. Current methods tackle this either by heuristically converting graphs into node sequences ( Chen et al. 2024a ; Ye et al. 2023 ) or by embedding graph structure into hidden tokens through auxiliary GNNs ( Liu et al. 2024b ; Zhang et al. 2024 ; He & Hooi 2024 ) , which are then inserted into query templates for reasoning. Thanks to the strong text comprehension of LLMs, these methods exhibit superior generalization via in-context learning ( Dong et al. 2022 ) compared to conventional GNNs. Despite recent advances in graph LLMs, their capabilities remain largely confined to text-based graphs, making it difficult to handle multimodal graphs (MMGs), where node attributes span multiple modalities, such as text and images. Yet, MMGs are increasingly common in real-world applications ( Epstein et al. 2023 ; Huang et al. 2022 ) . For instance, in e-commerce platforms, each product node is typically associated with both image and textual descriptions, including user reviews ( Zhu et al. 2024 ) . Integrating these diverse data modalities offers clear benefits for product profiling and user interest modeling. Therefore, it is imperative to push the boundaries of graph LLMs beyond textual attributes. However, building an effective graph LLM framework for multimodal graphs presents several non-trivial challenges. ➀ Effective Integration of Multiple Modalities . Unlike purely textual graphs, MMGs incorporate image, text, and potentially other signals (e.g., audio or sensor data). This variety complicates the alignment of multimodal attributes, which is essential for LLM comprehension. Moreover, the quality of multimodal data can be another stumbling block. As shown in ( Zhu et al. 2024 ) , a node’s textual and image attributes are often not well matched, especially compared to those in Computer Vision(CV) fields ( Radford et al. 2021 ; Li et al. 2023 ) . How to leverage nodes’ rich and multimodal attributes in graph learning is a challenging yet rewarding question. ➁ Structure-aware Multimodal Instruction Tuning . Standard multimodal LLMs, such as LLaVA ( Liu et al. 2023 ) , inject image patch sequences into query templates for visual instruction tuning. However, incorporating graph structures into these templates is difficult due to limited context windows and the heterogeneous nature of node attributes. While previous studies ( Chen et al. 2024a ; He & Hooi 2024 ) employed sampling strategies to extract the most relevant neighbor information, these approaches do not explicitly address the complexities of multimodal node attributes. Then, accounting for the diverse characteristics of node attributes is thus a necessary and beneficial step to fully harness the power of graph LLMs beyond text. To answer the aforementioned questions, we propose a novel M ultimodal Large L anguage a nd G raph A ssistant ( MLaGA ) for MMGs. MLaGA comprises two core innovations: the structure-aware multimodal aligner and the multimodal graph instruction tuning mechanism. The former adapts and integrates each node’s multimodal attributes—textual and visual—into a shared latent space, preserving common and highly informative signals through a cross-graph pre-training objective. The latter unifies nodes’ multimodal attributes and neighborhood information by training modality-aware projectors to map the learned multimodal embeddings into semantic tokens compatible with LLMs. By combining these strategies, MLaGA significantly enhances a graph LLM’s reasoning capabilities beyond textual descriptions, enabling holistic multimodal analysis. Our main contributions can be summarized as follows: \star We investigate how to harness the reasoning capabilities of LLMs within multimodal graphs and present MLaGA. To our best knowledge, MLaGA is the first approach that endows LLMs with multimodal graph reasoning capabilities while preserving their general-purpose nature. \star We devise a two-stage training strategy to achieve MLaGA: (1) Structure-aware multimodal aligner that unifies node’s textual and visual attributes via cross-graph pre-training. (2) Multimodal graph instruction tuning scheme that incorporates these aligned representations into LLMs through modality-aware, versatile projectors, thus endowing the model with strong generalization capabilities for multimodal graph reasoning. \star We conduct extensive experiments across multiple graph benchmark datasets from diverse domains, under both supervised and transfer evaluation settings. Empirical results reveal that MLaGA not only excels on datasets and tasks encountered during training, but also demonstrates robust generalization to previously unseen datasets and tasks without any additional tuning. 2 Related Work Our work is closely related to the following two research directions. Graph LLMs . Large Language Models (LLMs) have catalyzed significant advancements in the field of graph learning, enabling enhanced capabilities in processing and reasoning over complex, structured data. In terms of the exceptional generative capabilities of LLMs, models like GaugLLM ( Fang et al. 2024b ) , advance graph self-supervised learning through augmented node feature by a MoE module. TAPE ( He et al. 2023 ) leverages LLMs to further explain the node text attributes, which serve as new inputs to Graph Neural Networks. Other than the graph node feature augmentation paradigm, graph LLMs also explore effective frameworks to encode graph structural information into LLMs’