GL-Fusion: Rethinking the Combination of Graph Neural Network and Large Language model

Haotong Yang, Xiyuan Wang, Qian Tao, Shuxian Hu, Zhouchen Lin, Muhan Zhang · 2024

Read GL-Fusion: Rethinking the Combination of Graph Neural Network and Large Language model by Haotong Yang, Xiyuan Wang, Qian Tao, Shuxian Hu, Zhouchen Lin, Muhan Zhang (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-06849.txt.

Introduction

Haotong Yang and co-authors fuse graph neural nets with large language models in one deep build, in a paper from 2024. Past fusions lean one way: text models swallow graphs as tokens and lose form, or graph models squash text into fixed vectors and lose sense.

Their build threads graph message passing straight into transformer layers, so text and form are read at once. A cross focus step reads full node and edge text with no squash. Twin heads let the language side write freely while the graph side calls in one pass. Scores hit top marks on key graph sets.

Catalog record

CREATORHaotong Yang, Xiyuan Wang, Qian Tao, Shuxian Hu, Zhouchen Lin, Muhan Zhang
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-06849

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GL-Fusion: Rethinking the Combination of Graph Neural Network and Large Language model Haotong Yang, Xiyuan Wang, Qian Tao, Shuxian Hu, Zhouchen Lin, Muhan Zhang Abstract Recent research on integrating Large Language Models (LLMs) with Graph Neural Networks (GNNs) typically follows two approaches: LLM-centered models, which convert graph data into tokens for LLM processing, and GNN-centered models, which use LLMs to encode text features into node and edge representations for GNN input. LLM-centered models often struggle to capture graph structures effectively, while GNN-centered models compress variable-length textual data into fixed-size vectors, limiting their ability to understand complex semantics. Additionally, GNN-centered approaches require converting tasks into a uniform, manually-designed format, restricting them to classification tasks and preventing language output. To address these limitations, we introduce a new architecture that deeply integrates GNN with LLM, featuring three key innovations: (1) Structure-Aware Transformers, which incorporate GNN’s message-passing capabilities directly into LLM’s transformer layers, allowing simultaneous processing of textual and structural information and generating outputs from both GNN and LLM; (2) Graph-Text Cross-Attention, which processes full, uncompressed text from graph nodes and edges, ensuring complete semantic integration; and (3) GNN-LLM Twin Predictor, enabling LLM’s flexible autoregressive generation alongside GNN’s scalable one-pass prediction. GL-Fusion achieves outstand performance on various tasks. Notably, it achieves state-of-the-art performance on OGBN-Arxiv and OGBG-Code2, 1 Introduction Research in Graph Neural Networks (GNNs) has long focused on learning from graph with preprocessed vector features, often overlooking the rich textual information contained in raw data. Recently, many studies have recognized that better utilization of these text features can enhance performance. This has led to a focus on text-attributed graphs (TAGs), graphs with node, edge, and graph-level text attributes. In addition, some tasks may contain a task description, questions, and candidate answers in natural language. The combination of GNNs and pretrained Large Language Models (LLMs), which can efficiently encode these text attributes and information, has garnered significant interest. Two main approaches have emerged for combining GNNs and LLMs: GNN-centered methods and LLM-centered methods. GNN-centered methods focus on typical graph tasks like node classification and link prediction. These methods convert text into representation vectors for graph nodes or edges using LLMs, which are then fed into GNNs to produce final predictions. GNN-centered approaches excel at capturing structural information. However, compressing rich textual features into fixed-length vectors results in information loss. Moreover, The importance of different text features varies depending on the task. For example, in wiki knowledge graph (KG) datasets, entity descriptions may include a person’s name, nationality, or occupation. A name might be crucial for identifying relatives but irrelevant for finding their workplace. Additionally, GNN architectures cannot generate natural language, making it difficult to perform tasks like question answering. LLM-centered methods, on the other hand, use a multi-modal approach by projecting graph representations into the language space, allowing LLMs to generate text answers to graph-text questions. However, this approach faces similar challenges. Graph tasks vary widely in domains, and current LLM-centered models lack dynamic encoding by GNNs that adapt to tasks, so these models struggles to generalize to diverse datasets or tasks. Furthermore, language supervision is often indirect and less efficient compared to direct graph supervision, making training more difficult. For example, LLMs, with their autoregressive nature, cannot simultaneously predict all nodes in a graph, unlike GNNs, which can process nodes in parallel. Moreover, GNNs focus on classifying nodes into a finite set of classes, while LLMs operate in the much larger and noisier token space of natural language, complicating the training process. Figure 1: Workflow of the GL-Fusion model. (a) An example of a text-attributed graph, where each node and edge has a text attribute that will be encoded by the LLM. (b) The encoded text-attributed graph nodes ( <node> in the figure), along with a prompt or question in natural language, are merged into an input sequence for our GL-Fusion model. GL-Fusion consists of several structure-aware transformer blocks and standard transformer blocks. Cross-attention layers are inserted to retrieve the complete, uncompressed node and edge text. The output on graph nodes is further processed by graph readout components, while the output on standard text tokens predicts the next token, similar to an original autoregressive LLM. To handle tasks involving both graphs and text, we introduce Graph-Language Fusion (GL-Fusion), an architecture that combines the strengths of GNN-centered and LLM-centered models. As shown in Figure 1 (a), graphs are paired with node and edge text features to solve tasks described in natural language. In Figure 1 (b), the task description is included as part of the input text sequence, and a special token <node> is added for each graph node. Each node’s feature is generated by a text encoder based on its text attribute, which is then incorporated into the sequence. To process this combination of graph and text tokens, our architecture includes: • Structure-Aware Transformer Layers : We modify the original attention layers for language input to graph structure-aware attention with both graph and text inputs, which maintains causality for language generation and further enabling transformer to encode graph structure. Different from graph transformer, which encodes graph only and cannot process text, our transformer can naturally encodes graph tokens conditioning on the text tokens before it in sequence, and encodes text tokens after the graph based on the graph. This two-way information exchange between graph and text overcomes the limitations of older models, where either graph or text embedding is irrelevant from the other, and achieves better expressivity. • Graph-Text Cross-Attention : Previous models often compress either graph or text information into a single vector, leading to information loss. In our model, graph structure is preserved across each transformer layer without compression. To avoid compressing textual features, we introduce Graph-Text Cross-Attention layers, where attention is applied between node text and the main sequence (which includes the <node> tokens). These layers retain the complete text information of each node, allowing the model to more precisely extract task-relevant information by focusing on the most important tokens. • GNN & LLM Twin-Predictor : Different tasks require different prediction mechanisms. For traditional GNN tasks, such as node classification, having the LLM generate predictions for each node would slow down inference and be constrained by graph size. In such cases, direct readout from the GNN is more efficient. Conversely, tasks that require natural language outputs benefit from the autoregressive capabilities of LLMs. Unlike previous models, which rely on either a GNN or LLM predictor, GL-Fusion retains both as predictors. During training, supervision can be applied to both models, and during inference, both outputs are generated simultaneously. Users can choose the most appropriate output for the task or consider both outputs together. With these modifications, we propose GL-Fusion, a unified architecture that achieves competitive performance by integrating structural and textual information. To demonstrate this, we conducted experiments on various tasks, including basic graph property prediction, node classification, knowledge graph completion, commonsense question answering, and code graph-to-text generation. Our GL-Fusion achieves outstanding performance across all these datasets, outperforming various combinations of GNNs and LLMs. Notably, our model even achieves state-of-the-art performance on two Open Graph Benchmark ( Hu et al., 2020 ) datasets: ogbn-arxiv and ogbg-code2. These results demonstrate the powerful capability of our GL-Fusion architecture. 2 Preliminary For a tensor

Read the complete text