Generalization Principles for Inference over Text-Attributed Graphs with Large Language Models

Haoyu Wang, Shikun Liu, Rongzhe Wei, Pan Li · 2025

Read Generalization Principles for Inference over Text-Attributed Graphs with Large Language Models by Haoyu Wang, Shikun Liu, Rongzhe Wei, Pan Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-11836.txt.

Introduction

Haoyu Wang, Shikun Liu, Rongzhe Wei, and Pan Li set out in 2025 rules for making language models reason well over graphs whose nodes carry text. Labeled graph data is often scarce, and large language models promise to fill the gap with their broad knowledge. Two hurdles stand in the way: the models can read only so much neighborhood text at once, and their word space does not line up with graph node codes.

The authors state two principles for sound generalization. First, unify the attribute space with embeddings that adapt to the task, built by language model encoders with task-aware prompts. Second, aggregate graph information with a belief-spreading method whose settings the model estimates and can retune for each new graph. On eleven real benchmarks their system beat prior methods clearly, with task-fit embeddings giving most of the gain.

Catalog record

CREATORHaoyu Wang, Shikun Liu, Rongzhe Wei, Pan Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE76 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-11836

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Generalization Principles for Inference over Text-Attributed Graphs with Large Language Models Haoyu Wang, Shikun Liu, Rongzhe Wei, Pan Li Abstract Large language models (LLMs) have recently been introduced to graph learning, aiming to extend their zero-shot generalization success to tasks where labeled graph data is scarce. Among these applications, inference over text-attributed graphs (TAGs) presents unique challenges: existing methods struggle with LLMs’ limited context length for processing large node neighborhoods and the misalignment between node embeddings and the LLM token space. To address these issues, we establish two key principles for ensuring generalization and derive the framework LLM-BP accordingly: (1) Unifying the attribute space with task-adaptive embeddings , where we leverage LLM-based encoders and task-aware prompting to enhance generalization of the text attribute embeddings; (2) Developing a generalizable graph information aggregation mechanism , for which we adopt belief propagation with LLM-estimated parameters that adapt across graphs. Evaluations on 11 real-world TAG benchmarks demonstrate that LLM-BP significantly outperforms existing approaches, achieving 8.10% improvement with task-conditional embeddings and an additional 1.71% gain from adaptive aggregation. The code 2 2 2 https://github.com/Graph-COM/LLM_BP and task-adaptive embeddings 3 3 3 https://huggingface.co/datasets/Graph-COM/Text-Attributed-Graphs are publicly available. . 1 Introduction Inspired by the remarkable generalization capabilities of foundation models for text and image data ( Achiam et al., 2023 ; Liu et al., 2021 ; Radford et al., 2021 ) , researchers have recently explored extending these successes to graph data ( Liu et al., 2023b ; Mao et al., 2024 ; Zhao et al., 2023a ; Fan et al., 2024 ; He et al., 2024 ) , aiming to develop models that generalize to new or unseen graphs and thereby reduce reliance on costly human annotation ( Li et al., 2024f ; Chen et al., 2024d ; Feng et al., 2024 ; Li et al., 2024g ) . Among various types of graph data, text-attributed graphs (TAGs) have found a wide range of applications. These graphs combine both topological relationships and textual attributes associated with each node, which naturally arises in recommendation systems (where user and item nodes may have textual descriptions or reviews) ( Bobadilla et al., 2013 ) , academic graphs (where publications include extensive textual metadata) ( McCallum et al., 2000 ; Giles et al., 1998 ) , and financial networks (where transactions and accounts come with textual records) ( Kumar et al., 2016 ; Kumar et al., 2018 ) . Given the labeling challenges posed by cold-start problems in recommendation systems or fraud detection in financial networks, methods that can operate with limited labeled data are crucial. In particular, robust zero-shot node labeling across unseen TAGs has become an area of great interest. Figure 1 : The two generalization principles and the framework of LLM-BP. Numerous studies have been dedicated to inference tasks on TAGs. Early efforts have primarily focused on adapting pre-trained language model (LM) encoders ( Li et al., 2024e ; Fang et al., 2024 ) , sometimes in combination with graph neural networks (GNNs) ( Hou et al., 2022 ; Veličković et al., 2018 ) , to incorporate structural information. However, these approaches often struggle to achieve strong generalization performance, largely due to the limited capacity of the underlying models. With the advent of large language models (LLMs) ( Kaplan et al., 2020 ; Huang & Chang, 2022 ) , researchers have proposed two main strategies for integrating LLMs into TAG inference: 1) Direct Node-Text Input. Here, raw node texts are directly fed into LLMs. This method demonstrates reasonably good zero-shot performance on TAGs when text attributes are highly informative for node labels ( Chen et al., 2024c ; Li et al., 2024d ) . However, when the textual attributes are insufficient, it becomes necessary to aggregate information from a larger neighborhood in the graph, while this is constrained by the limited context length LLMs can digest and reason over. 2) Embedding-Based Integration. In this approach, node texts and their neighboring structural information are first encoded into compressed embeddings, which are then processed by LLMs ( Chen et al., 2024b ; Tang et al., 2024a ; Luo et al., 2024 ; Wang et al., 2024 ; Zhang et al., 2024a ) . Because LLMs are not inherently trained on arbitrary embedding spaces, aligning these embeddings with the LLM’s token space is essential - an idea partly inspired by how vision-language models align multimodal data ( Radford et al., 2021 ; Zhai et al., 2023 ) . However, unlike the vision-language domain, where large-scale text–image pairs ( Schuhmann et al., 2022 ) are abundant, the graph domain typically lacks comparable datasets. This scarcity reduces the model’s generalization in practice. In contrast to prior heuristic approaches, this work aims to design a method from first principles for robust zero-shot generalization on TAGs. Because TAGs are inherently multimodal, the proposed method must simultaneously address potential distribution shifts in both textual attributes and graph structure. Specifically, text attributes can vary widely, for example from scientific papers ( McCallum et al., 2000 ) to e-commerce product reviews ( Ni et al., 2019 ) . Edge connection patterns can range from homophilic graphs such as citation networks, where papers on similar themes are linked ( Giles et al., 1998 ) , to heterophilic graphs such as webpages, which connect nodes with distinct topics ( Mernyei & Cangea, 2020 ) . Moreover, the labeling task itself can shift which requires a task-adaptive approach to process both textual features and network structure. Consequently, the core insight behind our model design is grounded in the following two key principles. Principle I: Unifying the text space and obtaining task-adaptive embeddings. LLMs offer powerful text-understanding capabilities that naturally unify the textual feature space, and have recently been shown to benefit from task-adaptive prompting ( Kong et al., 2024 ) . However, to handle the large-scale graph aggregation discussed later, we require these capabilities to extend beyond raw text to an embedding space. Hence, we propose to adopt LLM-based encoder models such as LLM2Vec ( BehnamGhader et al., 2024 ; Li et al., 2024a ) for text embedding. Although this approach might appear to be a naive extension of smaller LM-based embedding methods (e.g., those relying on SBERT ( Reimers, 2019 ) or RoBERTa ( Liu, 2019 ) ), we argue that leveraging the decoder-induced encoder structure of LLMs is essential for achieving task-adaptive embeddings. In particular, we introduce a novel prompting strategy that encodes text attributes conditioned on inference-task descriptions, enabling significantly improved zero-shot inference - an ability not readily achieved by smaller LM-based embeddings. Principle II: Developing a generalizable adaptive graph information aggregation mechanism. Graph structure determines the node neighboring relationships and thus the information aggregation from which nodes may benefit the inference. Inspired by the belief propagation (BP) algorithm ( Murphy et al., 2013 ) that gives the optimal statistical inference over pairwise correlated random variables, we propose to regard the graph as a Markov Random Field (MRF), each node as a random variable, and mimic BP to aggregate information for node label inference. Because BP is rooted in basic mathematical principles, this approach is widely generalizable. Algorithmic adaptivity across different TAGs hinges on estimating the coupling coefficients in the graphs, which can be done by having LLMs analyze

Read the complete text