GraphICL: Unlocking Graph Learning Potential in LLMs through Structured Prompt Design
Yuanfu Sun, Zhengnan Ma, Yi Fang, Jing Ma, Qiaoyu Tan · 2025
Read GraphICL: Unlocking Graph Learning Potential in LLMs through Structured Prompt Design by Yuanfu Sun, Zhengnan Ma, Yi Fang, Jing Ma, Qiaoyu Tan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-15755.txt.
Introduction
Yuanfu Sun and co-authors build GraphICL, a 2025 benchmark for testing how well plain language models handle graph data through prompts alone. Graphs whose nodes carry text, like citation or product networks, matter widely, and specialized graph-tuned models are usually judged against chat models given naive prompts, an unfair fight.
Their benchmark supplies prompt templates that spell out graph structure and cope with few labels. With those prompts, ordinary large models beat both specialized graph models and graph neural networks when labels were scarce or the domain shifted. The study reframes the field's pecking order: good prompt design may matter more than graph-specific training.
Catalog record
| CREATOR | Yuanfu Sun, Zhengnan Ma, Yi Fang, Jing Ma, Qiaoyu Tan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-15755
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
GraphICL: Unlocking Graph Learning Potential in LLMs through Structured Prompt Design Yuanfu Sun, Zhengnan Ma, Yi Fang, Jing Ma, Qiaoyu Tan Abstract The growing importance of textual and relational systems has driven interest in enhancing large language models (LLMs) for graph-structured data, particularly Text-Attributed Graphs (TAGs), where samples are represented by textual descriptions interconnected by edges. While research has largely focused on developing specialized graph LLMs through task-specific instruction tuning, a comprehensive benchmark for evaluating LLMs solely through prompt design remains surprisingly absent. Without such a carefully crafted evaluation benchmark, most if not all, tailored graph LLMs are compared against general LLMs using simplistic queries (e.g., zero-shot reasoning with LLaMA), which can potentially camouflage many advantages as well as unexpected predicaments of them. To achieve more general evaluations and unveil the true potential of LLMs for graph tasks, we introduce Graph In-context Learning (GraphICL) Benchmark , a comprehensive benchmark comprising novel prompt templates designed to capture graph structure and handle limited label knowledge. Our systematic evaluation shows that general-purpose LLMs equipped with our GraphICL outperform state-of-the-art specialized graph LLMs and graph neural network models in resource-constrained settings and out-of-domain tasks. These findings highlight the significant potential of prompt engineering to enhance LLM performance on graph learning tasks without training and offer a strong baseline for advancing research in graph LLMs. 1 Introduction Text-Attributed Graphs (TAGs), which integrate textual descriptions as node attributes with relational edges, form the foundation for understanding modern complex systems and relationships ( Kipf and Welling, 2016a ; Hamilton et al., 2017 ) . Deep learning-based graph reasoning (GR) approaches, exemplified by graph neural networks (GNNs) ( Li et al., 2021 ; Sun et al., 2021 ; Zhou et al., 2020 ; Tan et al., 2019 ; Wu et al., 2022 ; Reiser et al., 2022 ) , have achieved remarkable success in many TAG-related reasoning tasks, such as node classification ( Fan et al., 2019 ; Shi et al., 2023 ; Wu et al., 2020 ) and link prediction ( Wu et al., 2022 ; Reiser et al., 2022 ; Tan et al., 2023 ; Kipf and Welling, 2016b ) . However, most GNN-based approaches face two major hurdles: 1 Limited generalization across different graphs, particularly in cross-domain scenarios . GNN models are typically tailored to specific graph structures they were originally trained on, and when applied to novel or cross-domain reasoning tasks, they exhibit a marked decline in performance ( Zhao et al., 2023 ; Xu et al., 2024 ) . Resolving this often requires fine-tuning or full retraining, resulting in substantial computational overhead and deployment efforts. 2 Performance depends heavily on labeled training graphs . While GNNs perform well in supervised settings, their efficacy drastically diminishes in limited-label scenarios. Although graph few-shot learning ( Garcia and Bruna, 2017 ) has been introduced to mitigate this issue, it still requires a significant number of related learning tasks to adequately train the model for transfer to unseen tasks. To address these challenges, recent research has shifted from GNNs to graph LLMs ( Tang et al., 2023 ; Chen et al., 2024b ; Zhang et al., 2024 ; Liu et al., 2024b ; He and Hooi, 2024 ; Hu et al., 2024 ; Li et al., 2024b ) , most of them leverage LLMs’ strong generalization capabilities for graph-related tasks through in-context learning (ICL) ( Dong et al., 2022 ) . Recent research on knowledge graph foundation models has also explored the idea of in-context learning for reasoning tasks ( Cui et al., 2024 ; Galkin et al., 2023 ) . The key challenge for graph LLMs is incorporating graph structures into queries. Current approaches tackle this by either heuristically converting graphs into node sequences ( Chen et al., 2024b ; Ye et al., 2023 ) or embedding graph structures into hidden tokens via an auxiliary GNN ( Tang et al., 2023 ; Zhang et al., 2024 ; Liu et al., 2024b ; He and Hooi, 2024 ) , which are then integrated into query templates for graph reasoning. By fine-tuning additional neural components or the general LLM backbones using graph-specific instruction tuning, these specialized methods have demonstrated superior zero-shot ICL capabilities compared to standard GNN studies. Despite the promising advances in specialized graph LLMs, their evaluation often relies on overly simplistic LLM baselines ( Tang et al., 2023 ; Chen et al., 2024b ) , such as zero-shot reasoning with models like LLaMA or ChatGPT. Moreover, these models are typically assessed in in-domain scenarios and struggle to fully utilize limited labeled data (i.e., few-shot ICL), a capability that general-purpose LLM can readily support through effective prompt design. In the absence of a well-constructed evaluation benchmark, comparisons between specialized graph LLMs and general LLMs remain underexplored and poorly designed, potentially camouflaging many advantages as well unexpected predicaments of graph LLMs. This raises important questions: Can general-purpose LLMs be effectively adapted to tackle real-world graph reasoning tasks (e.g., node classification and link prediction) solely through in-context learning? Have we truly made progress in the development of graph LLMs? To address these questions, we propose GraphICL : Benchmarking Large Language Models for Graph Reasoning via I n- C ontext L earning. In our framework, GraphICL refers to the design of task-specific prompts following a unified GraphICL template across diverse graph reasoning tasks. GraphICL facilitates graph reasoning in LLMs by leveraging four core components: task description, anchor node text, structure-aware information, and labeled demonstrations. By incorporating anchor nodes and their k -hop neighbors, we enable zero-shot graph reasoning, utilizing the inherent relationships between proximate nodes. Through strategical selection of neighbors and demonstrations, such as the top M most similar or influential nodes, we optimize few-shot reasoning, releasing the potential of LLMs. GraphICL pushes the boundaries of LLMs’ capabilities in graph tasks, enabling performance that was previously unattainable. Our key contributions are summarized as follows: • Novel Research Problem. We investigate whether better graph reasoning (GR) results can be achieved by simply prompting LLMs through GraphICL, without additional training, and whether this approach can outperform both supervised GNNs and specialized Graph LLMs in both in- and cross-domain scenarios. • A Comprehensive Prompt Benchmark for LLM in Graph Reasoning. Previous comparisons between general LLMs and specialized graph models have been biased by underdeveloped prompts, which fail to harness the full potential of LLMs. We propose GraphICL, a comprehensive prompt set that encompasses graph structure, labeled demonstration, and diverse evaluation tasks. • Systematic Evaluation. We conducted extensive experiments on 9 datasets, encompassing both in-domain and cross-domain scenarios, and benchmarked our approach against state-of-the-art graph LLMs as well as traditional supervised GNN models. Additionally, we performed comprehensive ablation studies to assess the impact of various prompt configurations within the GraphICL framework. • Promising Observations. Our extensive evaluation yielded several valuable insights that can inform the future application of LLMs in graph reasoning, particularly through in-context learning. These findings also establish a solid foundation for advancing research in graph LLMs. 2 Related Work Specialized Graph LLMs. Building on the success of large language models (LLMs), the application of LLMs to graph reasoning tasks has gained considerable attention. The core idea is to incorporate graph structures into queries and then instruction-tune the LLMs using graph-related tasks.