GPR: Empowering Generation with Graph-Pretrained Retriever

Xiaochen Wang, Zongyu Wu, Yuan Zhong, Xiang Zhang, Suhang Wang, Fenglong Ma · 2025

Read GPR: Empowering Generation with Graph-Pretrained Retriever by Xiaochen Wang, Zongyu Wu, Yuan Zhong, Xiang Zhang, Suhang Wang, Fenglong Ma (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-00261.txt.

Introduction

Xiaochen Wang and five co-authors present GPR in a 2025 paper on question answering with knowledge graphs. Systems that fetch graph facts before writing an answer need a retriever that truly understands graph shape. Most retrievers start from models trained on plain text, so they miss structure.

GPR is trained directly on knowledge graphs instead. A language model helps expand the graph during training, and a structure-aware goal teaches fine retrieval choices. Tests used two data sets, three language model backbones, and five rival retrievers. GPR improved both the facts fetched and the final answers in each setup. The authors offer it as a sturdy base for graph-based answering systems.

Catalog record

CREATORXiaochen Wang, Zongyu Wu, Yuan Zhong, Xiang Zhang, Suhang Wang, Fenglong Ma
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-00261

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GPR: Empowering Generation with Graph-Pretrained Retriever Xiaochen Wang, Zongyu Wu, Yuan Zhong, Xiang Zhang, Suhang Wang, Fenglong Ma Abstract Graph retrieval-augmented generation (GRAG) places high demands on graph-specific retrievers. However, existing retrievers often rely on language models pretrained on plain text, limiting their effectiveness due to domain misalignment and structure ignorance . To address these challenges, we propose GPR , a graph-based retriever pretrained directly on knowledge graphs. GPR aligns natural language questions with relevant subgraphs through LLM-guided graph augmentation and employs a structure-aware objective to learn fine-grained retrieval strategies. Experiments on two datasets, three LLM backbones, and five baselines show that GPR consistently improves both retrieval quality and downstream generation, demonstrating its effectiveness as a robust retrieval solution for GRAG. 1 Introduction Graph Retrieval-Augmented Generation (GRAG) has emerged as an effective paradigm for enhancing the capabilities of large language models (LLMs) Min et al. 2019 . By retrieving structured and high-quality knowledge from graphs, these models are able to acquire comprehensive context regarding questions and generate more accurate and grounded responses Zhang et al. 2025 . The effectiveness of GRAG hinges on the quality of the retrieved graph components, placing high demands on the retriever. To meet this challenge, retrievers based on pretrained language models (PLMs) Karpukhin et al. 2020 have emerged as a promising solution. These retrievers operate directly on natural language queries without relying on handcrafted rules Mavromatis and Karypis 2024 or task-specific features Luo et al. 2023 , offering greater flexibility and generalizability compared to traditional approaches such as non-parametric or graph neural network-based retrievers Peng et al. 2024 ; Li et al. 2023 . However, despite these advantages, existing PLM-based retrievers exhibit the following shortcomings: S1: Domain Misalignment . Most of the current PLM-based retrievers are built on models pretrained solely on plain text He et al. 2024 ; Li et al. 2023 . These models are proficient in understanding natural language queries, but struggle to interpret graph-structured data, which are composed of semi-structured triplets with irregular formats. The misalignment between query representations in text and the structured nature of graph data leads to suboptimal retrieval, constraining the overall effectiveness of GRAG systems. S2: Structure Ignorance . In addition, many approaches directly apply language models to retrieve individual nodes He et al. 2024 , triplets Li et al. 2023 , or subgraphs Li et al. 2024 ; Hu et al. 2024 , mirroring strategies used in traditional text-based retrieval-augmented generation Karpukhin et al. 2020 . However, this overlooks the fundamental property of knowledge graphs: connectivity. Encoding graph elements as isolated units fails to capture the relational structure essential for effective graph retrieval. To address these limitations, we propose G raph P retrained R etriever ( GPR ), a simple yet effective retriever pretrained directly on knowledge graphs. To resolve S1 , GPR leverages LLM-guided graph augmentation to align natural language questions with relevant subgraphs, without relying on additional supervision or schema-specific features. To tackle S2 , GPR employs a structure-aware pretraining objective that distinguishes triplets based on their relevance with questions, encouraging the model to selectively capture comprehensive context that could boost the LLMs. We evaluate GPR on two benchmark datasets using three backbone LLMs and five baselines. Results show that GPR consistently retrieves more relevant knowledge from graph and enhances downstream generation. These findings establish GPR as a generalizable and effective solution for graph-based retrieval, advancing the development of knowledge-grounded language models. 2 Related Work Retrieval-augmented generation (RAG) ( Gao et al. 2023 ; Guo et al. 2023 ; Ma et al. 2023 ) has emerged as a promising approach to mitigate intrinsic limitations of large language models, such as hallucinations ( Zhang et al. 2023 ; Tonmoy et al. 2024 ) . Its specialized variant, Graph RAG (GRAG) Min et al. 2019 , extends this paradigm by retrieving high-quality knowledge from structured knowledge graphs, demonstrating strong potential in knowledge-intensive tasks ( Zhang et al. 2025 ) . Pretrained Language Model (PLM)-based retrievers Karpukhin et al. 2020 have been widely adopted in GRAG systems, enabling knowledge retrieval at various granularities, including nodes He et al. 2024 , triplets Li et al. 2023 , and subgraphs Li et al. 2024 ; Hu et al. 2024 . In these approaches, knowledge is encoded using language models pretrained on general plain text, and the retrieved results are either directly fed into the large language models for reasoning or processed by additional adaptation modules to enhance the model’s ability to interpret the retrieved content. While most methods simply leverage pretrained language models, some prior work Dong et al. 2023 has explored pretraining these models on knowledge graphs using conventional objectives such as InfoNCE Oord et al. 2018 and Masked Language Modeling Devlin et al. 2019 . Nevertheless, these efforts overlook the alignment between the textual query modality and the graph-structured knowledge, often degrading retrieval effectiveness. 3 G raph P retrained R etriever (GPR) Table 1: Question answering results (%) on WebQSP and CWQ datasets. The best-performing results are highlighted in bold . Methods WebQSP CWQ Accuracy Precision Recall F1 Accuracy Precision Recall F1 ChatGPT 46.62 65.97 39.48 49.40 37.88 42.19 35.63 38.63 + G-Retriever 42.70 63.32 36.44 46.26 33.51 39.42 31.84 34.99 + G-RAG 39.40 57.55 33.63 42.45 31.87 36.44 30.55 33.21 + Hybrid 50.38 64.25 37.52 47.37 39.55 44.52 37.06 40.47 + SKP 44.80 62.10 36.98 46.36 33.43 38.71 31.71 34.83 + Two Tower 39.14 57.30 34.30 42.67 30.85 36.34 29.60 32.64 + GPR 62.40 73.46 46.31 56.79 43.25 47.59 39.59 43.22 LLaMA2-Chat-7B 40.16 59.82 34.85 43.76 28.23 9.99 28.23 14.70 + G-Retriever 44.00 66.22 37.68 47.84 30.77 36.51 29.32 32.52 + G-RAG 20.81 36.43 19.29 25.26 9.53 11.81 9.21 10.31 + Hybrid 53.40 71.44 42.08 52.79 35.12 40.70 33.25 36.61 + SKP 43.93 63.02 37.11 46.61 28.54 33.48 27.04 29.86 + Two Tower 38.51 58.23 33.54 42.44 27.38 32.57 26.12 29.01 + GPR 61.90 77.57 47.44 58.76 44.27 15.41 44.27 22.93 Flan-T5-xl 10.86 19.16 10.40 13.41 12.22 16.94 12.22 14.21 + G-Retriever 21.41 39.37 20.49 26.91 17.97 22.51 17.62 19.76 + G-RAG 19.72 35.20 18.86 24.32 16.89 20.31 16.40 18.17 + Hybrid 31.37 49.82 27.67 35.54 25.14 29.94 24.24 26.77 + SKP 21.29 37.04 20.53 26.45 18.82 22.66 18.32 20.26 + Two Tower 18.00 33.05 17.58 22.83 16.23 20.22 15.90 17.81 + GPR 39.48 56.94 35.23 43.51 28.20 38.54 28.20 32.47 Problem Formulation. We consider the graph retrieval-augmented generation (RAG) setting, where a large language model (LLM) generates answers based on a question q and a retrieved knowledge subgraph \mathcal{S}_{q} . The model takes the pair (q,\mathcal{S}_{q}) as input, where \mathcal{S}_{q} is retrieved from a knowledge graph \mathcal{G} conditioned on q , i.e., \mathcal{S}_{q}=\mathbb{Q}(q,\mathcal{G}) , and \mathbb{Q} denotes the retriever. We define \mathcal{S}_{q} as the union of triplets \tau=(h,r,t) , where each \tau\in\mathcal{G} represents a factual statement relevant to the question. Following prior work Li et al. 2024 , we reduce subgraph retrieval to a triplet ranking task, where the goal is to learn a retriever \mathbb{Q} that assigns higher scores to relevant triplets \tau given the question q . The retriever is optimized to improve downstream generation quality by supplying more informative context to the LLM. Establishing Question-triplet Alignment via Graph Augmentation. In typical retrieval

Read the complete text