KG-Hopper: Empowering Compact Open LLMs with Knowledge Graph Reasoning via Reinforcement Learning

Shuai Wang, Yinan Yu · 2026

Read KG-Hopper: Empowering Compact Open LLMs with Knowledge Graph Reasoning via Reinforcement Learning by Shuai Wang, Yinan Yu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-21440.txt.

Introduction

Wang and Yu train compact open language models to answer questions over knowledge graphs. Such tasks need accurate multi-hop reasoning, and stepwise pipelines often pass errors along. Their KG-Hopper framework uses reinforcement learning to fit the whole graph traversal into one thinking stage. The model can weigh cross-step links, explore paths, and backtrack within that single pass. On eight benchmarks, a 7B model beat larger multi-step systems and rivaled strong closed models, the authors report.

Catalog record

CREATORShuai Wang, Yinan Yu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33037
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.21440

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

KG-Hopper: Empowering Compact Open LLMs with Knowledge Graph Reasoning via Reinforcement Learning Shuai Wang, Yinan Yu Abstract Large Language Models (LLMs) demonstrate impressive natural language capabilities but often struggle with knowledge-intensive reasoning tasks. Knowledge Base Question Answering (KBQA), which leverages structured Knowledge Graphs (KGs) exemplifies this challenge due to the need for accurate multi-hop reasoning. Existing approaches typically perform sequential reasoning steps guided by predefined pipelines, restricting flexibility and causing error cascades due to isolated reasoning at each step. To address these limitations, we propose KG-Hopper, a novel Reinforcement Learning (RL) framework that empowers compact open LLMs with the ability to perform integrated multi-hop KG reasoning within a single inference round. Rather than reasoning step-by-step, we train a Reasoning LLM that embeds the entire KG traversal and decision process into a unified “thinking” stage, enabling global reasoning over cross-step dependencies and dynamic path exploration with backtracking. Experimental results on eight KG reasoning benchmarks show that KG-Hopper, based on a 7B-parameter LLM, consistently outperforms larger multi-step systems (up to 70B) and achieves competitive performance with proprietary models such as GPT-3.5-Turbo and GPT-4o-mini, while remaining compact, open, and data-efficient. The code is publicly available at: https://github.com/Wangshuaiia/KG-Hopper . † † aftertitle: I Introduction Large Language Models (LLMs) have achieved remarkable success across diverse domains but still exhibit notable shortcomings, such as hallucinations and factual inaccuracies, particularly in knowledge-intensive tasks. This is largely due to the implicit storage of knowledge within their model weights, making knowledge updates cumbersome and resource-intensive through fine-tuning. To address this limitation, Retrieval-Augmented Generation (RAG) has emerged as an effective strategy, enabling LLMs to dynamically access external knowledge during inference [ 22 ] . Among various external resources, Knowledge Graphs (KGs) stand out as structured and reliable knowledge bases, offering explicit, interpretable, and easily updateable knowledge beneficial in critical applications such as medicine and finance. Knowledge Base Question Answering (KBQA), which aims to leverage structured knowledge from KGs to answer questions, often requires complex multi-hop reasoning—traversing multiple interconnected relationships within a KG [ 21 ] . Current approaches typically follow step-by-step reasoning strategies, sequentially processing entities and relationships from a predefined pipeline. For example, as illustrated in Figure 1 (a), answering the question "What is the official flower of the area affected by Tropical Storm Fabio?" involves first identifying the affected area and subsequently retrieving its official flower. However, such rigid frameworks present critical drawbacks: Fig. 1 : Multi-step vs one-round multi-hop reasoning over a knowledge graph: (a) multi-step reasoning and (b) our one-round reasoning. The multi-step pipeline invokes multiple sequential LLM calls and fails due to the missing entity Yellow Hibiscus , leading to an incorrect path. In contrast, our one-round approach performs the entire reasoning process within a single Reasoning LLM call, maintaining coherence and demonstrating robustness to incomplete knowledge. • Limited Flexibility and Local Optima Susceptibility. Traditional methods sequentially aggregate information through individual inference steps guided by predefined reasoning paths. Consequently, these methods struggle to dynamically adjust when facing incomplete or misleading KG data. For instance, as shown in Figure 1 (a), missing the critical entity "Yellow Hibiscus" could mislead a multi-step method employing beam search into incorrectly choosing the national flower of Mexico, with limited capability for backtracking. • Error Cascading and Reasoning Bias. Stepwise methods inherently propagate errors from earlier steps. Incorrectly choosing a node, such as "Mexico" in Figure 1 (a), directly influences subsequent reasoning steps. Additionally, treating each reasoning step independently neglects inter-step dependencies, causing biases and potential deviations from the original query intent. To mitigate these limitations, two lines of research offer promising advances. First, Reinforcement learning (RL) have demonstrated its effectiveness in navigating discrete and combinatorial decision spaces, such as KG traversal by optimizing reasoning policies through exploration and long-term rewards [ 25 , 18 , 9 ] . Second, recent work on Reasoning LLMs , such as ChatGPT-o1 [ 7 ] and DeepSeek-R1 [ 6 ] , has significantly improved the reasoning capabilities of language models. These models perform a dedicated “thinking” phase before generating answers, where each token in the thought sequence can be viewed as a latent variable. Importantly, later tokens can revise earlier ones, enabling self-reflection and correction within the reasoning process [ 31 ] . This iterative refinement mechanism aligns naturally with the multi-hop nature of complex KBQA tasks, allowing LLMs to reason more coherently and correct potential errors before answer generation. Building on these insights, we propose KG-Hopper , a novel KBQA framework integrating the entire multi-hop reasoning process within a single LLM call, as depicted in Figure 1 (b). Specifically, KG-Hopper leverages RL to enhance the reasoning capabilities of LLMs by training them to utilize KG retrieval tools effectively. KG-Hopper addresses previous limitations with the following advantages: (1) By embedding the complete reasoning chain within a single inference step, KG-Hopper captures cross-step dependencies, mitigating local biases and ensuring coherent reasoning. (2) Integrating reasoning into the Reasoning LLM ’s "thinking" phase allows flexible exploration and effective backtracking, significantly reducing cascading errors without predefined reasoning patterns. (3) Our framework leverages RL to explicitly guide and refine reasoning processes, substantially improving the model’s performance in complex multi-hop scenarios. (4) It remains compact and deployable, using a 7B LLM to achieve competitive results with much larger models. To effectively train KG-Hopper, we initially generate cold-start data and employ Supervised Fine-Tuning (SFT), enabling autonomous KG retrieval. We subsequently strengthen multi-hop reasoning capabilities through targeted RL training, guided by carefully designed reward functions for retrieving and reasoning process. Additionally, we mask the retrieved triples from KG to minimize external knowledge interference and employ history resampling to enhance training efficiency. Our main contributions are as follows: • We introduce a RL framework for LLM–KG integration, combining structured reward signals for retrieval and reasoning with masked supervision over KG content. To the best of our knowledge, this is the first work to apply RL to enable end-to-end KG reasoning within LLMs. • We propose a one-round KBQA framework that performs multi-hop reasoning entirely within the “thinking” phase of a single LLM inference, enabling autonomous and iterative KG traversal without multi-step orchestration. • We achieve strong empirical results on eight KBQA benchmarks: our compact 7B-parameter model consistently outperforms multi-step methods using models up to 70B, and matches or exceeds the performance of proprietary models such as GPT-3.5-Turbo and GPT-4o-mini. II Task Definition We address the task of Knowledge Base Question Answering (KBQA) over a knowledge graph G=\{(e,r,e^{\prime})\mid e,e^{\prime}\in\mathcal{E},r\in\mathcal{R}\} , where each triple (e,r,e^{\prime}) denotes a relation r between two entities e and e^{\prime} . Given a natural language question q , along with an identified topic entity e_{t}\in\mathcal{E} mentioned in the question, the objective is to infer the correct answer entity e_{a}\in\mathcal{E} by reasoning over the knowledge graph G . The answering process typically starts from the topic entity e_{t} and involves exploring the graph to identify the answer entity e_{a} . In practice, large-scale knowledge graphs are often sparse and incomplete, making it difficult to locate the answer through simple one-hop queries. As a result, multi-hop reasoning is frequently required, where the answer entity e_{a} may reside several hops away from the topic entity e_{t} . III Method Our approach leverages reinforcement learning to train LLMs for KG retrieval and reasoning. This

Read the complete text