Beyond Chains: Bridging Large Language Models and Knowledge Bases in Complex Question Answering
Yihua Zhu, Qianying Liu, Akiko Aizawa, Hidetoshi Shimodaira · 2025
Read Beyond Chains: Bridging Large Language Models and Knowledge Bases in Complex Question Answering by Yihua Zhu, Qianying Liu, Akiko Aizawa, Hidetoshi Shimodaira (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-14099.txt.
Introduction
Yihua Zhu, Qianying Liu, Akiko Aizawa, and Hidetoshi Shimodaira proposed PDRR in 2025, a framework for answering complex questions from a knowledge base. Language models alone can be outdated, opaque, and prone to inventing facts. Methods that fetch from a knowledge base mostly handle simple chain-like questions and lack planning.
PDRR works in four stages. It predicts the question type, splits the question into structured triples, retrieves matching facts, and then guides the model to reason over and complete those triples. Across several model backbones it beat existing methods. It did well on both chain-shaped and more tangled complex questions, the authors report.
Catalog record
| CREATOR | Yihua Zhu, Qianying Liu, Akiko Aizawa, Hidetoshi Shimodaira |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-14099
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Beyond Chains: Bridging Large Language Models and Knowledge Bases in Complex Question Answering Yihua Zhu, Qianying Liu, Akiko Aizawa, Hidetoshi Shimodaira Abstract Knowledge Base Question Answering (KBQA) aims to answer natural language questions using structured knowledge from KBs. While LLM-only approaches offer generalization, they suffer from outdated knowledge, hallucinations, and lack of transparency. Chain-based KG-RAG methods address these issues by incorporating external KBs, but are limited to simple chain-structured questions due to the absence of planning and logical structuring. Inspired by semantic parsing methods, we propose PDRR : a four-stage framework consisting of P redict, D ecompose, R etrieve, and R eason. Our method first predicts the question type and decomposes the question into structured triples. Then retrieves relevant information from KBs and guides the LLM as an agent to reason over and complete the decomposed triples. Experimental results show that our proposed KBQA model, PDRR, consistently outperforms existing methods across different LLM backbones and achieves superior performance on various types of questions. 1 Kyoto University 2 University of Tokyo 3 NII LLMC 4 RIKEN AIP [email protected], [email protected], [email protected], [email protected] Code — https://github.com/YihuaZhu111/PDRR Extended version — https://arxiv.org/abs/2505.14099 Introduction Figure 1: Drawbacks of ToG (a chain-based KG-RAG approach). ToG and similar chain-based KG-RAG methods lack a planning module for explicit reasoning control and are limited to chain-type questions due to their insufficient logical structuring. Our PDRR framework resolves both issues. Knowledge bases (KBs) offer rich, structured repositories of world knowledge, where facts are organized as (subject, relation, object) triples. Large-scale KBs such as Freebase ( Bollacker et al. 2008 ) , DBpedia ( Lehmann et al. 2015 ) , Wikidata ( Pellissier Tanon et al. 2016 ) , and YAGO ( Suchanek, Kasneci, and Weikum 2007 ) provide useful resources for downstream applications. Knowledge base question answering (KBQA) leverages this structured information to translate natural language question queries into precise, verifiable answers, which poses challenges since it requires accurate multi-hop reasoning. This capability is essential for domains that require accurate and verifiable retrieval of facts. In response to the costly human effort required to annotate training data, recent studies have turned to leveraging large language models (LLMs) based methods for KBQA, for example, IO ( Brown et al. 2020 ) , CoT ( Wei et al. 2022 ) , and SC ( Wang et al. 2022 ) . These methods harness broad, pre-trained knowledge of LLMs and exhibit strong generalization across datasets. Nonetheless, LLM’s knowledge may not reflect recent or domain-specific facts; they could also exhibit hallucinations, which is difficult to audit or verify, leading to unfaithful and unsupported response. To address these limitations, knowledge-graph retrieval-augmented generation (KG-RAG) methods further extend LLMs with retrieved structural knowledge. Various studies such as ToG ( Sun et al. 2023 ) , GoG ( Xu et al. 2024 ) , and CoK ( Li et al. 2023 ) leverage chain-based KG-RAG approaches that performs reasoning over retrieved chains of KG facts. Specifically, they infer intermediate “bridge” entities at each hop, which repeatedly fetches triplets linked to the current entities from the KG, and uses the LLM to choose which relation and entity to retain for the next step. These chain-based KG-RAG methods ground multi-step reasoning in explicit KG paths, thereby mitigating outdated knowledge, reducing hallucinations through factual grounding and rendering each inference step to be transparent. Despite their strengths, existing chain-based KG-RAG methods exhibit two principal limitations as shown in Figure 1 . First, they reason over the question in a holistic manner and lack a mechanism to structure the inference into targeted sub-tasks. In the upper half “Lack of Plan” example, one should first identify the notable person and then determine the nation they governed. Instead, ToG treats the query as an indivisible whole, leading it to select the superficially matching location.containedby relation rather than the correct people.place_live.person . Second, chain-based approaches cannot handle richer logical structures beyond simple, linear hops, such as conjunction questions that require the intersection of multiple constraint sets. Logic structure refers to the structural form of reasoning required to answer a question. In the lower half “Lack of Logical” example, the answer hinges on intersecting (a) countries bordering France and (b) countries with airports serving Nijmegen. ToG’s linear, single-path search prunes away additional candidates after the first hop, making it incapable of resolving the conjunction and thus yielding an incorrect result. Motivated by these challenges, we propose the P redict– D ecompose– R etrieve– R eason (PDRR) framework. Inspired by pre-LLM semantic-parsing (SP-based) methods ( Lan et al. 2022 ) that translate questions into executable KB logic forms, PDRR introduces a planning module (Predict, Decompose) for structuring multi-step inference, followed by a retrieval and reasoning module (Retrieve, Reason) that grounds the plan in KB facts and guides the LLM to execute it step by step, effectively simulating logical form execution over KGs. Finally, the question answering module utilizes the reasoning triples provided by the previous step to generate an answer. First, our Predict stage classifies each query by type (i.e., chain or parallel) to determine an appropriate reasoning strategy and plan structure before retrieval begins. Next, the Decompose stage converts the question into a set of partial KG triples that reflect the plan, which breaks the overall query into manageable inference units, ensuring each reasoning step is focused and auditable. The Retrieve stage issues targeted KB lookups to fill in missing triple elements, which grounds the planned inference steps in factual knowledge. Finally, the Reason stage leverages the LLM to verify and complete each triple in accordance with the plan. By unifying logical-form-style planning with LLM-driven execution, PDRR supports a variety of reasoning patterns while preserving transparency. We evaluate PDRR on four standard KBQA benchmarks: CWQ ( Talmor and Berant 2018 ) , WebQSP ( Berant et al. 2013 ) , Simple Question ( Bordes et al. 2015 ) , and GrailQA ( Gu et al. 2021 ) . Experimental results show that our proposed KBQA model, PDRR, consistently outperforms existing methods across different LLM backbones and achieves superior performance on various types of questions, demonstrating the effectiveness of PDRR’s explicit planning module and the precise control over retrieval and reasoning modules. Related Work Semantic Parsing-Based Methods Before LLMs, KBQA was dominated by SP-based approaches, which generate logical forms to query structured KBs. CBR-KBQA ( Das et al. 2021 ) combines a non-parametric memory of question–logical form pairs with a parametric retriever to guide logical form generation. TemplateQA ( Zheng et al. 2018 ) bypasses complex parsing by matching questions to a large set of binary templates. SPARQA ( Sun et al. 2020 ) uses a skeleton grammar and a BERT-based coarse-to-fine parser to handle complex questions. While interpretable, SP-based methods are limited by incomplete KBs and the need for additional model training. LLM Retrieval Augmented Methods The emergence of LLMs has led to LLM-only approaches that require no additional training and leverage internalized general knowledge to mitigate KB incompleteness, such as IO ( Brown et al. 2020 ) , CoT ( Wei et al. 2022 ) , and SC ( Wang et al. 2022 ) . Despite their simplicity, these methods struggle with outdated knowledge, limited reasoning transparency, and hallucinations. To mitigate these issues, LLM+KG approaches have been