The benefits of query-based KGQA systems for complex and temporal questions in LLM era

Artem Alekseev, Mikhail Chaichuk, Miron Butko, Alexander Panchenko, Elena Tutubalina, Oleg Somov · 2025

Read The benefits of query-based KGQA systems for complex and temporal questions in LLM era by Artem Alekseev, Mikhail Chaichuk, Miron Butko, Alexander Panchenko, Elena Tutubalina, Oleg Somov (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-11954.txt.

Introduction

Artem Alekseev and co-authors argue that query-based graph question systems still beat plain language models on hard questions. Language models stumble on questions that need many hops or time sense. Query systems instead write a runnable query against a knowledge graph like Wikidata. Their frame builds queries in stages to lift scores on such benchmarks.

Robustness was checked across multi-hop and time-based data sets. A new linking method uses chain-of-thought reasoning to match names and relations. Small language models inside the frame did well, not just giant ones. The gains suggest modular query paths remain strong in the model era. Code and data are posted for others to build on.

Catalog record

CREATORArtem Alekseev, Mikhail Chaichuk, Miron Butko, Alexander Panchenko, Elena Tutubalina, Oleg Somov
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-11954

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The benefits of query-based KGQA systems for complex and temporal questions in LLM era Artem Alekseev, Mikhail Chaichuk, Miron Butko, Alexander Panchenko, Elena Tutubalina, Oleg Somov Abstract Large language models excel in question-answering (QA) yet still struggle with multi-hop reasoning and temporal questions. Query-based knowledge graph QA (KGQA) offers a modular alternative by generating executable queries instead of direct answers. We explore multi-stage query-based framework for WikiData QA, proposing multi-stage approach that enhances performance on challenging multi-hop and temporal benchmarks. Through generalization and rejection studies, we evaluate robustness across multi-hop and temporal QA datasets. Additionally, we introduce a novel entity linking and predicate matching method using CoT reasoning. Our results demonstrate the potential of query-based multi-stage KGQA framework for improving multi-hop and temporal QA with small language models. Code and data : https://github.com/ar2max/NLDB-KGQA-System . 1 Introduction Modern question-answering (QA) systems based on large language models (LLMs) achieve strong performance on open-domain benchmarks [ 13 , 34 , 39 ] . These models rely on retrieving information either from internal models parameters or external memory such as vector storage [ 21 ] . Despite their success, QA systems face limitations in both knowledge retrieval mechanisms. Internally, knowledge gaps arise due to uneven fact distribution in pretraining data – some facts are more popular in pre-training data than others [ 1 ] . Externally, retrieval effectiveness is closely tied to document representation, where issues can stem from dense retrieval methods [ 19 ] or suboptimal initial document knowledge representation. For instance, some studies indicate that models struggle to infer inverted knowledge directly from the data [ 6 ] . Thus LLM-based QA systems often struggle with answering rare, multi-hop, or time-sensitive questions – especially those requiring structured data, despite the possibility of answering such question with internal or external storage [ 18 ] . In contrast, parsing-based QA systems generate structured queries that directly retrieve information from structured knowledge bases (KBs), overcoming the limitations of memory-based approaches. Parsing-based systems construct executable queries, ensuring precise and deterministic retrieval from structured databases. While these approaches face challenges related to compositional generalization and domain adaptation [ 15 , 35 , 17 ] , they offer key advantages in terms of scalability, accuracy, and efficiency when answering open-domain questions that involve structured data [ 43 , 8 ] . In this work, we present a query-based multi-stage pipeline QA system that generates executable queries for retrieving answers from structured knowledge sources. Our system is evaluated across key metrics, including generalization to new domains, rejection ability, and overall system performance on multi-hop and temporal questions. Experimental results demonstrate that our approach achieves frontier performance on selected datasets, surpassing both ChatGPT’s direct answer generation methods and query generation method, which produces executable SPARQL queries. Our research specifically focuses on Wikidata [ 41 ] due to its unique characteristics, which present distinct challenges for QA systems. Unlike DBpedia [ 2 ] , where entities are directly linked to textual literals, Wikidata requires mapping to unique numeric identifiers (Q-IDs), significantly increasing the complexity of Entity Linking. Additionally, Wikidata’s structured representation, including qualifiers and references, introduces higher reasoning complexity in query execution. Moreover, Wikidata is continuously updated, ensuring access to up-to-date information which is crucial factor for real-world applications. To the best of our knowledge, no prior work has conducted a comprehensive evaluation of a query-based KGQA system on Wikidata in comparison with proprietary LLMs. In this study, we evaluate each component of the multi-stage pipeline independently, as well as assess the end-to-end performance of the entire system — from input question to predicted answer entity IDs. Our findings demonstrate the effectiveness of the proposed pipeline across multiple datasets, highlighting its strong generalization capabilities and improved rejection handling. These results underscore the significance of our approach in advancing structured QA systems and bridging the gap between natural language queries and structured knowledge retrieval. 2 Related Work Knowledge graph question answering (KGQA) has been a longstanding area of research in natural language processing (NLP). However, with the emergence of LLMs, KGQA has gained increased utility. Early approaches, which relied on rule-based systems and traditional semantic parsing components, demonstrated high precision for the input questions but suffered from low recall. In contrast, LLMs are capable of processing large volumes of input data and generating accurate queries, although they are often plagued by generalization issues and hallucinations. Modern KGQA systems based on large language models aim to mitigate these issues by leveraging advanced techniques to improve generalization and reduce hallucinations. Query-based KGQA systems were initially trained on text-to-SPARQL pairs, incorporating intermediate steps like predicate classification, entity linking, and slot-filling [ 25 , 27 , 16 ] . These approaches focused on simple one-hop questions but remain relevant due to their structured methodology. Efforts to overcome the one-hop limitation include in-context-learning (ICL) with proprietary LLMs [ 38 ] and fine-tuning approaches [ 5 , 8 ] . Research on ICL improvements includes retrieval-augmented generation with few-shot examples [ 14 ] , SPARQL decomposition into atomic sub-steps [ 22 ] , and interactive multi-hop question decomposition [ 42 ] . Fine-tuning strategies involve augmenting text-to-SPARQL pairs [ 30 ] , structured pre-training [ 29 ] , and incorporating linguistic features like Part-of-Speech tags and dependency parsing [ 28 ] . Ensuring the reliability of SPARQL queries in KGQA involves pre-execution validation to prevent errors. AI-based methods enhance reliability by employing syntactic and semantic query checks [ 44 ] , LLM-driven Chain-of-Thought (CoT) prompting [ 37 ] , error detection [ 33 , 36 ] , and graph-based logical consistency verification [ 4 ] . Transformer models compare generated queries against reference sets to flag logical flaws [ 11 ] , while hybrid symbolic-ML methods integrate rule-based reasoning with machine learning for enhanced accuracy [ 24 ] . These approaches collectively improve KGQA robustness, reducing incorrect or nonsensical query execution before reaching the user. As noted in [ 8 ] , proprietary ICL models like ChatGPT-4 or Claude are expensive due to large input sizes and multiple query requests. We build upon this by developing an multi-stage KGQA system using small fine-tuned models, trainable on consumer-grade GPUs. To our knowledge, no prior research has systematically evaluated the generalization and rejection accuracy of such a system, particularly for Wikidata-based temporal and multi-hop QA. 3 Datasets and Evaluation We evaluate our system on four multi-hop and temporal datasets: LC-QuAD 2.0 [ 12 ] , RuBQ 2.0 [ 31 ] , QALD-10 [ 40 ] , and PAT-Questions [ 26 ] . All samples from selected datasets consists of question, SPARQL, mentioned entities and corresponding Q-IDs. The SPARQL can be executed over Wikidata Query engine 1 1 1 https://query.wikidata.org . The key statistics on the complexity for these datasets are presented in Table 1 . Table 1: The key statistics for selected benchmarks. Dataset Train Test Multi-HOP % Avg. Hops Hops Range QALD-10 402 386 58.3% 2.1 1–8 LC-QuAD 2.0 17,864 4,541 62.9% 1.91 1–4 RuBQ 2.0 1,920 580 10% 1.1 1–2 PAT 4,795 1,199 53.3% 1.68 1–2 LC-QuAD 2.0 is a benchmark for KGQA over Wikidata, evaluating complex SPARQL-based query generation. It includes multi-hop reasoning, boolean queries, count-based queries, and qualifier-dependent questions. QALD-10 is a multilingual benchmark for KGQA,

Read the complete text