SPARQL Query Generation with LLMs: Measuring the Impact of Training Data Memorization and Knowledge Injection
Aleksandr Gashkov, Aleksandr Perevalov, Maria Eltsova, Andreas Both · 2025
Read SPARQL Query Generation with LLMs: Measuring the Impact of Training Data Memorization and Knowledge Injection by Aleksandr Gashkov, Aleksandr Perevalov, Maria Eltsova, Andreas Both (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-13859.txt.
Introduction
Aleksandr Gashkov and co-authors study how well large language models turn plain questions into SPARQL queries, the language used to search knowledge graphs. Their paper appeared in 2025. A worry hangs over such tests: the models may have memorized the test data during training, which would inflate their scores. The team designed a way to measure that effect directly.
They ran models under three setups: no help, added knowledge from the graph, and added knowledge with names hidden. Hiding the names strips out what memorization could supply. Comparing the three scores shows how much of the success comes from memory and how much from real query skill. The authors say this check helps judge whether a question-answering method will travel to new graphs or only shines on familiar ones.
Catalog record
| CREATOR | Aleksandr Gashkov, Aleksandr Perevalov, Maria Eltsova, Andreas Both |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-13859
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SPARQL Query Generation with LLMs: Measuring the Impact of Training Data Memorization and Knowledge Injection Aleksandr Gashkov, Aleksandr Perevalov, Maria Eltsova, Andreas Both Abstract Nowadays, the importance of software with natural-language user interfaces cannot be underestimated. In particular, in Question Answering (QA) systems, generating a SPARQL query for a given natural-language question (often named Query Building) from the information retrieved from the same question is the central task of QA systems working over Knowledge Graphs (KGQA). Due to the rise of Large Language Models (LLMs), they are considered a well-suited method to increase the quality of the question-answering functionality, as there is still a lot of room for improvement, aiming for enhanced quality and trustworthiness. However, LLMs are trained on web data, where researchers have no control over whether the benchmark or the knowledge graph was already included in the training data. In this paper, we introduce a novel method that evaluates the quality of LLMs by generating a SPARQL query from a natural-language question under various conditions: (1) zero-shot SPARQL generation, (2) with knowledge injection, and (3) with “anonymized” knowledge injection. This enables us, for the first time, to estimate the influence of the training data on the QA quality improved by LLMs. Ultimately, this will help to identify how portable a method is or whether good results might mostly be achieved because a benchmark was already included in the training data (cf. LLM memorization). The developed method is portable, robust, and supports any knowledge graph; therefore, it could be easily applied to any KGQA or LLM, s.t., generating consistent insights into the actual LLM capabilities is possible. 1 Introduction Figure 1 : Overview of the proposed evaluation approach and prompting strategies Question Answering (QA) is aimed at providing a user with precise answers to questions formulated in a natural language (NL). KGQA systems are filling up the gap between Linked Data and end-users by transforming NL questions into structured queries (e.g., represented as SPARQL 1 1 1 https://www.w3.org/TR/rdf-sparql-query/ ) to make the information accessible using NL requests. However, translating NL queries into SPARQL is still a challenge in the field of KGQA (cf. [ 4 , 29 , 19 ] ). This task is supposed to be solved with the advent of Large Language Models (LLMs) that have significantly furthered the field of natural language processing (NLP) lately (e.g., [ 21 , 32 , 20 ] ). The rapidly growing number of research papers over the past two years confirms the fact that the possibility of LLMs to efficiently generate machine-readable queries from the questions written in NL is being actively discussed. However, according to [ 23 ] , LLMs often perform well on entities and relationships with high frequencies but face challenges in less popular topics (long-tail knowledge problem). Furthermore, the capabilities of LLMs could be questioned as they show anomalies in terms of memorization concerning the actual ability to generate correct SPARQL queries obeying a defined context. From this context, we derive the following research questions : RQ1 Assuming a perfect knowledge injection (i.e., in-prompt context definition), how well does the SPARQL query generation perform? RQ2 What is the impact of memorization on the SPARQL query generation capabilities of LLMs? In summary, these research questions are intended to shed light on the capabilities of LLMs in generating SPARQL queries so that software engineers can better assess the usefulness of LLMs for non-popular or private knowledge graphs. In this paper, we present a novel method for evaluating the quality of LLMs by generating SPARQL queries from a NL question under various conditions (see Fig. 1 for the general idea): (1) SPARQL generation without additional information provided to the LLM ( zero-shot ), (2) Providing specific and complete knowledge about the information ( knowledge injection ) required to generate the correct SPARQL query (e.g., URI to label mappings), and (3) Reusing knowledge injection with anonymized URIs ( masked knowledge injection ). In contrast to prior contributions, our work focuses on estimating the influence of the training data on the QA quality improved by LLMs. Our method is ultimately aimed at recognizing of the fact whether good results are only achieved because an LLM or benchmark was already included in the training data (which might lead to LLM memorization effects). In our investigation, we assessed our method using the different (regarding the question structures) KGQA datasets, QALD-9-plus [ 18 ] and MCWQ [ 3 ] . In addition, we conducted our experiments on various LLMs of different sizes, ranging from 7B to 123B parameters. This paper is structured as follows. In Section 2 , we summarize related work on the integration of LLMs with KGs for improving semantic parsing and generation of SPARQL queries from a NL questions. Thereafter, in Section 3 , we present an overview of our approach, the used datasets, and LLMs. Section 4 evaluates and analyzes the experimental results and erroneous queries produced by LLMs during the experiments. The obtained results and limitations are discussed in Section 5 . Finally, we conclude our work in Section 6 . Reproducibility statement : The source code and data for experiments are available on GitHub 2 2 2 https://github.com/WSE-research/LLM-generated-SPARQL/ . 2 Related Work The integration of LLMs with various data sources (in particular, knowledge bases, KGs, and the Web of Data) for solving different QA tasks in the last two years has evidenced the high scientific interest in this topic in the research community. Some of the recent approaches try to exploit LLMs as semantic parsers (e.g., [ 7 , 14 , 16 , 17 ] ) directly or by fine-tuning the models. Mecharnia and d’Aquin [ 16 ] presented experiments in fine-tuning LLMs (Llama-3-8B, Llama-2-7B, Llama-3-70B, and Mixtral-8x7B) for the task of NLQ-to-SPARQL transformation on QALD-9-plus [ 18 ] and QALD-10 [ 26 ] . Their approach demonstrated promising results (for QALD, the average Macro F1 is near 60% for all LLMs). Meyer et al. [ 17 ] presented a set of automated benchmarking tasks to assess the basic capabilities of LLMs to deal with SPARQL SELECT queries without special architectures or fine-tuning. Completing experiments on different benchmarks like well-known LC-QuAD 2.0 [ 5 ] , CoyPu-Mini [ 2 ] , Bestiary [ 10 ] and state-of-the-art LLMs families (Claude, Gemini and GPT), the authors conclude that most evaluated LLMs have no significant challenges when perceiving SPARQL SELECT query syntax or its semantics, however, generating the SELECT queries with correct semantics still seems to be a difficult task for the models. Recent research also tried to integrate LLMs with KG and other solutions (e.g., [ 1 , 23 , 10 , 30 ] ). Zhang et al. [ 31 ] introduced Rule-KBQA, a framework that employs learned rules to guide the generation of logical forms. Initially, rules are extracted from existing data, after that the authors employed the Rule-Following Fine-Tuned (RFFT) LLM to generate additional rules, ultimately constructing a comprehensive rule library. Shen et al. [ 23 ] suggested a novel framework – Reasoning with Trees – that reformulates KGQA as a discrete decision-making problem, leveraging Monte Carlo Tree Search to iteratively refine reasoning paths. Zahera et al. [ 30 ] developed an approach to generate SPARQL