FRASE: Structured Representations for Generalizable SPARQL Query Generation
Papa Abdou Karim Karou Diallo ⋄†⋆ Amal Zouaq ⋄†⋆ · 2025
Read FRASE: Structured Representations for Generalizable SPARQL Query Generation by Papa Abdou Karim Karou Diallo ⋄†⋆ Amal Zouaq ⋄†⋆ (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-22144.txt.
Introduction
Diallo and Zouaq present a study from twenty twenty-five on turning plain questions into database queries that generalize. The target language, SPARQL, queries knowledge bases for factual answers. Existing training sets lean on templates, so models learn shallow matches. They then fail on freely phrased questions with new shapes. Their method enriches questions with frame labels that mark roles in each sentence. They also build a new dataset version with those enrichments added. Tests span recent large language models under varied fine-tuning setups.
Adding frame structure improves query generation across their tests. Gains run largest on unseen templates and on fully rephrased questions. The authors say role structure teaches real generalization, not pattern recall. The dataset and method are offered for wider use.
Catalog record
| CREATOR | Papa Abdou Karim Karou Diallo ⋄†⋆ Amal Zouaq ⋄†⋆ |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-22144
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FRASE: Structured Representations for Generalizable SPARQL Query Generation Papa Abdou Karim Karou Diallo ⋄†⋆ Amal Zouaq ⋄†⋆ Abstract Translating natural language questions into SPARQL queries enables Knowledge Base querying for factual and up-to-date responses. However, existing datasets for this task are predominantly template-based, leading models to learn superficial mappings between question and query templates rather than developing true generalization capabilities. As a result, models struggle when encountering naturally phrased, template-free questions. This paper introduces FRASE ( FRA me-based S emantic E nhancement), a novel approach that leverages Frame Semantic Role Labeling (FSRL) to address this limitation. We also present LC-QuAD 3.0, a new dataset derived from LC-QuAD 2.0, in which each question is enriched using FRASE through frame detection and the mapping of frame-elements to their argument. We evaluate the impact of this approach through extensive experiments on recent large language models (LLMs) under different fine-tuning configurations. Our results demonstrate that integrating frame-based structured representations consistently improves SPARQL generation performance, particularly in challenging generalization scenarios when test questions feature unseen templates (unknown template splits) and when they are all naturally phrased (reformulated questions). 1 Introduction I nformation democratization aims to ease access to the vast amount of factual information stored in Knowledge Bases (KBs) and relational databases. A crucial step towards this goal is the translation of natural language questions into structured queries such as SPARQL, SQL or S-expression Diallo et al. (2024) ; Banerjee et al. (2022) ; Sharma et al. (2025) . Recent advances in Large Language Models (LLMs), particularly decoder-only architectures, have significantly improved the semantic representation of natural language, achieving state-of-the-art results on a variety of natural language understanding tasks. Nevertheless, despite these successes, LLMs remain sensitive to input phrasing and prompt formulation, resulting in limited robustness and generalization across diverse question formulations and unseen patterns ( Leidinger et al., 2023 ; Zheng et al., 2023 ) . A key limitation of current question-to-SPARQL systems lies in the datasets on which they are trained Diallo et al. (2024) ; Reyd & Zouaq (2023) . Many benchmarks rely on rigid, template-based constructions, which lead models to learn surface-level mappings between input questions and query structures. This shortcut learning hinders their ability to generalize to naturally phrased, template-free questions, especially when the input deviates from patterns seen during training. To address these aforementioned challenges, we introduce FRASE ( FRA me-based S emantic E nhancement), a method that augments natural language questions with structured semantic information derived from Frame Semantics. These Frames have proven valuable in enhancing semantic understanding in tasks such as machine reading comprehension ( Guo et al., 2020 ; Flanigan et al., 2022 ; Bonn et al., 2024 ) and information extraction ( Su et al., 2021 ; Li et al., 2024b ; Chanin, 2023 ; Su et al., 2023 ) . As illustrated in Figure 1 , FRASE leverages a two-stage pipeline to detect the frames evoked by the question and identify their associated semantic roles, which are then used to enrich the question representation. A detailed presentation of the process is shown in Section 3.2 . We hypothesize that integrating such structured representation into question-to-query models can improve their ability to abstract and generalize beyond rigid surface forms. To evaluate this hypothesis, we introduce LC-QuAD3.0, a new dataset derived from LC-QuAD2.0 Dubey et al. (2019) , in which each question is annotated with its corresponding frame and frame-element and argument mapping using the FRASE pipeline. We conduct comprehensive experiments with several recent LLMs under different fine-tuning settings, assessing the impact of our semantic augmentation across various dataset splits, including out-of-distribution (OOD) settings. Our contributions are threefold: 1. We propose FRASE , a new method for frame detection and arguments identification that does not rely on manually identified target spans, leveraging a RAG-based system grounded in KB relation semantics and frame definitions. 2. We demonstrate that enriching questions with frame-based structured representations improves generalization in SPARQL query generation. 3. We show that this improvement holds not only for unseen-template test sets but also for challenging, naturally phrased reformulations. Figure 1: Overview of our proposed architecture. 2 Related Works 2.1 SPARQL query generation SPARQL generation from natural language has been extensively studied using both Small and Large Language Models (SLMs and LLMs) ( Diallo et al., 2024 ; Reyd & Zouaq, 2023 ; Sharma et al., 2025 ; Banerjee et al., 2022 ; Emonet et al., 2024 ; Zahera et al., 2024 ) . These models are typically fine-tuned end-to-end, often with enhancements such as copy mechanisms ( Banerjee et al., 2022 ; Diallo et al., 2024 ) or non-parametric memory modules ( Sharma et al., 2025 ) to reduce URI-related errors. While many systems explicitly generate SPARQL queries, others bypass query generation altogether by having LLMs directly produce answers grounded in the knowledge base ( Shavarani & Sarkar, 2024 ; Alawwad et al., 2024 ; Muennighoff, 2022 ) . Prompt engineering has also become a prominent strategy, using few-shot examples or explicit URI context to guide generation ( Luo et al., 2023 ; Muennighoff, 2022 ; Diallo et al., 2024 ) . Models such as Code Llama v2 ( Roziere et al., 2023 ) , Mistral 7B ( Jiang et al., 2023 ) , and Mistral 7B Instruct 1 1 1 https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3 have been widely adopted for such tasks. However, despite promising results, existing methods often under-perform when faced with naturally phrased or paraphrased questions. Studies on LC-QuAD 2.0 show a marked drop in accuracy when models are evaluated on reformulated (template-free) questions, highlighting poor generalization to unseen linguistic patterns ( Diallo et al., 2024 ; Reyd & Zouaq, 2023 ) . 2.2 Frame Semantic Parsing Frame semantic parsing approaches fall into two main categories: generative sequence-to-sequence methods and representation learning techniques leveraging FrameNet’s structured knowledge. Seq2Seq models treat frame parsing as a generation task that includes frame identification and argument extraction ( Sutskever, 2014 ; Raffel et al., 2020 ; Kalyanpur et al., 2020 ; Chanin, 2023 ) . Architectures such as T5 ( Raffel et al., 2020 ) , pre-trained on PropBank ( Kingsbury & Palmer, 2002 ) and FrameNet, are often used with lexical units and data augmentation to improve robustness. Some models adopt multi-task learning with shared encoders and task-specific decoders ( Kalyanpur et al., 2020 ) . Representation learning methods, in contrast, focus on aligning sentence-level or span-level embeddings with candidate frames ( Jiang & Riloff, 2021 ) . These approaches often use Graph Neural Networks ( Wu et al., 2020 ) or contrastive learning ( Ju et al., 2024 ; An et al., 2023 ) to integrate semantic relations among frames, elements, and lexical triggers ( Su et al., 2021 ; Zheng et al., 2022 ; Tamburini, 2022 ) . A major limitation of both paradigms is their reliance on an explicit target lexical unit—an annotation absent in datasets like LC-QuAD 2.0. Without this, frame-based parsing becomes inapplicable, and structured semantic representations cannot be extracted or evaluated in downstream tasks such as SPARQL generation. 3 Methodology 3.1 Data 3.1.1 LC-QuAD2.0 The LC-QuAD 2.0 dataset Dubey et al. (2019) (or LCQ2 for short) is composed of natural language questions paired with their corresponding SPARQL queries. Each entry includes two semantically