ReFactX: Scalable Reasoning with Reliable Facts via Constrained Generation
Riccardo Pozzi, Matteo Palmonari, Andrea Coletta, Luigi Bellomarini ⋆⋆, Jens Lehmann, Sahar Vahdati · 2025
Read ReFactX: Scalable Reasoning with Reliable Facts via Constrained Generation by Riccardo Pozzi, Matteo Palmonari, Andrea Coletta, Luigi Bellomarini ⋆⋆, Jens Lehmann, Sahar Vahdati (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-16983.txt.
Introduction
Riccardo Pozzi and co-authors published a 2025 paper on helping language models answer questions with facts they can trust. Left alone, these models sometimes invent answers when they lack knowledge. A common fix fetches text from outside sources first, but that adds extra models and long inputs. The authors offer a leaner route.
Their method, ReFactX, builds an index of facts from a knowledge graph before the model runs. Each fact is a short statement drawn from linked data about things and their relations. While answering, the model may only produce word sequences that match a real fact in the index. This rule keeps its statements grounded. Tests on question answering showed the method scales to a base of eight hundred million facts. It also adapts to special fields and adds little delay. The authors share their code openly.
Catalog record
| CREATOR | Riccardo Pozzi, Matteo Palmonari, Andrea Coletta, Luigi Bellomarini ⋆⋆, Jens Lehmann, Sahar Vahdati |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-16983
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ReFactX: Scalable Reasoning with Reliable Facts via Constrained Generation Thanks: The paper has been accepted for publication at ISWC 2025. This preprint has not undergone peer review or any post-submission improvements or corrections. Riccardo Pozzi, Matteo Palmonari, Andrea Coletta, Luigi Bellomarini ⋆⋆, Jens Lehmann, Sahar Vahdati Abstract Knowledge gaps and hallucinations are persistent challenges for Large Language Models (LLMs), which generate unreliable responses when lacking the necessary information to fulfill user instructions. Existing approaches, such as Retrieval-Augmented Generation (RAG) and tool use, aim to address these issues by incorporating external knowledge. Yet, they rely on additional models or services, resulting in complex pipelines, potential error propagation, and often requiring the model to process a large number of tokens. In this paper, we present a scalable method that enables LLMs to access external knowledge without depending on retrievers or auxiliary models. Our approach uses constrained generation with a pre-built prefix-tree index. Triples from a Knowledge Graph are verbalized in textual facts, tokenized, and indexed in a prefix tree for efficient access. During inference, to acquire external knowledge, the LLM generates facts with constrained generation which allows only sequences of tokens that form an existing fact. We evaluate our proposal on Question Answering and show that it scales to large knowledge bases (800 million facts), adapts to domain-specific data, and achieves effective results. These gains come with minimal generation-time overhead. ReFactX code is available at https://github.com/rpo19/ReFactX. 1 Introduction Large Language Models (LLMs) have demonstrated a variety of capabilities, ranging from natural language understanding and generation to reasoning, especially when enhanced with techniques like Chain-of-Thought (CoT) [ 28 ] prompting. However, LLMs are prone to hallucinations [ 20 ] and their internal knowledge remains limited to their training data [ 5 , 18 ] . This complicates their application to knowledge-intensive tasks like Question Answering (QA), especially when the tasks require accessing information not available in LLM parametric knowledge, such as time-critical data or corporate data. Retrieval-Augmented Generation (RAG) [ 16 , 30 ] and tool-use [ 31 , 27 , 15 ] represent two prominent families of solutions. They enable LLMs to consult external Knowledge Bases (KBs) or execute external tools to enrich their responses with grounded information. We will refer to these methods as Knowledge-Enhanced QA (KE-QA). They often rely on additional components such as retrievers or Entity Linking systems, or auxiliary models, which are orchestrated in information processing pipelines. However, such pipeline-based or service-based solutions have a number of drawbacks. They require the optimization of a larger number of parameters and make it difficult to effectively back-propagate supervision signals to early components of the pipeline [ 34 ] . These approaches are susceptible to error-propagation [ 14 ] , and knowledge updates may be complex to propagate to all the different components (e.g., to QA and Entity Linking knowledge bases). An interesting technique that may overcome such limitations is constrained generation [ 4 , 23 ] , which restricts the model’s output space during decoding to sequences that satisfy predefined structural, syntactic, or semantic constraints. A few recent approaches applied constrained generation on KE-QA, achieving promising results [ 17 , 19 ] . However, as the application of this technique to KE-QA is quite novel, there is an important research challenge to address to advance these techniques: the scalability to large Knowledge Bases. Can these approaches be applied to large KBs such as Wikidata 1 1 1 https://www.wikidata.org ? In this paper, we present Reliable Fact eXtractor (ReFactX), an approach that addresses this challenge and poses itself as an alternative to pipeline-based solution. ReFactX allows LLMs to integrate external knowledge without depending on additional models or external retrievers, but relying solely on constrained generation, supported by a pre-built prefix-tree index, designed to facilitate and speed-up fact retrieval. At inference time, the LLM is instructed via In-Context Learning (ICL) [ 7 ] to invoke the Fact command when external facts are required. Once the command is recognized, constrained generation is activated: the model produces tokens only along valid paths in the prefix tree, guaranteeing that the output matches a fact from the Knowledge Base. Once an entire KB fact is generated, the decoding mechanism reverts back to normal. It is important to note that every generated token is selected based on LLM’s probability estimates. Constrained generation only narrows the vocabulary to the tokens that form an existing fact in the KB, but always leaves the final choice to the LLM. Motivating Example. We illustrate our approach in Figure 1 , by depicting an example with the question “When was the director of Slumdog Millionaire born?” . The model is instructed to first reason on how to reach the correct answer, and then it starts to acquire facts. Once the Fact command is called, we enable constrained generation (underlined in blue) and we guide the model to generate an existing fact. This way the model is able to first find that “Danny Boyle” is the director of “Slumdog Millionaire”, then to find his birth date, and finally to answer the question correctly. Figure 1: ReFactX Answering an Open-domain Question. The LLM sketches a plan, makes two Fact calls, and—with constrained generation (blue underline)—inserts valid facts from a Wikidata-based prefix tree before giving the final answer. We can summarize the main contributions of this work as follows: • ReFactX : a generic, constrained-generation wrapper that lets any LLM access very large KBs, with no external retrievers or pipelines. • Efficient and scalable : a disk-backed prefix tree holds up to 800M facts and adds only \sim 1 % latency. • Empirical validation : insights on four QA benchmarks suggest that ReFactX can achieve competitive results and obtains accuracy gains of up to 20% at over 90% precision when compared to LLMs answering only with their parametric knowledge. 2 Related Work Several approaches have been proposed to incorporate external knowledge into LLMs [ 8 ] . In our work, we focus on Question Answering (QA)—more specifically, on Knowledge-Enhanced QA (KE-QA) , where a model acquires external knowledge for answering questions. Most of approaches can be classified as input-based KE-QA . They rely on external tools, such as search engines or dense retrievers [ 13 ] , which can be arranged in pipeline-based workflows [ 16 , 30 ] or actively called by LLMs [ 31 , 22 , 15 ] . These approaches usually feed external knowledge as input (either in plain text or with dense embeddings) to the LLMs. Input-based KE-QA approaches, while effective in reducing hallucinations [ 25 ] , require additional models or services for retrieving external information that can introduce delays, especially when calling remote APIs, increase training complexity [ 26 ] , and raise substantially the number of input tokens [ 11 ] , leading to higher response time and resource usage. Other approaches can be classified as memory-based KE-QA . They propose memory modules, separated from model parameters, which make the models’ internal mechanisms interact with external knowledge, e.g., by using cross-attention to incorporate external vectors [ 11 , 29 ] . While memory-based KE-QA approaches fuse external knowledge into LLMs’ generation process more deeply, they typically require architectural modifications, complicating the use of pretrained LLMs. Constrained generation