RelationalFactQA: A Benchmark for Evaluating Tabular Fact Retrieval from Large Language Models

Dario Satriani, Enzo Veltri, Donatello Santoro, Paolo Papotti · 2025

Read RelationalFactQA: A Benchmark for Evaluating Tabular Fact Retrieval from Large Language Models by Dario Satriani, Enzo Veltri, Donatello Santoro, Paolo Papotti (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-21409.txt.

Introduction

Dario Satriani and co-authors probe whether language models can answer with neat tables of facts, in a 2025 paper. Models often nail a lone fact yet fumble when asked for many rows and columns at once. Their benchmark, RelationalFactQA, pairs plain questions with database queries and gold-standard tables.

Tests vary in how many fields and records the answer needs. Even top models scored under 25 percent factual accuracy on the table tasks. Accuracy fell further as tables grew wider or longer, a failure mode that single-fact tests miss. The benchmark gives the field a hard, clear target for structured recall, a skill that data tools and assistants need daily.

Catalog record

CREATORDario Satriani, Enzo Veltri, Donatello Santoro, Paolo Papotti
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-21409

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RelationalFactQA: A Benchmark for Evaluating Tabular Fact Retrieval from Large Language Models Dario Satriani, Enzo Veltri, Donatello Santoro, Paolo Papotti Abstract Factuality in Large Language Models (LLMs) is a persistent challenge. Current benchmarks often assess short factual answers, overlooking the critical ability to generate structured, multi-record tabular outputs from parametric knowledge. We demonstrate that this relational fact retrieval is substantially more difficult than isolated point-wise queries, even when individual facts are known to the model, exposing distinct failure modes sensitive to output dimensionality (e.g., number of attributes or records). To systematically evaluate this under-explored capability, we introduce RelationalFactQA, a new benchmark featuring diverse natural language questions (paired with SQL) and gold-standard tabular answers, specifically designed to assess knowledge retrieval in a structured format. RelationalFactQA enables analysis across varying query complexities, output sizes, and data characteristics. Our experiments reveal that even state-of-the-art LLMs struggle significantly, not exceeding 25% factual accuracy in generating relational outputs, with performance notably degrading as output dimensionality increases. These findings underscore critical limitations in current LLMs’ ability to synthesize structured factual knowledge and establish RelationalFactQA as a crucial resource for measuring future progress in LLM factuality. 1 Introduction Large Language Models (LLMs) have emerged as powerful tools capable of understanding and generating human-like text. Despite these advances, factuality – the ability of LLMs to provide responses that are truthful and faithful to the real-world knowledge encountered during pre-training – remains a persistent challenge [ 20 , 33 ] . Effectively, a lack of factuality manifests as ‘hallucination’ — the generation of plausible yet incorrect information — a pervasive issue that is still observed in frontier models [ 10 , 1 ] . This issue is particularly critical when LLMs are used in settings demanding high factual precision, such as medical information synthesis [ 44 ] , financial reporting [ 13 ] , scientific data analysis [ 48 ] , or educational content generation [ 23 ] . To evaluate and improve factual performance, the research community has developed a variety of benchmarks. However, existing benchmarks predominantly focus on single-value factuality, where the expected output is a short text span or a single scalar value (e.g., a date or named entity, or a numerical value) [ 49 ] . These tasks often emphasize reasoning complexity (e.g., multi-hop QA or ambiguous phrasing) [ 27 , 50 , 52 ] but overlook a fundamental aspect of factual competence: the ability of LLMs to generate long, coherent outputs directly from their internal parametric knowledge (i.e., the facts stored implicitly within the model’s parameters), without retrieving external documents. In this work, we focus on structured, multi-record, tabular outputs to investigate the factuality of LLMs in synthesizing long sequences of facts. This task is motivated by two main arguments. Output Size Matters. First, experiments highlight that retrieving tabular data from parametric memory presents a significantly greater challenge than recalling isolated cell values, even when the underlying facts are known to the model. For instance, prompting an LLM to return two attributes (e.g., name and state) for US counties yields near-perfect results. However, requesting additional attributes for the same set of counties (e.g., including county area) introduces factual errors in the results. County State Area (sq mi) Los Angeles California 4 751 Cook Illinois 1 635 Maricopa Arizona 8 500 ✗ Q : What is the area of Maricopa county? A : 9 224 sq mi ✓ Crucially, if we then query the LLM for these specific incorrectly reported values in isolation (e.g., “What is the area of Maricopa county?”), the model returns the correct value, demonstrating that the error lies in the generation process, not in the absence of the underlying factual knowledge. Results show that the accuracy of retrieving a specific attribute (e.g., state) degrades linearly (from 1.0 to 0.2) as the total number of concurrently requested attributes increases from one to fifty, regardless of the target attribute’s position in the schema. These findings underscore that the structured, multi-attribute retrieval of factual data is not merely an extension of single-fact recall but a distinct capability with unique failure modes. Moreover, while it is hard to quantify precisely and with fine granularity the quality of the output in unstructured generation tasks [ 15 ] , structured data allows punctual comparison at the single fact (cell) level. Increasing Importance of Tabular Output. Second, we argue that the structured factual retrieval capability of LLMs is both under-explored and essential. Several tasks require not just isolated facts about common world knowledge, but the generation of relational data: lists of entities, comparisons, and collections of items satisfying specific conditions [ 36 , 29 , 43 , 41 ] . This requirement has been reported in sociology [ 45 ] , business use cases [ 53 ] , medical diagnosis [ 4 ] , and financial use cases [ 3 ] . Obtaining tabular data is also increasingly relevant for user-facing applications, such as generating comparative tables of e-commerce products or structuring personalized trip itineraries [ 14 , 47 ] . Yet, current benchmarks fall short in measuring this dimension. Existing datasets that do contain tabular data focus on its role as contextual input to the LLM, in the role of a corpus for question answering or fact checking [ 9 , 35 , 54 , 2 ] . We define the Relational Fact Retrieval task as follows: given a query, the LLM must generate a structured table (rows and columns) containing factual information drawn purely from its parametric memory, prohibiting the use of external tools like web browsers during generation. To address the need for evaluating this capability, we introduce RelationalFactQA , a new benchmark designed to test LLMs’ ability to return factual knowledge in relational (i.e., tabular) form in a closed-books setting. RelationalFactQA probes this capability across several dimensions. The benchmark contains triples with the natural language (NL) question, the corresponding SQL script, and the expected answer in tabular format. For the creation, we combine manually crafted questions (for linguistic variety) with systematically generated ones where the corresponding query complexity (e.g., specific SQL constructs) is controlled. Expected output tables spans from small ones, with few tuples and attributes, to large ones. These dimensions enable analysis of LLMs’ performance across different logical operations (e.g., aggregates, filtering), data types (e.g., numerical, categorical), and retrieval methods (prompts with NL questions vs SQL queries). Through extensive experimentation, we find that although larger models show improvement, the ability to produce correct structured answers remains limited — especially as the number of tuples and attributes increases or the query involves less common facts and numerical conditions. Moreover, we observe that even state-of-the-art models rarely exceed 25% of factual accuracy on our benchmark. To summarize, this paper makes the following contributions: • Task formulation. We introduce Relational Fact Retrieval — the closed-book generation of multi-tuple, multi-attribute tables directly from an LLM’s parametric memory — and clarify how it differs from single-fact recall and context-based table QA. • RelationalFactQA benchmark. We release a 696-question dataset covering nine knowledge domains, each triple-annotated with a natural-language query, its equivalent SQL statement, and a fully verified gold table (avg. 27 rows \times 5 attrs). • Hybrid construction pipeline. Our semi-automatic workflow unifies (i) manual curation from three

Read the complete text