StructText: A Synthetic Table-to-Text Approach for Benchmark Generation with Multi-Dimensional Evaluation
Satyananda Kashyap, Sola Shirai, Nandana Mihindukulasooriya, Horst Samulowitz · 2025
Read StructText: A Synthetic Table-to-Text Approach for Benchmark Generation with Multi-Dimensional Evaluation by Satyananda Kashyap, Sola Shirai, Nandana Mihindukulasooriya, Horst Samulowitz (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-21340.txt.
Introduction
Firms hold rich facts in tables, but tests that check how well computers pull facts from plain text are scarce and slow to build by hand. Satyananda Kashyap and co-authors built a way to make such tests on its own, in work from twenty twenty-five. Their frame, called StructText, starts from the tables as ground truth.
A plan-then-write pipeline turns each table row into natural text that holds the same facts. Judging then mixes model scores for truth and sense with hard checks on numbers and dates. Tests spanned tens of thousands of cases across many data sets. Models proved true to the facts and rarely made facts up, yet their text often read in ways that block easy fact pulling.
Catalog record
| CREATOR | Satyananda Kashyap, Sola Shirai, Nandana Mihindukulasooriya, Horst Samulowitz |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-21340
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
StructText: A Synthetic Table-to-Text Approach for Benchmark Generation with Multi-Dimensional Evaluation Satyananda Kashyap, Sola Shirai, Nandana Mihindukulasooriya, Horst Samulowitz Abstract . Extracting structured information from text, such as key-value pairs that could augment tabular data, is quite useful in many enterprise use cases. Although large language models (LLMs) have enabled numerous automated pipelines for converting natural language into structured formats, there is still a lack of benchmarks for evaluating their extraction quality, especially in specific domains or focused documents specific to a given organization. Building such benchmarks by manual annotations is labour-intensive and limits the size and scalability of the benchmarks. In this work, we present StructText , an end-to-end framework for automatically generating high-fidelity benchmarks for key-value extraction from text using existing tabular data. It uses available tabular data as structured ground truth, and follows a two-stage “plan-then-execute” pipeline to synthetically generate corresponding natural-language text. To ensure alignment between text and structured source, we introduce a multi-dimensional evaluation strategy that combines (a) LLM-based judgments on factuality, hallucination, and coherence and (b) objective extraction metrics measuring numeric and temporal accuracy. We evaluated the proposed method on 71,539 examples across 49 datasets. Results reveal that while LLMs achieve strong factual accuracy and avoid hallucination, they struggle with narrative coherence in producing extractable text. Notably, models presume numerical and temporal information with high fidelity yet this information becomes embedded in narratives that resist automated extraction. We release a framework, including datasets, evaluation tools, and baseline extraction systems, to support continued research. Our findings highlight a critical gap: Models can generate accurate text but struggle to maintain information accessibility, a key requirement for practical deployment in different sectors and demanding both accuracy and machine processability. † † authors: . VLDB Workshop Reference Format: VLDB 2025 Workshop: Tabular Data Analysis (TaDA). † † This work is licensed under the Creative Commons BY-NC-ND 4.0 International License. Visit https://creativecommons.org/licenses/by-nc-nd/4.0/ to view a copy of this license. For any use beyond those covered by this license, obtain permission by emailing [email protected] . Copyright is held by the owner/author(s). Publication rights licensed to the VLDB Endowment. Proceedings of the VLDB Endowment. ISSN 2150-8097. VLDB Workshop Artifact Availability: The source code, data, and/or other artifacts have been made available at https://huggingface.co/datasets/ibm-research/struct-text https://github.com/ibm/struct-text . 1. Introduction Large language models (LLMs) have unlocked new opportunities in automating knowledge extraction and generation tasks ( Kasner and Dusek 2024 ) across structured and unstructured data. Recent advances in model capabilities enable quality reporting from various sources. Industries increasingly automate and adopt LLM powered workflows for converting databases, sensor feeds and other heterogeneous structured records into human readable insights ( Berenguer et al. 2024 ; Qian et al. 2024 ) . One particularly promising use case for LLM-powered knowledge extraction is to perform text-to-table extraction ( Wu et al. 2022 ; Deng et al. 2024 ) , where free form text is processed into tabular data formats. Converting natural language text into tabular data can allow it to more easily integrate with existing tabular data as well as leverage the ecosystem of mature technologies surrounding efficient indexing and querying over tables. This in turn can enable developers and users to make use of text data more effectively and precisely – for example, rather than relying on full text search or embedding-based similarity metrics to retrieve potentially useful documents to answer a question, we can instead convert the documents to tables and query even large amounts of data effectively using SQL. However, research surrounding the topic of text-to-table extraction is still constrained by the absence of broad, domain-spanning benchmarks and evaluations. Classic datasets such as WebNLG ( Gardent et al. 2017 ) , WikiBio ( Lebret et al. 2016 ) , ToTTo ( Parikh et al. 2020 ) , LogicNLG ( Chen et al. 2020 ) all target a single content domain (typically Wikipedia) and judge mainly surface-form overlap, leaving numerical fidelity and schema coverage untested. Synthetic pipelines like SynthIE ( Josifoski et al. 2023 ) show that LLMs can bootstrap data, but their evaluations likewise remain confined to news or encyclopaedic prose. Industry applications require verifiable and high quality information transfer, especially in highly regulated areas such as the finance, healthcare, and legal domains ( Wu et al. 2023 ; Hardian Health 2023 ; Chaudhary 2025 ) . While some existing work has explored the quality of LLMs in data-to-text workflows ( Kasner and Dusek 2024 ) , most existing systems either focus on structured-to-structured pipelines, such as SQL querying and table QA, or on unstructured-to-unstructured tasks like summarization and open-ended generation. Existing structured to text benchmarks tend to focus on narrow domains rather than mapping real-world complexity. Furthermore, we lack the deeper level evaluations that measure information preservation or factual accuracy creating a disconnect between research benchmarks and practical needs. To facilitate the text-to-table extraction research and evaluation, we introduce StructText , an automated workflow to generate text, extract tables, and evaluate the quality of both structured and unstructured data. Our workflow operates over any existing tabular data, processing it to automatically generate natural language text from table rows, and subsequently evaluate the quality of structured extraction over the synthetically generated text. By generating unstructured text starting from a tabular data source, StructText enables the creation of new datasets to benchmark the quality of text-to-table extraction for any domain, using the original data source as the ground-truth for evaluation. As demonstrated by prior work such as SynthIE ( Josifoski et al. 2023 ) , there is an inherent asymmetry in difficulty for LLMs: generating fluent text from structured data inputs seems to be easier for LLMs than extracting structured information from natural language text. We leverage this asymmetry by first synthesizing natural-language passages from tabular data, then using these synthetic text to rigorously evaluate and advance LLM-based information-extraction capabilities. 1.1. Our contributions This work makes three primary contributions to text-to-table generation benchmarking: Synthetic text and benchmark generation : We present a methodology to automatically process existing tabular data to produce new benchmarking datasets. Given any tabular data source, we leverage LLMs to select meaningful groupings of columns to produce synthetic text for each row. Our method minimizes the need for human effort while enabling the creation of new text-to-table benchmark datasets for any domain. Multi-dimensional evaluation framework : We devise multi-dimensional evaluation criteria for the generated synthetic text. The evaluation measures generation across four critical dimensions: factuality, hallucination, coherence, and numeric/temporal accuracy. Our framework incorporates LLM-as-judge methodology, specifically adapted for structural data scenarios combined with specialized hybrid models for numerical and temporal reasoning. This evaluation approach measures if the generated text is faithful ( Li et al. 2022 ) to the original structured data that is being used as input, i.e. , whether generated text truly preserves source information so that it is possible for a system to extract the same information as in the ground truth. Furthermore, it ensures that the generated text is of high quality and coherent in addition to being factually correct. Benchmark dataset : To support the feasibility of our approach, we produce a large-scale benchmarking dataset spanning practical domains including finance, healthcare, and diverse sources of information originating from Wikidata. Unlike existing simplified academic scenarios, it reflects