From Factoid Questions to Data Product Requests: Benchmarking Data Product Discovery over Tables and Text

Liangliang Zhang, Nandana Mihindukulasooriya, Niharika S. D’Souza, Sola Shirai, Sarthak Dash, Yao Ma, Horst Samulowitz · 2025

Read From Factoid Questions to Data Product Requests: Benchmarking Data Product Discovery over Tables and Text by Liangliang Zhang, Nandana Mihindukulasooriya, Niharika S. D’Souza, Sola Shirai, Sarthak Dash, Yao Ma, Horst Samulowitz (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-21737.txt.

Introduction

Liangliang Zhang and six co-authors create the first benchmark for a practical enterprise task: assembling data products. A data product is a ready-made bundle of tables and text that answers a business question as a set. Search benchmarks stop at single factoid questions over single tables. Analysts in sprawling data lakes need coherent collections of related assets, not scattered snippets.

Their data set, DPDisc, holds over thirteen thousand validated requests with full provenance. A pipeline built it by clustering related tables and passages from three existing collections. An ensemble of language models drafted professional-grade requests, and phased checks validated them. Sparse, dense, and hybrid retrieval baselines all leave large performance gaps. The gaps vary by domain. That marks structure-aware product discovery as an open problem with a ready-made yardstick.

Catalog record

CREATORLiangliang Zhang, Nandana Mihindukulasooriya, Niharika S. D’Souza, Sola Shirai, Sarthak Dash, Yao Ma, Horst Samulowitz
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-21737

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

From Factoid Questions to Data Product Requests: Benchmarking Data Product Discovery over Tables and Text Liangliang Zhang, Nandana Mihindukulasooriya, Niharika S. D’Souza, Sola Shirai, Sarthak Dash, Yao Ma, Horst Samulowitz Abstract Data products are reusable, self-contained assets designed for specific business use cases. Automating their discovery and generation is of great industry interest, as it enables discovery in large data lakes and supports analytical Data Product Requests (DPRs). Currently, there is no benchmark established specifically for data product discovery. Existing datasets focus on answering single factoid questions over individual tables rather than collecting multiple data assets for broader, coherent products. To address this gap, we introduce DPBench, the first user-request-driven data product benchmark over hybrid table-text corpora. Our framework systematically repurposes existing table-text QA datasets by clustering related tables and passages into coherent data products, generating professional-level analytical requests that span both data sources, and validating benchmark quality through multi-LLM evaluation. DPBench preserves full provenance while producing actionable, analyst-like data product requests. Baseline experiments with hybrid retrieval methods establish the feasibility of DPR evaluation, reveal current limitations, and point to new opportunities for automatic data product discovery research. Code and datasets are available at: https://anonymous.4open.science/r/data-product-benchmark-BBA7/ K eywords Data Product \cdot Benchmark \cdot Table-text Copora \cdot Data Discovery 1 Introduction Data lakes are gaining popularity in the industry as a means to store large volumes of heterogeneous data [ 1 , 2 , 3 ] . The lack of a unifying schema, together with the rapid growth of data assets in modern enterprises, presents significant challenges in discovering, understanding, and utilising the right data for a given task [ 4 ] . For many business use cases, the goal of such data discovery is to find relevant data to address some high-level use case, e.g. “I need data that can help me analyze customer retention patterns across different market segments across quarters”. However, existing approaches to data discovery, such as traditional data catalogs and semantic search systems, focus mainly on low-level technical needs [ 5 , 6 , 4 , 7 ] . While such approaches are effective for locating tables or documents through keyword matching or schema-based filtering, they fall short when it comes to bridging the semantic gap between abstract business requirements and the specific data assets fulfill them. The notion of a Data Product [ 8 , 9 ] – a reusable and discoverable data asset designed to support specific analytical needs – has gained traction in both industry and academia [ 7 , 9 , 10 ] . Rather than being used for basic question answering (QA) over a collection of data, a Data Product aims to package multiple relevant data assets that add value to a high-level business use case. Despite the growing recognition of data products as key building blocks for data-driven decision-making, discovering the right data product from a user’s natural language request remains a difficult and largely unsolved problem. While prior benchmarks have explored complex QA tasks over multiple structured and unstructured data sources [ 11 , 12 , 13 , 14 ] , benchmarks for data product discovery remain unexplored due to the difficulty of constructing data discovery benchmarks for data lakes [ 15 , 9 , 16 ] . To address this gap, we propose repurposing traditional table–text QA datasets to construct new benchmarks for data product discovery. Rather than focusing on single-table QA, we reinterpret these resources at the table level. By clustering similar QA pairs across multiple tables and their associated passages, we simulate broader data products. We then generate high-level Data Product Requests (DPRs) that abstract away from the low-level questions, while the associated tables and passages serve as ground-truth data products. This reframing enables us to systematically transform QA datasets into DPR benchmarks, providing a cost-effective, scalable alternative to manual construction. Contributions. Our work makes four key contributions: (1) formalizing the task of data product retrieval, mapping high-level NL requests (DPRs) to tables and passages; (2) introducing a pipeline that automatically constructs aligned DPR–data product pairs from QA datasets; (3) instantiating the first large-scale Data Product benchmark, derived from HybridQA, TAT-QA, and ConvFinQA with full provenance; and (4) establishing baselines and outlining key challenges for semantically aware data product discovery. Figure 1: Overall workflow of DPBench construction. QA pairs are clustered and grouped based on their focal tables to produce data products . For each data product, multiple LLMs generate corresponding DPRs, validate their quality and refine. 2 Related Work Data Discovery involves identifying, curating, and organizing data assets to support analytical tasks, especially in enterprise environments with large, heterogeneous data lakes comprising structured, semi-structured, and unstructured sources [ 17 , 18 , 19 ] . Traditional systems rely on metadata catalogs and keyword search, but lack semantic understanding, prompting research into automated profiling using schema, distributions, and column statistics [ 20 , 21 ] . Platforms such as Amundsen [ 22 ] and DataHub [ 23 ] provide large-scale data discovery by integrating metadata management, lineage tracking, and user feedback, enabling scalable and context-aware exploration of heterogeneous data assets. Recent advances focus on semantic and context-aware methods, including schema/content-based embeddings and task-oriented frameworks that align datasets with user needs [ 24 , 25 ] , while crowdsourcing and interaction logs help capture organizational knowledge [ 26 ] . Benchmarks such as TableUnion [ 27 ] and open-domain table search tasks [ 28 , 14 ] show the challenge of discovering coherent groups of datasets essential for complex workflows. More recently, large language models (LLMs) have been explored for data discovery. Birdie [ 29 ] improves table retrieval with a differentiable search index and LLM-generated queries. A recent survey [ 30 ] outlines broader opportunities and challenges for LLM-based systems. DiscoveryBench [ 31 ] formalizes multi-step data-driven discovery with tasks from real scientific workflows, highlighting the difficulty of grounding high-level requests in complex data. Table–Text QA datasets evaluate models’ ability to answer natural language questions using both structured tables and unstructured text, evolving from early text-only benchmarks like SQuAD [ 32 ] and TriviaQA [ 33 ] . HybridQA [ 14 ] introduced table-anchored questions requiring reasoning over linked Wikipedia passages, while FeTaQA [ 12 ] enabled free-form answers grounded in tables. Financial QA datasets such as FinQA [ 34 ] , ConvFinQA [ 35 ] , and TAT-QA [ 28 ] integrate numerical reasoning and textual narratives while broader document-based tasks include [ 36 , 37 ] . More recent benchmarks diversify domains (e.g., [ 38 , 11 ] ) and reasoning types (e.g., [ 39 , 40 , 41 ] ) while remaining table-anchored. Despite these advances, most datasets remain centered on single tables or narrow domains. In contrast, our benchmark moves beyond table-specific factoid QA toward simulating data product requests, which require discovering and curating sets of tables and passages and align more closely with realistic analytical workflow requirements. Semantic Search has evolved from traditional keyword-based methods like BM25 [ 42 ] and TF–IDF [ 43 ] , which rely on lexical overlap, to approaches that capture user intent and contextual meaning through dense vector representations learned by neural networks. Pretrained models such as BERT [ 44 ] enabled contextual embeddings, while dual-encoder architectures

Read the complete text