TARGET: Benchmarking Table Retrieval for Generative Tasks

Xingyu Ji, Parker Glenn, Aditya G. Parameswaran, Madelon Hulsebos · 2025

Read TARGET: Benchmarking Table Retrieval for Generative Tasks by Xingyu Ji, Parker Glenn, Aditya G. Parameswaran, Madelon Hulsebos (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-11545.txt.

Introduction

Xingyu Ji and co-authors benchmark table finding for AI answers in a 2025 paper. Much stored value sits in tables, and chat systems that answer from data must first fetch the right one. Text search habits may not carry over to tables.

Their benchmark, TARGET, tests retrievers alone and inside full answer pipelines. Dense embedding retrievers crushed the classic keyword baseline, which fares worse on tables than on prose. Missing titles and other metadata swings changed scores sharply. Performance also swung widely across datasets and tasks. The authors publish the benchmark for builders to test against.

Catalog record

CREATORXingyu Ji, Parker Glenn, Aditya G. Parameswaran, Madelon Hulsebos
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-11545

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TARGET: Benchmarking Table Retrieval for Generative Tasks Xingyu Ji, Parker Glenn, Aditya G. Parameswaran, Madelon Hulsebos Abstract The data landscape is rich with structured data, often of high value to organizations, driving important applications in data analysis and machine learning. Recent progress in representation learning and generative models for such data has led to the development of natural language interfaces to structured data, including those leveraging text-to-SQL. Contextualizing interactions, either through conversational interfaces or agentic components, in structured data through retrieval-augmented generation can provide substantial benefits in the form of freshness, accuracy, and comprehensiveness of answers. The key question is: how do we retrieve the right table(s) for the analytical query or task at hand? To this end, we introduce target : a benchmark for evaluating TA ble R etrieval for GE nerative T asks. With target we analyze the retrieval performance of different retrievers in isolation, as well as their impact on downstream tasks. We find that dense embedding-based retrievers far outperform a BM25 baseline which is less effective than it is for retrieval over unstructured text. We also surface the sensitivity of retrievers across various metadata (e.g., missing table titles), and demonstrate a stark variation of retrieval performance across datasets and tasks. target is available at https://target-benchmark.github.io . 1 Introduction Large Language Models (LLMs) have become an indispensable tool in the knowledge worker’s arsenal, providing a treasure trove of information at one’s fingertips. Retrieval-Augmented Generation (RAG) Lewis et al. 2020 further extends the capabilities of these LLMs by grounding generic dialog using information from external data stores. Despite progress in long-context LLMs, RAG still provides benefits in cost and inference time Li et al. 2024b ; Yu et al. 2024 . Moreover, it allows us to augment generic, off-the-shelf LLMs with proprietary data they haven’t been trained on. Progress on RAG has largely been enabled by benchmarks that help exhaustively evaluate the effectiveness of various methods Yang et al. 2024 ; Muennighoff et al. 2023 . Figure 1: Pipeline of “open domain” question answering over tabular data, in which no tables containing “evidence” for the question are provided. Most research considers the “closed domain” setting, and focuses on query interpretation and augmentation, or table reasoning or query generation (e.g. text-to-SQL). With target we intend to stimulate and facilitate research on the critical retrieval step in the “open domain” setting. While RAG has been extensively explored for free-form text, this is unfortunately not the case for structured data, stored either in relational databases or otherwise. Prior work has shown that structured data is of a different nature, for example regarding data types and dimensionality, requiring dedicated research Cong et al. 2023 . Moreover investigating retrieval of structured data for RAG is important: contextualizing LLMs using frequently updated statistical data sources, such as Data Commons Guha et al. 2023 , or using proprietary relational databases within organizations, can yield rich dividends Radhakrishnan et al. 2024 , all underscoring the need for better models, approaches and evaluation for retrieval over structured data. Another important motivation for research on table retrieval stems from research on LLM-powered interfaces and agentic systems for processing and querying structured data. Most research in this direction, e.g., for question answering Nan et al. 2022 or text-to-SQL Gao et al. 2024 , assumes that a table or relational database is provided, while identifying the relevant table is, in fact, a non-trivial task for a user (or agent). Figure 1 depicts an end-to-end pipeline as we envision: starting with a natural language query (which can be a “lookup” or analytical question), the first step is to interpret and augment the query, for which the retrieval component identifies the relevant tabular data needed to generate a response (which can be in code, natural language, or other format). We find that table retrieval in end-to-end (analytical) query systems is an understudied area, motivating the creation of a benchmark. While there has been initial work exploring open-domain question answering on public table corpora such as Wikipedia Chen et al. 2021 ; Herzig et al. 2021 , this does not represent the full spectrum of data characteristics and tasks for structured data retrieval. The development of a broad and comprehensive benchmark covering diverse tasks and datasets of varying difficulty is therefore key in advancing retrieval systems for structured data. In this paper, we present target : the first benchmark evaluating Table Retrieval for Generative Tasks . With target we provide a consistent and comprehensive framework for evaluating models and pipelines for table retrieval in isolation, as well as end-to-end for downstream tasks. We use target to analyze retrieval methods based on sparse lexical representations Chen et al. 2021 , dense embeddings of metadata Liu 2022 , dense table embeddings Zhang et al. 2025 , and dense row embeddings Kumar et al. 2023 . We find that sparse lexical representations are far less effective for retrieval over tabular data as it is found to be for rich free-form text Muennighoff et al. 2023 . In our analysis with target , we find that dense table- and row- embeddings Zhang et al. 2025 outperform baselines but still show high variation in performance across tasks and datasets. Finally, we highlight the sensitivity of retrievers to the provided metadata inputs (e.g., web page titles) and table data availability (e.g., embedding full tables, column names only, or generated table summaries). Our findings identify a performance gap in retrieval accuracy and robustness across data and tasks, emphasizing the need for more research in this area for which target is an instrumental stepping stone. 2 Related Work Figure 2: Overview of the target benchmark for evaluating table retrieval methods and downstream generation for various datasets across three downstream tasks: tabular question answering, fact verification, and Text-to-SQL. Representation Learning and LLMs for Tables Tables have recently become a modality of interest for representation learning and generative models for tasks such as table understanding Hulsebos et al. 2019 ; Deng et al. 2022 , fact verification Herzig et al. 2020 ; Zhang et al. 2020 , and question answering Herzig et al. 2020 , and more recently text-to-SQL Gao et al. 2024 . These models either deploy LLMs out-of-the-box for tabular data, or develop tailored architectures to capture the properties of tables, which pose specific challenges Cong et al. 2023 . These models typically take one or more tables and a query as input to generate an answer, however, the relevant tables per query can be difficult to identify. target is intended to close this gap and facilitate research on end-to-end querying over tabular data such as text-to-SQL and question answering. Table Retrieval Retrieval of structured data has been studied across use-cases in data management and machine learning. Dataset search where the objective is to find a dataset for a given task (e.g. training a machine learning model or doin data analysis) is a well studied topic in the data management literature Halevy et al. 2016 ; Castelo et al. 2021 . These table retrieval systems typically take a semantic description of the data as input and return the relevant tables. In target we focus on retrieval components embedded into end-to-end query systems, where input queries are natural language queries and the

Read the complete text