[Experiment, Analysis, and Benchmark] HCT-QA : A Benchmark for Question Answering on Human-Centric Tables

Mohammad S. Ahmad, Zan Naeem *, Michael Aupetit, Ahmed Elmagarmid, Mohamed Eltabakh, Xiaosong Ma *, Mourad Ouzzani, Chaoyi Ruan *, Hani Al-Sayeh · 2025

Read [Experiment, Analysis, and Benchmark] HCT-QA : A Benchmark for Question Answering on Human-Centric Tables by Mohammad S. Ahmad, Zan Naeem *, Michael Aupetit, Ahmed Elmagarmid, Mohamed Eltabakh, Xiaosong Ma *, Mourad Ouzzani, Chaoyi Ruan *, Hani Al-Sayeh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-20047.txt.

Introduction

Ahmad and eight co-authors build a benchmark for answering questions over the messy tables people actually make. Tables in PDFs and web pages pack dense facts into complex layouts that resist database queries. Their benchmark, HCT-QA, gathers thousands of such human-centric tables with questions and answers.

The set pairs 1,880 real tables with nearly ten thousand questions, plus synthetic tables carrying many more. Tests of 25 language models and 9 vision models show wide gaps in skill. Fine-tuning one model on the benchmark lifted its score by up to 25 points. The authors offer the set, with rich metadata, as a measuring stick for table understanding.

Catalog record

CREATORMohammad S. Ahmad, Zan Naeem *, Michael Aupetit, Ahmed Elmagarmid, Mohamed Eltabakh, Xiaosong Ma *, Mourad Ouzzani, Chaoyi Ruan *, Hani Al-Sayeh
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE113 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-20047

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

[Experiment, Analysis, and Benchmark] HCT-QA : A Benchmark for Question Answering on Human-Centric Tables Thanks: * Work done while at QCRI. Mohammad S. Ahmad, Zan Naeem *, Michael Aupetit, Ahmed Elmagarmid, Mohamed Eltabakh, Xiaosong Ma *, Mourad Ouzzani, Chaoyi Ruan *, Hani Al-Sayeh Abstract Tabular data embedded in PDF files, web pages, and other types of documents is prevalent in various domains. These tables, which we call human-centric tables ( HCTs for short), are dense in information but often exhibit complex structural and semantic layouts. To query these HCTs, some existing solutions focus on transforming them into relational formats. However, they fail to handle the diverse and complex layouts of HCTs, making them not amenable to easy querying with SQL-based approaches. Another emerging option is to use Large Language Models (LLMs) and Vision Language Models (VLMs). However, there is a lack of standard evaluation benchmarks to measure and compare the performance of models to query HCTs using natural language. To address this gap, we propose the Human-Centric Tables Question-Answering extensive benchmark ( HCT-QA ) consisting of thousands of HCTs with several thousands of natural language questions with their respective answers. More specifically, HCT-QA includes 1,880 real-world HCTs with 9,835 QA pairs in addition to 4,679 synthetic HCTs with 67.7K QA pairs. Also, we show through extensive experiments the performance of 25 and 9 different LLMS and VLMs, respectively, in an answering HCT-QA ’s questions. In addition, we show how finetuning an LLM on HCT-QA improves F1 scores by up to 25 percentage points compared to the off-the-shelf model. Compared to existing benchmarks, HCT-QA stands out for its broad complexity and diversity of covered HCTs and generated questions, its comprehensive metadata enabling deeper insight and analysis, and its novel synthetic data and QA generator. I Introduction Answering natural language (NL) questions over structured and semi-structured data is a fundamental problem [ 1 , 2 , 3 , 4 ] . The task of Table QA focuses on retrieving or computing answers to NL questions grounded in tabular data [ 5 , 6 , 7 , 8 , 9 , 10 ] . Early studies used primarily relational tables from Wikipedia and widely used databases [ 6 , 11 , 12 ] . Over time, research has evolved in two directions: (1) Direct question answering using neural models [ 5 , 13 , 14 ] , and (2) translating NL questions into SQL queries (NL-to-SQL) [ 15 , 7 , 8 , 9 , 10 ] . Despite increasing attention to this area, existing efforts remain concentrated on relational or relational-like table structures. Tabular data embedded in PDFs files, web pages, and other types of documents is widespread across many domains. Figure 1 illustrates examples of such data obtained from the Qatar National Planning Council [ 16 ] , scientific publication archives [ 17 ] , the US Census [ 18 ] , and the Pakistan Bureau of Statistics [ 19 ] . These tables employ intricate structural and visual layouts to convey large volumes of information in a concise representation to support strategic decision-making. Fig. 1: Examples of Human-Centric Tables (HCTs) from different real-world data sources exhibiting various levels of structural and semantic complexity including column nesting (L1), row nesting (L2), row group labels (L3), row aggregation (L4), and column aggregation (L5) Unlike traditional relational tables, which have a simple layout, document-embedded tables are primarily designed for human readability rather than automated processing. We refer to this form of tabular data as Human-Centric Tables (HCTs) . HCTs represent a distinct data modality, characterized by intricate layouts, rich semantic content, and significant practical value. However, systematically processing HCTs for tasks such as information extraction, interpretation, and question answering remains a major challenge. With the rapid advances in multimodal document understanding [ 20 , 21 , 22 , 23 ] , the scope of Table QA is progressively extending beyond relational tables [ 10 , 8 , 24 ] . This evolution warrants the need for not only systems capable of robust question answering over HCTs, but also methodologies and benchmarks that enable rigorous evaluation and identification of existing limitations of these systems while answering questions on HCTs. Although a few benchmarks have been recently introduced, they suffer from key shortcomings—most notably, the absence of metadata-driven analyses (i.e., analyses incorporating HCTs and question characteristics), and limited scalability due to the reliance on extensive manual effort. Properties and Challenges of HCTs. HCTs exhibit highly diverse structural features designed for human interpretation. Figure 1 showcases common structural properties observed in HCTs including column nesting (L1), row nesting (L2), and row group labels (L3), as well as embedded aggregations that may appear row-wise (L4) or column-wise (L5). Visual cues-such as variations in text or row color (e.g., L3 in Table (d) and L4 in Table (c)), bold or emphasized text (e.g., L4 in Table (a) and Table (c)), and the positioning of elements within cells (e.g., L2 in Table (b)) convey additional semantic information. These structural and visual complexities make direct question answering with neural QA models challenging. Also, as HCTs are not inherently relational, SQL queries cannot be executed on them directly. Existing efforts, such as AutoTables [ 25 ] , have attempted to transform HCT-like tables into relational formats. However, such approaches are limited to relatively simple layouts, often struggle on complex real-world HCTs. Consequently, the pipeline that first converts HCTs to relational tables and then applies NL to SQL transformations remains fragile and error-prone [ 26 , 10 , 8 , 24 ] . New Opportunities with LLMs and VLMs. Recent advances in Large Language Models (LLMs) and Vision Language Models (VLMs) pave the way for a new LLM-oriented approach for querying HCTs. This approach can (1) efficiently handle HCTs alongside other human-centric data formats (e.g., charts and images), (2) avoid the complex and error-prone NL-to-SQL pipeline, and (3) directly understand NL questions. However, a systematic evaluation to identify the strengths, weaknesses, and potential improvements of LLM-oriented approaches remains largely an uncharted area. To address this gap, we propose HCT-QA , a novel benchmark designed to assess LLMs and VLMs as the engines for querying HCTs. HCT-QA can be also used to evaluated other types of data processing engines designed to answer NL questions on HCTs. TABLE I: Comparison of major Table QA benchmarks with HCT-QA . Benchmark Name Data Type Num HCT Tables Table Modalities Num Sources Num QA Pairs Table Metadata QA Metadata Avg. Questions Per Table Question Syntactic Similarity HiTab [ 27 ] Tables 3,597 JSON 3 10,672 ✗ ✗ 3 0.45 MultiHiertt [ 28 ] Tables + Text 9,842 HTML 2 10,440 ✗ ✗ N/A 0.23 TableVQA-Bench [ 29 ] Tables 205 HTML, Image 1 250 ✗ ✗ 1.2 0.186 TAT-QA [ 30 ] Tables + Text 2,757 2D List 1 12,650 ✗ ✗ N/A 0.2879 HCT-QA (Ours) Tables 6,559 HTML, CSV, Image, MD 4 + Synthetic 77,582 ✓ ✓ 11 0.325 Challenges and Contributions. Devising the proposed benchmark involves several challenges: (1) Collecting and generating HCTs with an extensive diversity of complex layouts (Figure 1 ). (2) Constructing an extensive variety of NL questions ranging from simple selection to complex aggregation and ranking questions. (3) Obtaining accurate ground

Read the complete text