TabXEval : Why this is a Bad Table? An eXhaustive Rubric for Table Evaluation
Vihang Pancholi, Jainit Bafna, Tejas Anvekar, Vivek Gupta, Arizona State University, IIIT Hyderabad · 2025
Read TabXEval : Why this is a Bad Table? An eXhaustive Rubric for Table Evaluation by Vihang Pancholi, Jainit Bafna, Tejas Anvekar, Vivek Gupta, Arizona State University, IIIT Hyderabad (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-22176.txt.
Introduction
Vihang Pancholi and colleagues grade machine-made tables the way a careful teacher would: first check the structure, then read the content, then explain the grade. Standard metrics miss small structural and factual slips that make a table misleading. Writing in twenty twenty-five, their framework begins with a rubric mixing multi-level structural descriptors with fine-grained contextual signals. A first phase aligns the candidate table against the reference table's layout; a second compares meaning and wording cell by cell, producing interpretable, detailed feedback rather than one opaque score.
The system was tested on a new benchmark spanning many domains, with realistic table corruptions and human annotations. A sensitivity and specificity analysis backed its robustness across table tasks. Code and data are public. The work targets the growing pipeline of generated reports and data summaries, where tables carry the facts readers trust most.
Catalog record
| CREATOR | Vihang Pancholi, Jainit Bafna, Tejas Anvekar, Vivek Gupta, Arizona State University, IIIT Hyderabad |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-22176
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TabXEval : Why this is a Bad Table? An eXhaustive Rubric for Table Evaluation Vihang Pancholi, Jainit Bafna, Tejas Anvekar, Vivek Gupta, Arizona State University, IIIT Hyderabad Abstract Evaluating tables qualitatively and quantitatively poses a significant challenge, as standard metrics often overlook subtle structural and content-level discrepancies. To address this, we propose a rubric-based evaluation framework that integrates multi-level structural descriptors with fine-grained contextual signals, enabling more precise and consistent table comparison. Building on this, we introduce TabXEval , an eXhaustive and eXplainable two-phase evaluation framework. TabXEval first aligns reference and predicted tables structurally via TabAlign , then performs semantic and syntactic comparison using TabCompare , offering interpretable and granular feedback. We evaluate TabXEval on TabXBench , a diverse, multi-domain benchmark featuring realistic table perturbations and human annotations. A sensitivity-specificity analysis further demonstrates the robustness and explainability of TabXEval across varied table tasks. Code and data are available at https://coral-lab-asu.github.io/tabxeval/ . Figure 1: Sensitivity–Specificity trade-off across metrics. Bubble size reflects the harmonic mean of accuracy, sensitivity, and specificity; color shows F1-score. The dashed line marks ideal balance top-right methods perform best. Green denotes optimal ( Goldilocks ) zone; pink favors sensitivity, yellow favors specificity. 1 Introduction Tables are a ubiquitous data format across critical workflows: budget forecasts, patient dashboards, and experimental logs alike where even a one-cell error can trigger costly re-statements or clinical misinterpretations. As large language models (LLMs) and other neural systems are increasingly tasked with generating or transforming such tables, reliable automatic evaluation becomes a bottleneck. Despite the structured nature of tables, most evaluation metrics treat them as plain text. Metrics like BLEU, ROUGE, METEOR, and chrF rely on n-gram overlap, ignoring row–column alignment and unit consistency Papineni et al. (2002) ; Lin (2004) ; Banerjee and Lavie (2005) ; Popović (2015) . Embedding-based scores such as BERTScore improve semantic sensitivity but overlook structural errors like column swaps Zhang* et al. (2020) . Token-level metrics, including Exact Match and PARENT , address factual grounding but fail under reordered or merged schemas Dhingra et al. (2019) . Structural benchmarks highlight these issues: StructBench exposes failures on partial cell mismatches Gu et al. (2024) , TanQ reveals brittleness under unit conversions Akhtar et al. (2025) , and Data-QuestEval sacrifices structure for corpus-level QA-based comparisons Rebuffel et al. (2021) . Atomic decomposition methods like “Is this a bad table?” improve detection but add opacity and computational cost Ramu et al. (2024) . In contrast, work like THumB shows the benefit of rubric-based human ratings in improving evaluation transparency Kasai et al. (2022) . Taken together, existing metrics tend to emphasize either semantics or structure, but rarely both. They offer limited diagnostic insight, often masking specific error types and failing to provide actionable feedback for model improvement. As shown in image-captioning work like THumB , coupling automatic scores with rubric-based human ratings yields more interpretable and reliable evaluations Kasai et al. (2022) . These findings underscore the need for a rubric-based evaluation framework that explicitly assesses both structural alignment and semantic fidelity. In contrast to single-score metrics that collapse diverse errors—such as schema mismatches, contextual omissions, or subtle content shifts—into a single value, a rubric-based approach offers fine-grained, interpretable feedback. Such granularity is essential for complex or high-stakes tasks, where even minor discrepancies can significantly affect downstream performance. To overcome these challenges, we introduce TabXEval , a novel evaluation framework built on a structured, multi-level rubric that combines high-level structural descriptors with fine-grained contextual signals. TabXEval operates in two phases: TabAlign first performs precise alignment of table elements using both rule-based and LLM-assisted strategies, followed by TabCompare , which conducts detailed semantic and syntactic analysis over the aligned cells. This design allows TabXEval to capture both table-level and cell-level discrepancies that prior metrics often overlook. To rigorously test our rubric and framework, we construct TabXBench , a diverse, synthetic benchmark that emulates realistic table perturbations across multiple domains. TabXBench includes human-annotated ratings grounded in our rubric, serving as a gold standard for evaluating metric sensitivity, specificity, and alignment with human judgment. By providing controlled, interpretable scenarios, TabXBench fills a critical gap in current evaluation practice, enabling robust and explainable assessment of structured table outputs. Unlike prior metrics, TabXEval , as shown in Figure 1 , excels at detecting subtle discrepancies i.e. sensitive and accurately localizing errors between tables i.e. specific enough. We summarize our main contributions below: • We introduce the first rubric integrating multi-level structural descriptors and fine-grained contextual quantification for robust table comparisons. • We propose TabXEval , a two-phase LLM-based table evaluation method that aligns reference tables structurally and compares them semantically and syntactically via our rubric. • We construct TabXBench , a diverse benchmark derived from multi-domain datasets, validating evaluation metrics through structured perturbations and human assessments. • We analyze the strengths and weaknesses of existing evaluation methods via Sensitivity-Specificity Trade-off. • We present TabXEval ’s qualitative and quantitative effectiveness in table generation task, enabling explainable automatic evaluation. 2 TabXEval We establish a transparent and systematic protocol for table evaluation by comparing a reference table—the candidate table produced by a human or an LLM with a ground-truth table. These tables may differ in formatting, interpretation, or unit representation, necessitating a rigorous evaluation framework. For instance, an LLM-generated table might omit entire rows or abbreviate numeric values (e.g., “100k” vs. “100,000”), while a human-curated table may specify units only in the header (e.g., “velocity” vs. “velocity (m/s)”). To address such discrepancies, we design a set of rules based rubrics TabXEval , which aim to improve the reliability and interpretability of table evaluation. Figure 2: End-to-end schematic of TabXEval . (1) TabAlign aligns rows, columns, and cells using deterministic rules plus an LLM refinement loop. (2) TabCompare classifies each aligned cell as extra, missing, or partial and combines the counts with rubric weights (\alpha,\beta,\gamma) . This workflow populated the rubrics and outputs a table-level score and cell-level error trace, enabling fine-grained analysis. 2.1 Evaluation Rubric Towards ensuring consistency and fairness in structure evaluation we provide an exhaustive rubric that enhances clarity and transparency in reference-based table evaluation. To quantify the degree of correctness and coverage of information at the most granular level, we advocate 4 categories of evaluation protocols: Structure Descriptor coarsely assesses the overall structure against ground truth. This component compares information at the table level (missing information, extra information and exact matches), giving a high level description of information integrity. Column Descriptor identifies data types of each column based on missing and extra information. This allows for a fine-grained strategy for evaluating cell values, as tables with heterogeneous columns like dates, numbers, strings etc should not be penalized on the same scale. Cell Level Descriptor looks at both semantic and syntactical representation of cell values. Other methods such as Zhang* et al. (2020) ; Ramu et al. (2024) ; Gu et al. (2024) fail in recognizing different representations of the same data and hence can wrongly penalize correct information. This component of the rubric is necessary to craft an ideal table evaluation framework. Granular Cell Level Difference that determines the magnitude of discrepancies between reference and ground truth table necessary to quantify instances w.r.t. cell level descriptions. It also captures variations by changing the format(e.g. m to