MMTU : A Massive Multi-Task Table Understanding and Reasoning Benchmark
Junjie Xing, Yeye He, Mengyu Zhou, Haoyu Dong, Shi Han, Lingjiao Chen, Dongmei Zhang, Surajit Chaudhuri, H. V. Jagadish · 2025
Read MMTU : A Massive Multi-Task Table Understanding and Reasoning Benchmark by Junjie Xing, Yeye He, Mengyu Zhou, Haoyu Dong, Shi Han, Lingjiao Chen, Dongmei Zhang, Surajit Chaudhuri, H. V. Jagadish (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-05587.txt.
Introduction
Tables run much of the world's work, from sheets to databases, yet models that claim skill with them are tested on a narrow slice of tasks. Junjie Xing and eight co-authors build a wide test in 2025. Their benchmark holds more than twenty-eight thousand questions across twenty-five real table tasks.
Past tests stuck close to two tasks: turning words into database queries, and answering questions on a table. Real users, from data engineers to analysts, face far more. The new set spans that breadth, drawn from the work such users do each day. Scores on it show where a model's table skill is real and where it is thin. The authors frame the set as a spur: a fair, hard yardstick should pull the field forward.
Catalog record
| CREATOR | Junjie Xing, Yeye He, Mengyu Zhou, Haoyu Dong, Shi Han, Lingjiao Chen, Dongmei Zhang, Surajit Chaudhuri, H. V. Jagadish |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-05587
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MMTU : A Massive Multi-Task Table Understanding and Reasoning Benchmark Junjie Xing, Yeye He, Mengyu Zhou, Haoyu Dong, Shi Han, Lingjiao Chen, Dongmei Zhang, Surajit Chaudhuri, H. V. Jagadish Abstract Tables and table-based use cases play a crucial role in many important real-world applications, such as spreadsheets, databases, and computational notebooks, which traditionally require expert-level users like data engineers, data analysts, and database administrators to operate. Although LLMs have shown remarkable progress in working with tables (e.g., in spreadsheet and database copilot scenarios), comprehensive benchmarking of such capabilities remains limited. In contrast to an extensive and growing list of NLP benchmarks, evaluations of table-related tasks are scarce, and narrowly focus on tasks like NL-to-SQL and Table-QA, overlooking the broader spectrum of real-world tasks that professional users face. This gap limits our understanding and model progress in this important area. In this work, we introduce MMTU , a large-scale benchmark with over 30K questions across 25 real-world table tasks, designed to comprehensively evaluate models ability to understand, reason, and manipulate real tables at the expert-level. These tasks are drawn from decades’ worth of computer science research on tabular data, with a focus on complex table tasks faced by professional users. We show that MMTU require a combination of skills – including table understanding, reasoning, and coding – that remain challenging for today’s frontier models, where even frontier reasoning models like OpenAI o4-mini and DeepSeek R1 score only around 60%, suggesting significant room for improvement. We highlight key findings in our evaluation using MMTU and hope that this benchmark drives further advances in understanding and developing foundation models for structured data processing and analysis. Our code and data are available at https://github.com/MMTU-Benchmark/MMTU and https://huggingface.co/datasets/MMTU-benchmark/MMTU . 1 Introduction Remarkable progress has been made in foundation models [ 39 , 40 , 68 , 112 , 33 ] , partly thanks to an expanding array of large-scale benchmarking efforts. Prominent examples include benchmarks for general language understanding (e.g., GLUE [ 114 ] , Super-GLUE [ 115 ] , BIG-Bench [ 108 ] , MMLU [ 73 ] , MMLU-pro [ 118 ] ), as well as benchmarks focused on coding and STEM reasoning (e.g., SWE-bench [ 13 ] , GPQA-diamond [ 103 ] , AIME [ 5 ] , LiveCodeBench [ 11 ] ). These efforts have played a critical role in deepening our understanding and accelerating the progress of foundation models. Tables and table-based use cases are central to many real-world applications, including spreadsheets [ 21 , 2 ] , databases [ 17 , 18 ] , and computational notebooks [ 25 , 19 ] , which often require expert-level users such as data engineers, analysts, and database administrators to operate. While LLMs have shown great promise in working with tables [ 22 , 16 , 24 ] , existing evaluations of table-tasks remain narrow in scope – primarily focusing on NL-to-SQL [ 86 , 137 , 128 ] and Table-QA [ 127 , 54 , 49 ] – and fail to reflect the broader spectrum of real-world tasks that professional users face. Table Transform Example task : Table transform by output table (TTBT) Question : Please examine the input table, and a desired output table below. Generate a SQL script that can run on the input to produce the output table, in JSON {"sql": <SQL>} Answer : {"sql": "SELECT Product, SUM(Sales) from table Group By Product ..."} Table Matching Example task : Schema matching (SM) Question : Please inspect the two related tables below, and identify pairs of columns that correspond to the same semantic concept, in JSON {"match": <pairs of match>} Answer : {"match": [(tab-A.Product, tab-B.Item), (tab-A.Sales, tab-B.Revenue), ...] } Data Cleaning Example task : Data imputation (DI) Question : Please review the table below, and predict the value in the <MISSING> cell in the table, in JSON {"missing": <missing value>} Answer : {"missing": "Europe"} Table Join Example task : Equi-join (EJ) Question : Given a database with multiple tables below, identify all key/foreign key join relationships between tables, using JSON {"join": <join key columns>} Answer : {"join": [(tab-A.Department, tab-B.D_id), (tab-A.Emp_id, tab-C.E_id), ...] } Column Transform Example task : Program transform by example (PTBE) Question : Please examine the two example output values in "Output Column" below, and generate Python Pandas code that can run the input to produce the desired output, using JSON {"python": <python>} Answer : {"python": "df['Output'] = "df['Last'] + ', '+ df['First'] + df['Middle'].apply(lambda..." Column Relationship Example task : Arithmetic relationships (AR) Question : Please review the table below, and identify arithmetic relationships that exist between columns, using JSON {"relationship": <relationships>} Answer : {"relationship": ["Profit = Sales - Cost", "Margin = Profit / Sales", ...] } Figure 1 : Example questions sampled from different task categories in MMTU for illustrative purposes. Note that questions in MMTU all follow a standardized triplet format of: (Instruction, Input-table(s), Ground-truth answer). The list of all tasks can be seen in Table 1 . In this work, we introduce MMTU , a large-scale and challenging benchmark comprising 30,647 questions across 25 real-world table tasks. It is designed to comprehensively evaluate models’ ability to understand, reason, and manipulate real tables at the expert-level. The tasks we collect in MMTU are drawn from decades of computer science research on tabular data, with a particular focus on complex tasks that professional users face, as illustrated in Figure 1 . Our evaluation shows that the complex and technical nature of MMTU demands a combination of capabilities – including table understanding, reasoning, and coding – that remain challenging for today’s frontier models. Even the top-performing models, such as OpenAI o4-mini and DeepSeek , achieve only 63.9% and 59.6% on MMTU , suggesting substantial room for improvement, and highlighting MMTU as a strong testbed for models aspiring toward general human-level intelligence, e.g., to meet or surpass the top 10% of skilled adults in diverse technical tasks like characterized in [ 96 ] . We perform extensive experiments benchmarking a large collection of models using MMTU , and performed extensive analysis. Some of the key findings from our evaluation include: • LLMs demonstrate strong potential in understanding and manipulating tabular data. Newer and larger models substantially outperform older and smaller ones, indicating significant advancements in table-related capabilities as captured by MMTU . • Reasoning-focused models, such as DeepSeek and OpenAI o4-mini, show a clear advantage over general-purpose chat models like GPT-4o and Llama. The top reasoning models outperform the top chat models by over 10 percentage points (Table 3 ), underscoring the complexity of the tasks (which often require coding in SQL/Pandas) and the importance of reasoning in MMTU . • Unlike earlier models, today’s frontier models are less sensitive to how tables are formatted and serialized (e.g., markdown/CSV/JSON/HTML), reflecting general progress in models’ abilities in understanding diverse data formats (Figure 8 ). • LLMs still struggle with long table context, or large tables with many rows and columns. Complex tasks requiring holistic reasoning across cell values, especially in the column direction, remains challenging when the table context is long (Figure 6 and Figure 12 ). • LLM performance can degrade under table-level perturbations such as row or column shuffling, even when these changes are supposed to be semantically