TableDreamer: Progressive and Weakness-guided Data Synthesis from Scratch for Table Instruction Tuning

Mingyu Zheng, Zhifan Feng, Jia Wang, Lanrui Wang, Zheng Lin, Yang Hao, Weiping Wang · 2025

Read TableDreamer: Progressive and Weakness-guided Data Synthesis from Scratch for Table Instruction Tuning by Mingyu Zheng, Zhifan Feng, Jia Wang, Lanrui Wang, Zheng Lin, Yang Hao, Weiping Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-08646.txt.

Introduction

Zheng and co-authors (2025) built TableDreamer, a framework that creates training data to teach language models how to work with tables. Models need many worked examples of table questions. Existing ways of making such data explore only a narrow slice of what tables can ask. They also ignore what the target model is bad at. TableDreamer starts by synthesizing varied tables and questions as seed data. It tests the model, then keeps generating new data aimed at the weaknesses it finds.

Fine-tuned on 27,000 of these synthetic examples, a mid-sized open model gained about 11.6 points of average accuracy across ten table benchmarks. It rose from about 49 to about 61 percent. That beat competing data-making methods that used more training data. The authors release their code and data. They present weakness-guided growth as a more efficient route to capable table models.

Catalog record

CREATORMingyu Zheng, Zhifan Feng, Jia Wang, Lanrui Wang, Zheng Lin, Yang Hao, Weiping Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE74 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-08646

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TableDreamer: Progressive and Weakness-guided Data Synthesis from Scratch for Table Instruction Tuning Mingyu Zheng, Zhifan Feng, Jia Wang, Lanrui Wang, Zheng Lin, Yang Hao, Weiping Wang Abstract Despite the commendable progress of recent LLM-based data synthesis methods, they face two limitations in generating table instruction tuning data. First, they can not thoroughly explore the vast input space of table understanding tasks, leading to limited data diversity. Second, they ignore the weaknesses in table understanding ability of the target LLM and blindly pursue the increase of data quantity, resulting in suboptimal data efficiency. In this paper, we introduce a progressive and weakness-guided data synthesis framework tailored for table instruction tuning, named TableDreamer, to mitigate the above issues. Specifically, we first synthesize diverse tables and related instructions as seed data, and then perform an iterative exploration of the input space under the guidance of the newly identified weakness data, which eventually serve as the final training data for fine-tuning the target LLM. Extensive experiments on 10 tabular benchmarks demonstrate the effectiveness of the proposed framework, which boosts the average accuracy of Llama3.1-8B-instruct by 11.62% ( 49.07 % → 60.69 % 49.07\%\rightarrow 60.69\% ) with 27K GPT-4o synthetic data and outperforms state-of-the-art data synthesis baselines which use more training data. The code and data is available at https://github.com/SpursGoZmy/TableDreamer . † † † This work was done during an internship at Baidu Inc. † † ‡ Corresponding author: Zheng Lin. 1 Introduction Table understanding technique aims to enable models to automatically comprehend tables and complete various table-related tasks Lu et al. 2025 ; Shigarov 2023 . With the recent advancement of large language models (LLMs), the dominant paradigm for table understanding has shifted to instruction tuning general LLMs with tabular task data, leading to the rise of powerful Tabular LLMs Zhang et al. 2024a ; Li et al. 2023 . Figure 1: The comparison of performance and training data volume between TableDreamer and previous table instruction tuning data synthesis methods over 10 tabular benchmarks. In early work on tabular LLMs, instruction-tuning samples were manually collected by human annotators or converted from public datasets using fixed instruction templates. However, the reusing of existing datasets often leads to poor task and data diversity, while human annotation also faces the challenge of prohibitively expensive cost. Therefore, researchers turned to employ LLMs to generate table instruction tuning data. For instance, Zhang et al. 2024b uses GPT-3.5 to generate questions based on benchmark tables, which serve as the training data for fine-tuning CodeLlama Rozière et al. 2024 . The resulting TableLLM model outperforms general LLMs on several tabular benchmarks, demonstrating the potential of synthetic data in table instruction tuning. Although existing data synthesis approaches have achieved commendable performance, they still face two limitations in generating table instruction tuning data. First, existing data synthesis methods are unable to fully explore the vast input space composed of input tables and instructions, leading to limited data diversity. On the one hand, general data generation methods like Self-Instruct Wang et al. 2023 primarily focus on generating unstructured text data, and they did not adequately consider the unique characteristics of structured tables (e.g., diverse table structures, different table formats). As a result, they tend to produce simple tables and instructions of limited tabular tasks. On the other hand, existing studies on tabular LLMs only explore how to synthesize more instructions based on directly available tables from public datasets to improve instruction diversity, but they lack the ability to synthesize more diversified tabular data, which also limits the diversity of the final table instruction tuning data. Second, existing data synthesis methods ignore the LLM’s weaknesses in table understanding ability, resulting in suboptimal efficiency of synthetic data. The combination of the input table and the instruction allows us to easily create a large amount of table instruction tuning data, e.g., we can utilize an LLM to generate dozens of questions based on a single table. However, published studies have indicated that merely pursuing an increase in the quantity of instruction tuning data does not necessarily yield performance improvement Zhou et al. 2023a ; Si et al. 2023 . Given the vast input space for table understanding tasks, it is more efficient to synthesize valuable data points that expose the deficiencies of the target LLM, rather than blindly increase the amount of synthetic data, which may result in both a waste of training resources and a decline in model performance. To address these issues, we introduce a progressive and weakness-guided data synthesis framework for table instruction tuning, named TableDreamer , which can not only generate diverse tables and instructions from scratch, but can also continuously explore the input space under the guidance of newly identified weakness data to more effectively enhance the model performance. As illustrated in Figure 2 , our framework consists of two stages. In stage 1, we first synthesize various table titles of different topics and subtopics, and then employ the LLM to create diverse tables. In stage 2, based on synthetic tables and tabular task descriptions, a group of seed data is generated and will undergo data evolution in three directions. The synthesized new samples are evaluated by LLM-as-a-judge to identify weakness-exposing data, which is used as the seed data for the next round of data evolution. This process can be iterated multiple times, with the accumulated weakness data serving as the final table instruction tuning data. We compare TableDreamer with a series of data synthesis methods, general LLMs and tabular LLMs on 10 tabular benchmarks. As shown in Figure 1 , experimental results demonstrate the effectiveness of the proposed framework, which boosts the average accuracy of Llama3.1-8B-instruct by 11.62% ( 49.07\%\rightarrow 60.69\% ) with 27K GPT-4o synthetic data and outperforms the state-of-the-art data synthesis baselines that use more training data (100K+). We also demonstrate the effectiveness of TableDreamer as data augmentation in the few-shot learning scenario, where only a small number of original training samples are available (e.g., 20 samples for each benchmark). Extensive ablation experiments are conducted to reveal the contributions of different components in the framework (e.g., the influence of weakness data selection and data evolution). We hope this work could establish a strong base for future research on the table instruction tuning data synthesis and help researchers improve models’ table understanding ability especially with limited annotation budget. We conclude our contributions as follows: 1) We introduce a data synthesis framework TableDreamer tailored for table instruction tuning with better data diversity and efficiency, mitigating the limitations of current approaches. 2) We construct and release 27K table instruction tuning data, which include diverse tables and instructions of a wide range of tabular tasks that the current open-source community lacks. 3) We make a systematic investigation of existing methods to show the effectiveness of TableDreamer, which outperforms strong baselines on 10 tabular benchmarks including recent tabular LLMs. Figure 2: The overview of the proposed TableDreamer framework, which includes two stages. In stage 1, we first synthesize table titles based on different topics and subtopic, and then employ the LLM to generate diverse tables covering a wide range of key table attributes such as table structures and sizes. In stage 2, starting from a group of seed data, we perform an iterative exploration

Read the complete text