CoReTab : Improving Multimodal Table Understanding with Code-driven Reasoning

Van-Quang Nguyen, Takayuki Okatani · 2026

Read CoReTab : Improving Multimodal Table Understanding with Code-driven Reasoning by Van-Quang Nguyen, Takayuki Okatani (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-19193.txt.

Introduction

Van-Quang Nguyen and Takayuki Okatani teach models to answer table questions with visible reasoning. Tables in images and documents hold much of the world's data. Yet models trained on past table sets give short factual answers with no steps shown. Brief answers hide how the model got there and often miss the mark.

Their method, CoReTab, drives reasoning through code, so each step can be checked as it runs. Code based steps turn a guess into a trace a user can audit. Better table skill matters for finance, science, and any office work where numbers live in grids.

Catalog record

CREATORVan-Quang Nguyen, Takayuki Okatani
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-19193

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CoReTab : Improving Multimodal Table Understanding with Code-driven Reasoning Van-Quang Nguyen, Takayuki Okatani Abstract Existing datasets for multimodal table understanding, such as MMTab, primarily provide short factual answers without explicit multi-step reasoning supervision. Models trained on these datasets often generate brief responses that offers insufficient accuracy and limited interpretability into how these models arrive at the final answer. We introduce CoReTab , a code-driven reasoning framework that produces scalable, interpretable, and automatically verifiable annotations by coupling multi-step reasoning with executable Python code. Using the CoReTab framework, we curate a dataset of 115K verified samples averaging 529 tokens per response and fine-tune open-source MLLMs through a three-stage pipeline. We evaluate the resulting model trained on CoReTab across 17 MMTab benchmarks spanning table question answering, fact verification, and table structure understanding. Our model achieves significant gains of +6.2%, +5.7%, and +25.6%, respectively, over MMTab-trained baselines, while producing transparent and verifiable reasoning traces. These results establish CoReTab as a robust and generalizable supervision framework for improving multi-step reasoning in multimodal table understanding. Figure 1: (Left-top) Typical response annotation from MMTab 61 which provides limited reasoning supervision. (Left-bottom) Average length of responses of three datasets. (Middle) A pair of ⟨table image, question⟩. (Right) Our CoReTab annotation example with step-by-step reasoning and executable Python code. 1 Introduction Tables serve as a primary medium for organizing and communicating (semi-)structured information, with applications spanning finance, science, government, and diverse practical domains ( 18 ; 42 ) . In real-world settings, tables are often embedded as screenshots in documents such as slides, PDFs, or spreadsheets, making the table inherently a visual object. Text-only large language models (LLMs) are ill-suited for such inputs, as extracting text can disrupt spatial layout and introduce inconsistencies. Moreover, directly interacting with table images (e.g., cropping or highlighting) is often more natural and efficient than relying on OCR followed by complex textual representations such as HTML or LaTeX. Prior work shows that even OCR+LLM pipelines substantially underperform multimodal large language models (MLLMs) on table image understanding ( 61 ) . Consequently, growing attention has focused on multimodal table understanding, which aims to enable machines to comprehend and reason over table images ( 22 ; 31 ; 61 ; 64 ) . Recent advances in multimodal large language models ( 2 ; 36 ; 3 ) have moved us closer to general-purpose multimodal table understanding, yet their performance remains constrained. A central bottleneck lies in supervision. Human annotation, as in ( 5 ; 9 ; 38 ) , requires annotators to generate question–answer pairs for each table image, making it accurate but prohibitively costly and slow. Automated synthesis with MLLMs has been explored as an alternative ( 58 ; 33 ) , where the model itself is instructed to generate ⟨question, answer⟩ pairs, given table images. While scalable, this approach struggles with complex table images, resulting in hallucinations and reasoning errors. Another line of work leverages table codes (textual representations of tables in HTML, L a T e X , or Markdown) that can be rendered into images and paired with generated annotations. 61 curated MMTab, a large-scale multimodal dataset of 232K ⟨table image, question, answer⟩ samples by constructing from existing textual table datasets, partially employing a large language model to paraphrase the answers. 64 collected more table codes and scaled training resources to 1.8M samples using LLMs to generate question–answer pairs directly from table codes. The annotations in MMTab and SynTab are designed to primarily consist of short answers without multi-step reasoning (Figure 1 , left). Models trained on such supervision may excel at question answering but offer little transparency into how these models arrive at the final answer. For real-world applications, where correctness and interpretability are essential, this gap is critical. It is feasible to use (M)LLMs to automatically generate multi-step reasoning annotations; however, validating the correctness of lengthy reasoning text remains challenging and costly. We address this challenge with CoReTab , a Co de-driven Re asoning framework for multimodal Ta ble understanding. CoReTab couples natural-language multi-step reasoning with executable code, enabling automatic verification of reasoning correctness. In this framework, an LLM annotator (e.g., Qwen3 ( 55 ) ) is prompted with a table code, a question, and a short ground-truth answer to produce a new annotation that includes both multi-step reasoning and corresponding Python code (Figure 1 , right). The Python code is executed and verified against the ground-truth answer, and only consistent samples are retained. Multi-step reasoning text associated with verified code tends to be more accurate and less prone to hallucination. As a result, this closed-loop, code-verifiable framework allows scalable generation of accurate, interpretable reasoning annotations with minimal human effort. Using this framework, we curate a CoReTab dataset, an augmented version of MMTab containing 115K verified samples across 11 tasks, with average responses being on average an order of magnitude longer than those in MMTab or SynTab. Each annotation includes step-by-step reasoning and executable code, ensuring both interpretability and verifiability. Moreover, the process is cost-efficient: while annotating 2,000 examples for TableBench 54 required an estimated $12,000, CoReTab scales to 115K samples at a tiny fraction of that cost using local GPU inference. To evaluate the effectiveness of CoReTab , we fine-tune Qwen2.5-VL ( 3 ) using a three-stage training pipeline with LoRA ( 14 ) : (1) table recognition pretraining, (2) instruction tuning on CoReTab , and (3) reinforcement learning optimization. The resulting model consistently outperforms MMTab-trained baselines (e.g., Table-Qwen2.5-VL) across 17 MMTab benchmarks, with average improvements of +6.2% in table question answering, +5.7% in table fact verification, and +25.6% in table structure understanding. Overall, the code-driven reasoning supervision enhances robustness—by teaching models to couple reasoning with executable code—and interpretability, as each prediction provides multi-step reasoning that can be inspected and verified. We summarize our contributions as follows: • We introduce CoReTab , the first framework for code-driven reasoning supervision with automatic verification for multimodal table understanding. • We curate a dataset of 115K verified annotations combining natural-language and executable reasoning, offering richer supervision than existing MMTab and SynTab datasets. • We demonstrate that the resulting MLLM trained under CoReTab supervision achieves substantial improvements across 17 MMTab benchmarks, delivering more accurate, interpretable, and verifiable answers. 2 Related Work 2.1 Multimodal Table Understanding Datasets A variety of datasets have been introduced for multimodal table understanding (MTU), such as table recognition 63 ; 62 , MMCoQA 22 , and MMTab 61 . Among them, MMTab is the most comprehensive, covering 24 tasks across four categories: Table Question Answering (TQA) 38 ; 9 ; 29 ; 65 ; 16 ; 35 ; 15 , Table Fact Verification (TFV) 5 ; 12 ; 1 , Table-to-Text generation (T2T) 37 ; 53 , and Table Structure Understanding (TSU). Other works such as UniMMQA 31 , MMTabQA 32 , and MMTBench 45 further integrated visual features into table images for table-based QA and reasoning tasks. More recently, SynTab 64 scaled training resources to 1.2M samples by employing LLMs to synthesize question–answer pairs in short and detailed forms without executable verification, which may limit their consistency and correctness. 2.2 (Multimodal) Large Language Models Large language models (LLMs) have recently been applied to table understanding via strategies such as prompt engineering ( 6 ; 43 ) , instruction

Read the complete text