TRivia: Self-supervised Fine-tuning of Vision-Language Models for Table Recognition
Junyuan Zhang, Bin Wang, Qintong Zhang, Fan Wu, Zichen Wen, Jialin Lu, Junjie Shan, Ziqi Zhao, Shuya Yang, Ziling Wang, Ziyang Miao, Huaping Zhong, Yuhang Zang, Xiaoyi Dong, Ka-Ho Chow, Conghui He · 2025
Read TRivia: Self-supervised Fine-tuning of Vision-Language Models for Table Recognition by Junyuan Zhang, Bin Wang, Qintong Zhang, Fan Wu, Zichen Wen, Jialin Lu, Junjie Shan, Ziqi Zhao, Shuya Yang, Ziling Wang, Ziyang Miao, Huaping Zhong, Yuhang Zang, Xiaoyi Dong, Ka-Ho Chow, Conghui He (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-01248.txt.
Introduction
Junyuan Zhang and a large team present TRivia in a 2025 paper on table recognition. Table recognition turns a picture of a table into a structured form like HTML or Markdown. It is a core part of document parsing, and it has long relied on supervised learning with labeled examples.
Recent work fine-tunes vision-language models on those labels at scale. TRivia instead fine-tunes such models in a self-supervised way for table recognition. Learning without hand-labeled tables could make document tools cheaper to build for new languages and layouts. Accurate table extraction underpins search, finance, and research tools that read documents.
Catalog record
| CREATOR | Junyuan Zhang, Bin Wang, Qintong Zhang, Fan Wu, Zichen Wen, Jialin Lu, Junjie Shan, Ziqi Zhao, Shuya Yang, Ziling Wang, Ziyang Miao, Huaping Zhong, Yuhang Zang, Xiaoyi Dong, Ka-Ho Chow, Conghui He |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-01248
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TRivia: Self-supervised Fine-tuning of Vision-Language Models for Table Recognition Junyuan Zhang, Bin Wang, Qintong Zhang, Fan Wu, Zichen Wen, Jialin Lu, Junjie Shan, Ziqi Zhao, Shuya Yang, Ziling Wang, Ziyang Miao, Huaping Zhong, Yuhang Zang, Xiaoyi Dong, Ka-Ho Chow, Conghui He Abstract Table recognition (TR) aims to transform table images into semi-structured representations such as HTML or Markdown. As a core component of document parsing, TR has long relied on supervised learning, with recent efforts dominated by fine-tuning vision-language models (VLMs) using labeled data. While VLMs have brought TR to the next level, pushing performance further demands large-scale labeled data that is costly to obtain. Consequently, although proprietary models have continuously pushed the performance boundary, open-source models, often trained with limited resources and, in practice, the only viable option for many due to privacy regulations, still lag far behind. To bridge this gap, we introduce TRivia, a self-supervised fine-tuning method that enables pretrained VLMs to learn TR directly from unlabeled table images in the wild. Built upon Group Relative Policy Optimization, TRivia automatically identifies unlabeled samples that most effectively facilitate learning and eliminates the need for human annotations through a question-answering-based reward mechanism. An attention-guided module generates diverse questions for each table image, and the ability to interpret the recognition results and answer them correctly provides feedback to optimize the TR model. This closed-loop process allows the TR model to autonomously learn to recognize, structure, and reason over tables without labeled data. Leveraging this pipeline, we present TRivia-3B, an open-sourced, compact, and state-of-the-art TR model that surpasses existing systems (e.g., Gemini 2.5 Pro, MinerU2.5) on three popular benchmarks. Model and code are released at: https://github.com/opendatalab/TRivia † † footnotetext: ∗ These authors contributed equally to this work. † † footnotetext: ‡ Corresponding Authors: Ka-Ho Chow, Conghui He. Figure 1 : TRivia-3B learns from unlabeled table images and achieves TR quality beyond the limit attainable by fine-tuning with labeled data. Unlike proprietary systems such as Gemini 2.5 Pro, it is open-sourced, compact, and can be deployed offline for privacy-sensitive document processing. 1 Introduction Document parsing plays a pivotal role in digitalization by converting scanned or photographed documents into machine-readable formats 42 ; 33 for downstream tasks such as retrieval-augmented generation 41 ; 11 . Among document elements, tables remain a longstanding challenge. They are both information-dense and structurally complex, requiring not only accurate text extraction, as in standard optical character recognition (OCR), but also precise reconstruction of their spatial and logical organization into semi-structured representations such as HTML or Markdown. Recent advances in vision-language models (VLMs) 2 ; 4 ; 34 ; 12 have revolutionized table recognition (TR) 14 ; 18 ; 27 ; 3 ; 23 . General-purpose VLMs can be adapted to TR either through prompt engineering or fine-tuning with image-annotation pairs. Proprietary systems such as Gemini 2.5 Pro, built with massive human and computational resources, already exhibit unprecedented TR capabilities as indicated in Figure 1 . Yet, they are accessible only through commercial APIs, raising privacy and compliance concerns in many practical scenarios involving sensitive documents. In contrast, open-source VLMs offer greater flexibility and transparency but are constrained by the limited scale of existing TR datasets 10 ; 20 , which are insufficient to reach state-of-the-art performance comparable to proprietary models. For example, UniTable, trained solely on open-source data 10 ; 20 , performs poorly across real-world benchmarks. Even large-scale open-source efforts such as MinerU2.5, which combines millions of samples, costly human annotations, and distillation from proprietary models, still inherit the ceiling imposed by its teacher, Gemini 2.5 Pro. Existing approaches to data acquisition for TR follow three paradigms. (1) Synthetic data, generated by rendering HTML tables, provides perfect annotations and scalability 14 ; 7 ; 18 but lacks the visual diversity and domain alignment of real-world data 18 . (2) Real-world data better capture the complexity of practical documents, but are expensive and time-consuming to annotate 3 . (3) Distilling pseudo-labels from proprietary models offers a potential compromise 27 , yet this approach remains costly, inherently caps performance at that of the teacher model, and may violate service agreements. Consequently, the curation of large-scale labeled datasets has become a major bottleneck, even though it is a critical factor for success. This raises an important question: Can unlabeled table images from the wild be harnessed to improve TR performance? In this paper, we present TRivia, a self-supervised fine-tuning framework that enables VLMs to learn table recognition directly from unlabeled table images, which can be collected at scale. As not all samples contribute equally to learning, identifying those that are most informative and deriving verifiable supervisory signals without human intervention are key challenges. TRivia leverages Group Relative Policy Optimization (GRPO) 31 with a decoupled reward function 38 ; 28 to strengthen TR capabilities via unlabeled data. Given a base VLM and a pool of unlabeled table images, it features a response-consistency sampling strategy that builds on GRPO’s principle of optimizing relative reward differences across multiple responses to select the most informative samples for learning. TRivia then generates supervisory signals through a proxy task: table question-answering (QA). An attention-guided QA generation module produces diverse, verifiable questions about each table image, and the VLM is fine-tuned to recognize, structure, and reason over tables by producing recognition results that allow a language model to correctly answer these questions. The main contributions of this paper are as follows. • We investigate self-supervised fine-tuning of VLMs on unlabeled table images and propose TRivia, a framework that harvests such data to push the frontier of TR. • We introduce a response-consistency sampling strategy to automatically identify unlabeled samples that most effectively enhance VLM fine-tuning via GRPO. • We design an attention-guided QA mechanism to ensure diverse, verifiable, and stable supervisory signals for optimization. Based on the proposed framework, we present TRivia-3B, an open-source, state-of-the-art TR model fine-tuned from Qwen2.5-VL-3B using unlabeled table images. As shown in Figure 1 , TRivia-3B surpasses both expert TR models (e.g., MinerU2.5 and PaddleOCR-VL) and general-purpose models (e.g., Gemini 2.5 Pro and GPT-5). We envision that TRivia will open a new avenue for advancing TR beyond the limitations of labeled data via self-supervised fine-tuning. Figure 2 : TRivia features (a) an adaptive dataset curation module to set the stage for (b) reinforcement learning to learn TR from unlabeled data. During dataset curation, TRivia uses a response-consistency sampling strategy (Section 3.2.1 ) to identify informative samples and generate verifiable, diverse QA for each image through an attention-guided module (Section 3.2.2 ). Based on curated data, TRivia fine-tunes the VLM to recognize, structure, and reason over tables through QA-based rewards (Section 3.1 ). 2 Related Work Early research on TR typically decomposes the task into two subtasks: table structure recognition and OCR. For structure recognition, bottom-up approaches 17 ; 29 ; 37 treat text or cell bounding boxes as graph nodes and predict their row, column, or cell associations. Split-and-merge methods 32 ; 44 ; 43 ; 16 first partition a table into a uniform grid and then merge adjacent cells to reconstruct its structure. While effective on clean layouts, these modular pipelines suffer from cumulative error propagation and rely heavily on explicit visual cues, making them brittle under real-world conditions such as