Fine-Tuning Vision-Language Models for Markdown Conversion of Financial Tables in Malaysian Audited Financial Reports

Jin Khye Tan, En Jun Choong, Ethan Jeremiah Chitty, Yan Pheng Choo, John Hsin Yang Wong, Chern Eu Cheah, [3ex] Faculty of Computer Science, Information Technology, Universiti Malaya, 50603 Kuala Lumpur, Malaysia · 2025

Read Fine-Tuning Vision-Language Models for Markdown Conversion of Financial Tables in Malaysian Audited Financial Reports by Jin Khye Tan, En Jun Choong, Ethan Jeremiah Chitty, Yan Pheng Choo, John Hsin Yang Wong, Chern Eu Cheah, [3ex] Faculty of Computer Science, Information Technology, Universiti Malaya, 50603 Kuala Lumpur, Malaysia (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-05669.txt.

Introduction

Tan and five co-authors present a study from twenty twenty-five on turning money tables in reports into clean text form. The reports are audited accounts from Malaysia, with turned pages and stacked headers. Such layouts make table reading by machine hard. The team tuned an image-and-text model on a set of over two thousand page and text pairs. They scored the results with both a judge model and a new tree-based score for layout match.

Their tuned model scored high on both checks of exactness and layout. It beat its own base model, bigger general models, and famed paid models. It also ran faster than the self-hosted rivals they tested. The authors say focused tuning can rival much larger tools at lower cost.

Catalog record

CREATORJin Khye Tan, En Jun Choong, Ethan Jeremiah Chitty, Yan Pheng Choo, John Hsin Yang Wong, Chern Eu Cheah, [3ex] Faculty of Computer Science, Information Technology, Universiti Malaya, 50603 Kuala Lumpur, Malaysia
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-05669

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Fine-Tuning Vision-Language Models for Markdown Conversion of Financial Tables in Malaysian Audited Financial Reports Jin Khye Tan, En Jun Choong, Ethan Jeremiah Chitty, Yan Pheng Choo, John Hsin Yang Wong, Chern Eu Cheah, [3ex] Faculty of Computer Science, Information Technology, Universiti Malaya, 50603 Kuala Lumpur, Malaysia Abstract Fine-Tuning Vision-Language Models for Markdown Conversion of Financial Tables in Malaysian Audited Financial Reports Jin Khye Tan 1 , En Jun Choong, Ethan Jeremiah Chitty, Yan Pheng Choo, John Hsin Yang Wong, and Chern Eu Cheah. 1 Faculty of Computer Science and Information Technology, Universiti Malaya, 50603 Kuala Lumpur, Malaysia. Abstract Accurately extracting and representing the structure of tabular data from financial documents remains a critical challenge in document understanding, particularly for regulatory and analytical use cases. This study addresses the complexity of converting financial tables from Malaysian audited financial reports into Markdown format, a task complicated by rotated layouts, multi-level headers, and implicit structural cues. We propose a fine-tuned vision-language model (VLM), based on Qwen2.5-VL-7B, optimized for high-fidelity Markdown generation from document images. Our approach includes a curated dataset of 2,152 image-text pairs with augmentations and a supervised fine-tuning strategy using LoRA. To assess performance, we evaluated our model on 100 out-of-sample tables using a dual framework: a criteria-based LLM-as-a-judge for fine-grained accuracy and our novel Markdown Tree-Edit-Distance-based Similarity (TEDS) metric for holistic structural fidelity. Our model achieves a 92.20% overall accuracy on the criteria-based assessment and a 96.53% Markdown TEDS score . This performance significantly surpasses its Qwen2.5-VL-7B base model, larger-scale VLMs, and specialized reasoning-enabled models. Compared to these self-hosted alternatives, it also significantly reduces inference time. Furthermore, its accuracy exceeds that of widely used proprietary models such as OpenAI’s GPT-4o and Gemini 2.5 Flash. These results demonstrate that domain-specific fine-tuning provides an effective and efficient method to bridge the gap between unstructured financial documents and downstream automation, rivalling much larger and more general models without their computational overhead. Keywords: Vision-Language Models (VLM), Fine-Tuning, Table Structure Recognition, Markdown Conversion, Document AI, Financial Reporting 1 Introduction 1.1 Background and Motivation Accurate extraction and structural representation of tabular data from unstructured documents remain central challenges in document understanding, commonly referred to as table structure recognition ( Schreiber et al. 2017 ; Siddiqui et al. 2019 ) . Early approaches relied on rule-based systems and OCR engines such as Tesseract ( Smith 2007 ) , which often struggled with layout distortions and formatting variability. Subsequent advances introduced neural models such as TableNet ( Paliwal et al. 2019 ) and GraphTSR ( Chi et al. 2019 ) , leveraging visual and spatial cues to infer table structures more robustly. More recently, vision-language models (VLMs) have emerged as powerful tools for document intelligence. Models such as LayoutLM ( Xu et al. 2020 ) , Donut ( Kim et al. 2022 ) , and proprietary models, such as GPT-4o ( OpenAI and et al. 2024 ) and Gemini 2.5 ( Comanici et al. 2025 ) , jointly model textual, visual, and spatial information, enabling generalizable and high-fidelity table understanding. 1.2 Challenges in Financial Table Understanding In the financial domain, accurate extraction of tabular data from reports is essential for tasks such as auditing, corporate analysis, and regulatory compliance. Financial reports typically contain structured tables in key sections such as income statements, balance sheets, cash flow statements, statements of changes in equity, and notes to the financial statements. These tables are central to assessing a company’s financial health and performance ( Rejison 2025 ) . Preserving the structural integrity of these tables, including the correct alignment of headers, values, and contextual labels, is critical for enabling downstream applications such as ratio analysis, risk modeling, and automated reporting. As highlighted by Samantapudi 2025 , even small structural errors can lead to significant issues in financial interpretation or regulatory compliance. High-accuracy automation of this process is therefore a fundamental requirement for scalable and reliable financial analysis, supporting use cases such as earnings evaluation, reconciliation, peer benchmarking, and market surveillance. However, financial tables in real-world documents often vary significantly in layout, structure, and presentation. This variability is also observed in Malaysian audited financial reports, which commonly contain rotated tables, multi-level headers, implicit columns, and missing grid lines. These complexities pose significant challenges for document parsing, even for state-of-the-art VLMs. Section 2 provides illustrative examples demonstrating their impact on structured output generation. 1.3 Our Approach: VLM Fine-Tuning for Malaysian Financial Reporting Adaptation To address these challenges, we introduce a "Markdownification" pipeline that converts financial tables into Markdown representations, guided by domain-specific formatting rules tailored to Malaysian financial reporting. This process flattens hierarchical headers, preserves multi-entity and multi-period distinctions, and explicitly identifies implicit structural elements (e.g., note indicators). By standardizing diverse layouts into a consistent textual representation, "Markdownification" enhances the reliability of downstream LLM-based financial analysis. Initial experiments with open-source VLMs indicated suboptimal performance in interpreting complex financial tables, revealing significant structural errors. Proprietary models (Gemini 2.5, GPT-4o) showed improved results, yet still required extensive prompt adjustments and produced inconsistencies. Their closed-source nature and elevated costs further limit scalability and transparency, highlighting the need for customized, self-hosted solutions with strong domain expertise. We present a fine-tuned adaptation of Qwen2.5-VL-7B-Instruct ( Bai et al. 2025 ) optimized for Markdown generation from Malaysian financial tables. Our contributions are: • Domain-specific Dataset : A development set of 2,152 text-image pairs from financial statements and notes, including 30% rotated augmentations, was used for training and validation. Final performance was measured on a held-out, distinct test set of 100 tables. • Fine-Tuning Methodology : Supervised Fine-Tuning (SFT) with LoRA ( Hu et al. 2021 ) using the LLaMA-Factory framework ( Zheng et al. 2024 ) on two units of A100 40GB GPUs. • Significant Accuracy Uplift : Achieved a 92.20% overall accuracy on a criteria-based LLM evaluation and a 96.53% Markdown TEDS score for holistic structural fidelity. This performance dramatically surpasses not only the Qwen2.5-VL-7B base model (32.80% accuracy, 52.08% TEDS), but also larger models such as Qwen2.5-VL-32B (68.60% accuracy, 71.20% TEDS) and Qwen2.5-VL-72B AWQ (79.80% accuracy, 74.72% TEDS), reasoning-intensive models such as MiMo-VL-7B-RL (58.20% accuracy, 58.56% TEDS), and proprietary models such as Gemini 2.5 Flash (82.40% accuracy, 79.19% TEDS) and GPT-4o (65.20% accuracy, 74.41% TEDS). 2 Challenges in Malaysian Financial Tables Malaysian audited financial reports exhibit significant heterogeneity in table design, posing unique challenges for automated extraction. Even advanced VLMs struggle with accurate Markdown conversion due to layout irregularities. Below, we detail six key challenges, their implications, and mitigation strategies. 1. Inconsistent Table Formats : Tables vary widely in structure, including column count, header depth, use of grid lines, even within the same document. This lack of standardization prevents the use of fixed parsing rules, requiring adaptive, context-aware models. Figure 1 : Variations in table formatting across different financial reports. 2. Rotated Layouts : Wide tables are frequently rotated 90° to fit page constraints. Such rotations disrupt a VLM’s standard text-flow interpretation, causing models to misalign rows and columns. Figure 2 : Example of a rotated table and its inaccurate Markdown output. Figure 2 shows how this orientation leads the model to misread the table structure, resulting in a transposed and inaccurate Markdown output. 3. Multi-level Headers : Hierarchical headers are common but incompatible with standard Markdown,

Read the complete text