Rule-Based Reinforcement Learning for Document Image Classification with Vision Language Models

Michael Jungo, Andreas Fischer · 2025

Read Rule-Based Reinforcement Learning for Document Image Classification with Vision Language Models by Michael Jungo, Andreas Fischer (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-22283.txt.

Introduction

Michael Jungo and Andreas Fischer study whether reinforcement learning can teach vision-language models to classify document images better. Sorting scanned documents into types is a basic task in document analysis, yet reinforcement learning has rarely been applied there. Their 2025 paper uses the rule-based style of reinforcement learning made popular by reasoning models. The reward is a simple checkable rule, such as whether the predicted class is right, rather than a learned judge.

The authors find that reinforcement learning tends to generalize better to data unlike the training set. They test that claim in three scenarios: images from a different distribution, classes the model never saw in training, and documents arriving in a different modality. In each case the reinforcement-learned model held up better than the comparison. The work suggests that the reasoning gains associated with this training style transfer to document tasks.

Catalog record

CREATORMichael Jungo, Andreas Fischer
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-22283

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Rule-Based Reinforcement Learning for Document Image Classification with Vision Language Models Michael Jungo, Andreas Fischer Abstract Rule-based reinforcement learning has been gaining popularity ever since DeepSeek-R1 has demonstrated its success through simple verifiable rewards. In the domain of document analysis, reinforcement learning is not as prevalent, even though many downstream tasks may benefit from the emerging properties of reinforcement learning, particularly the enhanced reason capabilities. We study the effects of rule-based reinforcement learning with the task of Document Image Classification which is one of the most commonly studied downstream tasks in document analysis. We find that reinforcement learning tends to have better generalisation capabilities to out-of-distritbution data, which we examine in three different scenarios, namely out-of-distribution images, unseen classes and different modalities. Our code is available at https://github.com/jungomi/vision-finetune . 1 Introduction Document Image Classification remains an important task as a first step for many document analysis approaches. It can be as simple as deciding to which department it needs to be sent to, up to more complex systems that perform automated analyses to extract and process the specific contents of various types of documents. With the rise of popularity in Large Language Models (LLMs) and their steadily improving capabilities across multiple modalities, they have become of interest for a wide variety of tasks. A key reason for this, is their ability of in-context learning [ 5 ] , which makes them adaptable to any new task with only a few examples. Many different Vision Language Models (VLMs) [ 43 , 20 ] are available, with commercial models like GPT-4-Vision [ 1 ] and Gemini [ 34 ] , but also open-source models such as Llama 3.2-Vision [ 10 ] , Qwen-2.5-VL [ 3 ] and Gemma 3 [ 35 ] . Even though the training data of most models is rarely fully disclosed, they are expected to also have been pre-trained on document understanding, including publicly available datasets for document visual question answering [ 25 , 24 , 23 , 26 , 33 ] , as well as synthetic datasets [ 17 , 8 , 41 ] . These models are therefore equipped with a good understanding of documents and provide an excellent starting point for Document Image Classification. The RVL-CDIP [ 12 ] dataset is the most commonly used benchmark for Document Image Classification. It consist of 400 000 images of scanned documents across 16 classes, which is a labelled subset of the IIT-CDIP collection of tobacco litigation documents [ 18 ] . Over the years, many different models have been evaluated on this dataset, but LLMs, including VLMs, have not been explored in much detail. The contributions of this paper can be summarised as follows: • We show that reinforcement learning (RL) can be used as an alternative to supervised fine-tuning (SFT) for document image classification. • We compare the generalisation capabilities of RL and SFT in three scenarios: out-of-distribution images, unseen classes and different modalities. • We examine the effect of the reasoning ability that is induced by RL. 2 Related Work 2.1 Rule-Based Reinforcement Learning for Vision Language Models After the success of rule-based reinforcement learning achieved by DeepSeek-R1 [ 11 ] , naturally, the desire of applying it to other models emerged and researchers started applying it to other domains and modalities. R1-onevision [ 40 ] , Vision-R1 [ 42 ] and VLM-R1 [ 31 ] successfully applied the R1 style reinforcement learning to vision language models shortly after the release of DeepSeek-R1 with the general consensus that rule-based RL on its own can achieve competitive performance compared to SFT or even surpass it. Zhou et al. [ 44 ] managed to reproduce the “aha-moment” from DeepSeek-R1 in the form of visual reasoning in a 2B VLM without any prior SFT. Jigsaw-R1 [ 36 ] applies RL to jigsaw puzzles, under the premise that the reassembling of shuffled patches provide visual understanding that is transferable to downstream tasks, and found that the model is able to generalise to other visual tasks that require spatial reasoning. Rule-based RL has found its way to many other visual task, for example in the medical domain, Med-R1 [ 14 ] and MedVLM-R1 [ 28 ] concurrently investigated rule-based reinforcement learning, where they observed that it improves the generalisation and reliability of the VLM across eight distinct medical imaging modalities, such as MRI, CT or X-ray. To the best of our knowledge, downstream tasks involving document images have not yet been studied, which compels us to explore it in the context of document image classification. 2.2 Document Classification with Large Language Models Scius-Bertrand et al. [ 29 ] utilised LLMs to classify a subset of the RVl-CDIP with zero-shot and one-shot prompting as well as fine-tuning them. At the time, VLMs were a novelty and only GPT-4-Vision was available, hence they focused primarily on classifying the documents based on their textual content that was extracted with an OCR engine. The zero-shot and one-shot already achieved good performances, with the highest being GPT-4 at 61.8% accuracy when only given the OCR as input and 69.9% when the images are included in the input. When they fine-tuned Mistral on the 1 600 training samples from that subset, its accuracy increased from 45.4% up to 83.4%. They showed that fine-tuning with a relatively small subset, compared to the full dataset, improves the results considerably. Given the fact that the image based classification has achieved better results compared to the OCR, it would be expected that fine-tuning a VLM has even more potential. We use this opportunity to study the fine-tuning of VLMs on the subset they created, which ensures that we have a dataset that is sufficiently large to see the impact of the training while not being too large to the point of not being able to finish the experiments in a reasonable time. 3 Fine-Tuning Methods There are multiple stages in the pre-training of LLMs. After being trained on large unsupervised data, where the goal is a simple next token prediction in order to learn fundamental text understanding and intricacies of the language, there are two primary stages to shape the LLMs into the adaptable form that is commonly used for all sorts of tasks. Firstly, supervised fine-tuning (SFT) is used to learn specific answers to a given question, this also includes instruction following, which teaches the model to respond to a query rather than just completing the input by adding additional text that is most likely to follow, as it was done during the initial pre-training. Afterward, reinforcement learning (RL) is employed to steer the model’s responses into a more preferential form, which may be stylistic choices, but also for safety [ 4 , 21 , 7 ] by avoiding inappropriate phrases or by refusing to answer unconscionable queries. While those are well established practices, most downstream tasks, e.g. document classification, only apply additional SFT to adapt the model to their specific tasks. One of the most prevalent reasons is the fact that RL requires a reward and value model to be trained alongside the LLM itself. This not only makes it much more demanding in terms of resources, but also adds a training complexity that made it unsuitable for

Read the complete text