Text-VQA Aug: Pipelined Harnessing of Large Multimodal Models for Automated Synthesis
Soham Joshi, Shwet Kamal Mishra, Viswanath Gopalakrishnan · 2025
Read Text-VQA Aug: Pipelined Harnessing of Large Multimodal Models for Automated Synthesis by Soham Joshi, Shwet Kamal Mishra, Viswanath Gopalakrishnan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-02046.txt.
Introduction
Soham Joshi, Shwet Kamal Mishra, and Viswanath Gopalakrishnan tackle a data bottleneck for systems that answer questions about text visible in images, such as signs and labels. Building such datasets normally needs skilled human annotators, which is slow and costly. Their alternative is an automated pipeline that manufactures question and answer pairs from scene text in a given image.
The pipeline chains optical character recognition, region detection, caption generation, and question generation, then validates what it produces. The authors report a dataset of about seventy-two thousand question and answer pairs drawn from about forty-four thousand images. They present the method as the first pipeline able to synthesize and check a text-based visual question answering dataset at this scale.
Catalog record
| CREATOR | Soham Joshi, Shwet Kamal Mishra, Viswanath Gopalakrishnan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-02046
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Text-VQA Aug: Pipelined Harnessing of Large Multimodal Models for Automated Synthesis Soham Joshi, Shwet Kamal Mishra, Viswanath Gopalakrishnan Abstract Creation of large-scale databases for Visual Question Answering tasks pertaining to the text data in a scene (text-VQA) involves skilful human annotation, which is tedious and challenging. With the advent of foundation models that handle vision and language modalities, and with the maturity of OCR systems, it is the need of the hour to establish an end-to-end pipeline that can synthesize Question-Answer (QA) pairs based on scene-text from a given image. We propose a pipeline for automated synthesis for text-VQA dataset that can produce faithful QA pairs, and which scales up with the availability of scene text data. Our proposed method harnesses the capabilities of multiple models and algorithms involving OCR detection and recognition (text spotting), region of interest (ROI) detection, caption generation, and question generation. These components are streamlined into a cohesive pipeline to automate the synthesis and validation of QA pairs. To the best of our knowledge, this is the first pipeline proposed to automatically synthesize and validate a large-scale text-VQA dataset comprising around 72K QA pairs based on around 44K images. K eywords Text-VQA, Large Multimodal Models, Optical Character Recognition, Data synthesis 1 Introduction Figure 1: An illustrative example of the input image and the intermediate outputs in our pipeline. The green boxes in the image represent the object crops on which the captions are generated. The corresponding answers are displayed in parentheses. Large-scale pre-training has greatly impacted vision [ 1 , 2 ] and vision-language foundation models [ 3 , 4 , 5 , 6 , 7 , 8 ] , enriching their multimodal and multi-task capabilities. The pre-training datasets for these models contain millions of images and billions of tokens. The efficiency and precision of these models are influenced by large-scale pre-training on pristine datasets. These datasets can be human-annotated, web-crawled [ 9 ] , or synthesized from a larger-language model (like GPT [ 10 ] ) [ 11 ] . To the best of our knowledge, task-specific pre-training datasets aren’t explored well for specific downstream tasks—object detection, image classification, visual question answering, text-visual-question answering, and text-captioning. While large-scale generalized pre-training datasets contain billions of scene-text queries, they can’t be directly utilized to pre-train specialist models for a particular downstream task. These downstream tasks require large amounts of task-specific data and precise annotations. Human annotation for these tasks is tremendously tedious and costly. Thus, this approach faces the challenge of scaling up. Another approach that can be leveraged is web-crawling of data. However, this scenario also faces challenges in effectively aggregating task-specific data because most web data is unstructured. Thus, there is a need to explore efficient and effective methods for curating text-specific pre-training data, which the models can leverage for their pre-training and make the fine-tuning easier and more impactful. We aim to solve this problem in the text-VQA paradigm and, the above limitations motivated us to build an automated pipeline that can cater to the pre-training of the text-VQA models. Our proposed pipeline for text-VQA QA pairs synthesis of high-quality image QA pairs for text-VQA tasks involves multiple steps (See Fig. 1 for example). Initially, we begin by curating the images with scene text from public datasets like OpenImages V5 [ 12 ] following Krylov et al. [ 13 ] . We collected nearly 200K labelled images (which have text out of 9M images) from the dataset, and these will be the images that we will be working on in our pipeline. The entire pipeline utilizes pre-trained large multimodal models and specific algorithmic approaches. Since we are synthesizing text-VQA pairs, it is critical to build an understanding of the text in the image and the context of the image throughout the pipeline. The primary tasks are text-spotting and local context identification using object crop generation. Text-spotting models integrate text recognition and text detection in a single end-to-end module, forming the first block module in our pipeline. We typically used the GLASS [ 14 ] module for text-spotting on our text-rich image dataset. To understand the image thoroughly, we must associate every text occurrence with the foreground and the background objects in the scene. This requires local context identification by the generation of object crops and then associating the text with each of them. We achieve this with the help of a large grounding model—Kosmos-2 [ 15 ] owing to its zero-shot object grounding capabilities. Thus, after the first step, we will have the spotted text along with some of the object crops generated from the grounding model. The next task is to combine these two and create pairs of object crops and associated text. We align the spotted text and the object crops with their bounding box information. Further, we create captions for these object crops with a large multimodal model. There will be a caption generated for every object crop, along with the knowledge from the spotted text associated with that crop. Owing to an impactful multimodal performance based on the architecture of LLaVA [ 4 ] , we shortlisted LLaVA-R [ 16 ] as our go-to model for caption generation. We get captions for the object crops passed on in the previous step. However, these captions might possess only local context as we have passed the cropped input image to the caption generator. Thus, there is a need to aggregate information from all the captions of the object crops and we use a simple concatenation approach to tackle this. We need to note that the expected output of the process is the synthesis of text-VQA QA pairs, and we will follow separate steps for these: 1. OCR-based Answer selection and 2. Question Generation using a Large Language Model. In the former, we have created an algorithm (details in 3.6 ) which essentially identifies if there’s a token or a group of OCR tokens (from spotted text) that can potentially be an answer. We use the potential answers from this to feed as a part of the input to the next module—Question Generation using a Large Language Model. This Large Language Model accepts the prompt (described in 3.7 ) along with the OCR tokens and outputs the question with an appropriate context of the image and the answer. Further, these QA pairs might not be fully precise and there’s a chance that the model might hallucinate while generating the answer as well as the question. We fix this by inputting the question and the answer to the LLM to validate whether the generated answer is appropriate for the question, and expect a Boolean True / False as the response from the language model. As a result of the entire procedure, we successfully synthesized around 72K text-VQA image QA pairs, which can be leveraged by any text-VQA specialist model for its pre-training. All in all, we can summarize our contributions as follows: 1. We propose a novel training-free and unsupervised pipeline comprising large multimodal models for synthesizing text-VQA question-answer pairs from images with scene-text. 2. We introduce a text-VQA dataset synthesized from the pipeline, which consists of around 44K images and nearly 72K question-answer pairs. Figure 2: System Overview. The blue boxes represent the