Describe Anything Model for Visual Question Answering on Text-rich Images
Yen-Linh Vu, Dinh-Thang Duong, Truong-Binh Duong, Anh-Khoi Nguyen, Thanh-Huy Nguyen, Le Thien Phuc Nguyen, Jianhua Xing, Xingjian Li, Tianyang Wang, Ulas Bagci · 2025
Read Describe Anything Model for Visual Question Answering on Text-rich Images by Yen-Linh Vu, Dinh-Thang Duong, Truong-Binh Duong, Anh-Khoi Nguyen, Thanh-Huy Nguyen, Le Thien Phuc Nguyen, Jianhua Xing, Xingjian Li, Tianyang Wang, Ulas Bagci (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-12441.txt.
Introduction
Yen-Linh Vu and a large team adapted a region-aware vision model in two thousand twenty-five to answer questions about text-rich images. The base model, DAM, can describe any picked region of a picture in detail. The team guessed that skill would help with questions whose answers hide in dense text inside documents and scenes.
Their frame, DAM-QA, gathers answers from many regional views of an image and merges them to find the right evidence. On six test sets it beat the base model each time, with a gain of more than seven points on a document test. It also led other region-aware models while using fewer parameters.
Catalog record
| CREATOR | Yen-Linh Vu, Dinh-Thang Duong, Truong-Binh Duong, Anh-Khoi Nguyen, Thanh-Huy Nguyen, Le Thien Phuc Nguyen, Jianhua Xing, Xingjian Li, Tianyang Wang, Ulas Bagci |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-12441
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Describe Anything Model for Visual Question Answering on Text-rich Images Thanks: Corresponding author: [email protected] . We thank AI VIETNAM for facilitating computational resources and financial support for this paper. Yen-Linh Vu, Dinh-Thang Duong, Truong-Binh Duong, Anh-Khoi Nguyen, Thanh-Huy Nguyen, Le Thien Phuc Nguyen, Jianhua Xing, Xingjian Li, Tianyang Wang, Ulas Bagci Abstract Recent progress has been made in region-aware vision-language modeling, particularly with the emergence of the Describe Anything Model (DAM). DAM is capable of generating detailed descriptions of any specific image areas or objects without the need for additional localized image-text alignment supervision. We hypothesize that such region-level descriptive capability is beneficial for the task of Visual Question Answering (VQA), especially in challenging scenarios involving images with dense text. In such settings, the fine-grained extraction of textual information is crucial to producing correct answers. Motivated by this, we introduce DAM-QA, a framework with a tailored evaluation protocol, developed to investigate and harness the region-aware capabilities from DAM for the text-rich VQA problem that requires reasoning over text-based information within images. DAM-QA incorporates a mechanism that aggregates answers from multiple regional views of image content, enabling more effective identification of evidence that may be tied to text-related elements. Experiments on six VQA benchmarks show that our approach consistently outperforms the baseline DAM, with a notable 7+ point gain on DocVQA. DAM-QA also achieves the best overall performance among region-aware models with fewer parameters, significantly narrowing the gap with strong generalist VLMs. These results highlight the potential of DAM-like models for text-rich and broader VQA tasks when paired with efficient usage and integration strategies. Our code is publicly available at https://github.com/Linvyl/DAM-QA.git . 1 Introduction Figure 1 : DAM-QA handles text-rich VQA by applying DAM to both global and local image views, capturing focused regions through a sliding-window process. The resulting answers are aggregated through voting to enhance overall prediction accuracy. Text-rich Visual Question Answering (VQA) is a challenging multimodal task that demands a system to answer natural language questions based on visuals with considerable textual content, such as documents, infographics, and charts [ 38 , 39 , 36 ] . Solving this problem entails not only recognizing and reading text in various styles and layouts but also comprehending how textual pieces interact with the surrounding visual context. The intricacy and diversity of visual-textual interactions in such images present significant problems for current approaches [ 44 ] . Traditionally, dense text VQA methods rely heavily on Optical Character Recognition (OCR) to extract embedded text, followed by fusion modules that integrate visual and textual features [ 4 , 50 , 51 , 42 , 28 ] . However, these pipelines are often brittle, sensitive to OCR errors and reliant on complex post-processing. Similar to many works that leverage global-local visual mechanism [ 41 , 47 , 48 ] , end-to-end Vision-Language Models (VLMs) have emerged as unified alternatives that jointly encode visual and textual inputs [ 6 , 60 , 1 ] . While effective, many VLMs primarily focus on global image-text alignment, limiting their ability to capture fine-grained information in dense or localized regions. Region-aware VLMs have gained attention by incorporating localized supervision or attention, showing promise in tasks like referring expression comprehension and grounded question answering [ 9 , 15 , 8 , 58 ] . Notably, the Describe Anything Model (DAM) [ 30 ] supports diverse region formats (points, masks, boxes) and excels at generating fine-grained region-level descriptions with minimal supervision. While DAM has proven effective for region-level description, its potential in complex reasoning tasks like dense text VQA remains underexplored. We posit that DAM’s flexible region-aware modeling makes it a strong candidate for handling text-rich visual reasoning tasks. To this end, we introduce DAM-QA, a framework that extends DAM’s capabilities to text-rich VQA. Inspired by prior work on localized visual understanding [ 19 , 32 , 5 , 33 ] , DAM-QA applies a sliding-window approach alongside full-image inference to extract regional patches, enabling multi-scale context modeling. These multiple views are fused via a voting-based strategy to enhance robustness against incomplete or ambiguous inputs. A custom evaluation protocol ensures fair assessment of DAM’s VQA ability without modifying its architecture. We assess DAM-QA against six available VQA benchmarks, which include both document-centric and general-purpose datasets. While DAM-QA does not yet equal the performance of generalist foundation models, it beats the baseline DAM and other region-aware VLMs on most benchmarks, including a significant increase of over 7 points on DocVQA [ 38 ] over the original DAM model. These findings demonstrate that modifying region-aware models, such as DAM, has the potential to enhance performance in VQA tasks, particularly those that involve fine-grained reasoning over localized textual information. Our contributions can be summarized as follows: • We provide a systematic investigation into applying region-aware VLMs to text-rich VQA tasks, using the Describe Anything Model (DAM) as a representative case. • We introduce DAM-QA, a simple yet effective framework combining region-level inference and answer aggregation without requiring additional supervision. • We report comprehensive results on six VQA benchmarks, showing improved average performance over the DAM baseline and demonstrating the value of region-aware representations. 2 Related Works 2.1 Vision-Language Models Vision-Language Models (VLMs) support tasks like captioning, retrieval, grounding, and Visual Question Answering (VQA) by learning joint image-text representations using Transformer architectures trained with masked or contrastive objectives. Masked VLMs align visual regions and text via masked prediction. Early models [ 25 , 34 ] fused object features with word tokens under BERT-style pretraining. Later models [ 27 ] introduced richer image-text objectives, setting new benchmarks in VQA. Advances such as [ 17 ] dropped region proposals via masked visual modeling, while [ 26 ] improved robustness and scale. InternVL3 [ 60 ] added native multimodal pretraining and variable visual encoding, rivaling proprietary models like Gemini 2.5 [ 46 ] . Autoregressive VLMs connect vision encoders to frozen LLMs via instruction tuning. The BLIP family [ 24 , 23 ] introduced Q-Formers for vision-to-language alignment. Models like [ 2 ] extend this to few-shot generation, while [ 10 ] use GPT-4-style dialogue to enhance instruction following. Recent open models [ 1 , 12 ] show strong VQA and OCR performance, offering transparency and comparability, unlike proprietary models such as Gemini 2.5 Flash [ 46 ] . Despite these advances, most VLMs operate on whole-image inputs and lack explicit mechanisms for region-level grounding, especially in text-heavy images. This motivates region-aware VLMs, which aim to better align spatially localized visual and textual features, as explored in the next section. Figure 2 : Overview of the proposed DAM-QA framework (a) Patch Generation ( 3.3.1 ): An input image is processed with a sliding window to generate a full-image view alongside multiple overlapping local patches. (b) Weighted Voting Scheme ( 3.3.2 ): All views are fed into the DAM model with a unified VQA prompt. The resulting answers are aggregated via a weighted voting system, where “unanswerable” predictions are assigned a weight of 0, while the full-image prediction receives a weight of 1 and patch-level predictions are weighted by their area. The final answer is determined by the highest total score. 2.2 Region-aware Vision-Language Models Most existing VLMs operate at image-level granularity, lacking fine-grained spatial grounding capabilities. Yuan