RoiMAM: Region-of-Interest Medical Attention Model for Efficient Vision-Language Understanding
Jiayan Yang, Zhuoyu Wu, Wenqi Fang · 2026
Read RoiMAM: Region-of-Interest Medical Attention Model for Efficient Vision-Language Understanding by Jiayan Yang, Zhuoyu Wu, Wenqi Fang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-15561.txt.
Introduction
Jiayan Yang, Zhuoyu Wu, and Wenqi Fang present RoiMAM, a compact vision-language model for medical visual question answering. A training-free module finds lesion-relevant regions and suppresses other semantics, while a prompt enhancer adds modality-specific context without new trained parameters. Compared with a widely used medical model, it reaches accurate diagnosis at under a fifth of the size, improving accuracy by about two percent on one benchmark and 4.6 percent on another.
Catalog record
| CREATOR | Jiayan Yang, Zhuoyu Wu, Wenqi Fang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 18349 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.15561
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
RoiMAM: Region-of-Interest Medical Attention Model for Efficient Vision-Language Understanding Jiayan Yang, Zhuoyu Wu, Wenqi Fang Abstract Vision–Language Models (VLMs) facilitate medical visual question answering (MedVQA) by jointly interpreting images and text. However, existing models typically depend on large architectures and closed-set answers, which limits their efficiency and potential clinical applicability. To overcome these shortcomings, we introduce RoiMAM, an efficient VLM. It integrates a training-free ROI Generation Module with Semantic Selective Suppression to focus on lesion-relevant regions, alongside a Text Prompt Enhancer module that provides modality-specific context without introducing training parameters. Compared to the widely used MedVInT-TD model, our design achieves efficient and accurate diagnosis at less than 20% of the model size, while improving accuracy by approximately 2% on SLAKE and 4.6% on PMC-VQA. † † address: 1 Shenzhen Institutes of Advanced Technology, Chinese Academy of Science 2 CyPhi( \Psi\Phi ) AI Research Lab, School of IT, Monash University, Malaysia Campus 1 INTRODUCTION Computer-aided medical diagnosis has progressed rapidly with the advent of deep-learning technologies. Recent Vision–Language Models (VLMs), usually built on Large Language Models (LLMs), have advanced medical visual question answering (MedVQA) systems by jointly interpreting clinical images and textual queries. These systems show promise for addressing healthcare challenges such as resource shortages, physician workload, and diagnostic errors. Despite notable progress, MedVQA systems still face significant barriers to practical adoption. Most existing VLMs remain confined to closed-set answering—selecting from predefined options rather than generating free-form responses—creating a mismatch with the open-ended nature of clinical consultations [ 7 ] . Although LLMs exhibit strong conversational ability, their heavy computational overhead and large parameter counts make training and inference inefficient, limiting scalability and delaying real-world implementation [ 3 ] . These scalability concerns have driven research into techniques that enhance model capability without incurring heavy computational cost. A promising avenue is the use of visual prompts to guide networks toward lesion-relevant regions in medical images. Yet, these region-of-interest (ROI)–based prompting approaches still rely on supervised detection or segmentation models, which demand expensive, large annotated datasets [ 14 ] [ 16 ] . Consequently, their pipelines remain resource-intensive, and hallucinations in lesion localization further erode the reliability required for clinical deployment. To overcome these limitations, we present RoiMAM , an ultra-lightweight and efficient VLM with only 1.7 billion (B) parameters that delivers state-of-the-art results on multiple MedVQA benchmarks while maintaining both accuracy and computational efficiency. RoiMAM combines a training-free ROI Generation Module (RGMo), which uses the Semantic Selective Suppression ( S^{3} ) mechanism to emphasize text-relevant image regions, with a Text Prompt Enhancer (TPE) module that adds modality-specific context without extra parameters. The main contributions of this study are outlined below: • To alleviate the computational burden of traditional VLMs in the MedVQA tasks, we propose an efficient framework called RoiMAM, which integrates RGMo and TPE modules to enhance performance while reducing parameters. • A training-free RGMo module is proposed, employing the S^{3} mechanism to minimize irrelevant visual attention and enable accurate lesion localization with minimal computation and no pre-training. • We introduce the TPE module, leveraging existing vision encoders to enrich text context and improve cross-modal robustness without adding parameters. • Our model outperforms large models such as MedVInT-TD on three representative open-source datasets, obtaining higher accuracy while reducing the number of parameters by roughly 80%, thus mitigating data privacy and computational constraints [ 15 ] . 2 RELATED WORKS MedVQA systems aim to answer questions from medical images, mirroring real-world diagnostic reasoning. Early works, mainly based on the CLIP-based model, framed MedVQA as a classification task with predefined answer sets. For instance, PubMedCLIP demonstrated CLIP’s applicability to the medical domain, and later datasets like PMC-VQA and PMC-OA facilitated larger-scale pretraining of VLMs [ 15 ] [ 4 ] [ 8 ] . While effective for closed-set tasks, these approaches do not fully reflect clinical practice, where diagnoses are open-ended and answers are not limited to fixed options. With the rise of generative LLMs such as LLaMA and their visual extensions, VLMs are being increasingly adapted for medical applications. HuatuoGPU-Vision fine-tuned VLMs on task-specific datasets, while LLaVA-Med introduced a dedicated medical dataset and a biomedical multimodal instruction-following pipeline [ 7 ] [ 1 ] . In addition, visual prompts have also been explored: FAVP proposed an Adaptive Visual Prompt Creator (AVPC) [ 14 ] [ 2 ] , and MedVP utilized a fine-tuned Grounding DINO to locate ROIs from question keywords [ 16 ] . However, these methods remain unsuitable for on-device deployment because of their large model sizes, and pretraining visual prompt generators is still computationally costly due to the need for annotated images. In contrast, our RoiMAM model provides a training-free visual prompt generation method to pinpoint relevant regions. This keeps the parameter size of RoiMAM below two billion, making it feasible for future edge deployment. Figure 1: Left: Overview of the RoiMAM framework. Right: The workflow of the RGMo module. 3 PROPOSED METHOD In this section, we detail our RoiMAM framework, designed to process and synthesize information from three distinct streams: an input image ( \mathcal{I} ), a question ( \mathcal{T} ), and a prompt specifying the image modality ( \mathcal{L} ). As shown in the left panel of Figure 1 , the input image is processed through parallel pathways: a Vision Encoder generates image-level embedding ( E_{I} ), while the RGMo module jointly processes \mathcal{I} and \mathcal{T} to produce the ROI related embedding ( E_{ROI} ). Simultaneously, the modality prompt, along with E_{I} , is passed through a TPE module to yield an enhanced text embedding ( \mathcal{M} ). The embeddings E_{I} and E_{ROI} , both processed by a Linear Align layer, together with \mathcal{M} and \mathcal{T} , are subsequently passed to a Small LLM. This LLM serves as the core reasoning engine, integrating multi-modal inputs to produce a precise textual answer, such as identifying the primary organ in the image. 3.1 Text Prompt Enhancer (TPE) To address the inherent limitations of smaller VLMs, whose reduced parameter size can hinder deep reasoning and lead to attention on irrelevant regions, we introduce the TPE module. This module enriches the input prompt with salient, high-level semantic information from the image, compensating for the model’s limited capacity. The TPE module is built on a pre-trained CLIP model, sharing the vision encoder with our medical VLM and incorporating a single BERT model as the text encoder, thus keeping the parameter overhead low. Given a medical image \mathcal{I} , TPE leverages CLIP’s zero-shot classification ability to perform an initial analysis, such as identifying the image modality (e.g., CT, MRI, or X-ray). This classification is then used as a prompt for the VLM’s LLM, providing foundational knowledge that guides reasoning and enables more accurate, context-aware responses, effectively compensating for the limitations of a smaller model. Specifically, we construct a text list \mathcal{L} , representing possible image modalities, as input to the BERT model in the TPE module to generate the embedding E_{\mathcal{L}} . The most relevant text embedding (i.e., the most probable modality) \mathcal{M} is then formally defined as: \mathcal{M}=\text{argmax}_{i}\frac{E_{I}\cdot E_{\mathcal{L}_{i}}}{\left\|E_{I}\right\|\left\|E_{\mathcal{L}_{i}}\right\|} (1) where E_{\mathcal{L}_{i}} denotes the embedding of the i -th element in \mathcal{L} . 3.2 ROI Generation Module (RGMo) In general domains, visual prompts significantly improve a model’s ability to analyze images