How Good LLMs Are at Answering Bangla Medical Visual Questions? Dataset and Benchmarking
Rafid Ahmed, Intesar Tahmid, Mir Sazzat Hossain, Tasnimul Hossain Tomal, Md Fahim, Md Farhad Alam Bhuiyan · 2026
Read How Good LLMs Are at Answering Bangla Medical Visual Questions? Dataset and Benchmarking by Rafid Ahmed, Intesar Tahmid, Mir Sazzat Hossain, Tasnimul Hossain Tomal, Md Fahim, Md Farhad Alam Bhuiyan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-18111.txt.
Introduction
Rafid Ahmed and five colleagues introduce BanglaMedVQA, a dataset of clinically validated image-question-answer pairs for Bangla medical visual question answering. They benchmark current foundation models on it. Bangla scores run far below English MedVQA scores, showing the strain of a low-resource language. Even top models like Gemini and GPT-4.1 mini fail specialized diagnostic questions, exposing weak fine-grained medical reasoning. Open-source Gemma-3 sometimes tops them in general categories but also struggles with clinically complex questions, underscoring the need for top-notch evaluation.
Catalog record
| CREATOR | Rafid Ahmed, Intesar Tahmid, Mir Sazzat Hossain, Tasnimul Hossain Tomal, Md Fahim, Md Farhad Alam Bhuiyan |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44049 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.18111
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
How Good LLMs Are at Answering Bangla Medical Visual Questions? Dataset and Benchmarking Rafid Ahmed, Intesar Tahmid, Mir Sazzat Hossain, Tasnimul Hossain Tomal, Md Fahim, Md Farhad Alam Bhuiyan Abstract Recent advancements in Large Language Models (LLMs) and Large Vision Language Models (LVLMs) have enabled general-purpose systems to demonstrate promising capabilities in complex reasoning tasks, including those in the medical domain. Medical Visual Question Answering (MedVQA) has particularly benefited from these developments. However, despite Bangla being one of the most widely spoken languages globally, there exists no established MedVQA benchmark for it. To address this gap, we introduce BanglaMedVQA, a dataset comprising clinically validated image–question–answer pairs, along with a comprehensive evaluation of current foundation models on this resource. Consistent with prior findings that report low performance of current models on English MedVQA benchmarks, our analysis reveals that Bangla performance is substantially lower, reflecting the challenges inherent to low-resource languages. Even top-performing models such as Gemini and GPT-4.1 mini fail to accurately answer specialized diagnostic questions, indicating severe limitations in fine-grained medical reasoning. Although certain open-source models, such as Gemma-3, occasionally outperform these models in general categories, they too struggle with clinically complex questions, underscoring the urgent need for top-notch evaluation method. Introduction In recent years, foundation models such as Large Language Models (LLMs) ( 1 ; 35 ; 2 ) and LVLMs ( 6 ; 16 ; 19 ; 5 ) have attracted significant attention for their ability to process, generate, and reason over complex textual and visual inputs. These models produce human-like language and achieve high performance across a wide range of benchmarks. Their integration into the medical domain has already shown promising results, demonstrating potential for real-world clinical applications. Figure 1: Workflow of the dataset curation process. Images and metadata were obtained from two widely used biomedical datasets, enabling the automatic generation of QA pairs, which were subsequently verified by domain experts. Applications such as automated radiology report generation ( 32 ) , clinical decision support ( 25 ) , and interactive medical question answering ( 14 ) highlight the capability of LLMs to assist both patients and healthcare professionals. Existing medical VQA benchmarks ( 18 ; 13 ; 42 ) , when combined with current foundation models, have reported encouraging results. However, despite being the seventh most spoken language in the world, with approximately 278 million speakers, Bangla has received very limited attention in the context of medical visual question answering. It is therefore crucial to evaluate the performance of LLMs in this domain for Bangla. The effectiveness of medical AI systems is closely linked to the quality of the datasets on which they are trained ( 9 ) . In high-resource languages such as English, medical VQA datasets have gradually evolved in complexity, progressing from simple factual questions to tasks requiring deeper clinical reasoning ( 22 ; 28 ) . In contrast, low-resource languages face a significant bottleneck: the absence of sufficiently comprehensive datasets capable of supporting even fundamental diagnostic question answering. In the Bangla medical VQA domain, progress has been highly limited. To date, the only publicly available dataset is ( 27 ) . However, this resource does not provide any information regarding the source of the images, nor does it describe the process of dataset creation or validation by medical experts. Furthermore, it suffers from annotation errors and lacks clinical oversight, making it unsuitable as a reliable benchmark. To address this gap, we propose a new Bangla medical Visual Question Answering (VQA) dataset built from images drawn from multiple medical domains ( 33 ; 36 ) , paired with diverse and clinically relevant questions. The dataset contains 1,374 unique image–caption pairs, with questions spanning five categories: modality, organ, abnormality, condition, and position. These categories can be grouped into two types: (i) generalized questions, which are relatively straightforward, and (ii) specialized questions, which require deeper medical understanding. Importantly, unlike the existing resource, our dataset was validated by two certified physicians to ensure both clinical accuracy and reliability. We systematically evaluate both open-source and closed-source LLMs on our proposed dataset. The best-performing models, Gemini and GPT, achieve overall accuracies of 40.38% and 26.50%, respectively. While results on generalized questions appear promising, performance on specialized diagnostic categories such as Condition/Finding and Position falls below random chance, revealing critical gaps in the reliability of current LVLMs for fine-grained medical reasoning. Previous studies have reported worse-than-random behavior on English medical VQA tasks ( 39 ) ; however, the problem is even more pronounced for Bangla, a low-resource language. Comparative experiments further confirm that closed-source models consistently perform better in English than in Bangla. Incorporating chain-of-thought reasoning and supplementing inputs with visual descriptions generated by GPT-4.1 mini substantially improves performance, suggesting that limited visual understanding remains a key bottleneck. These results highlight the importance of augmenting LVLMs with richer visual and linguistic context to enable more reliable medical VQA in low-resource languages. These findings highlight the urgent need for more robust Bangla medical datasets and stronger evaluation frameworks, despite the impressive progress of foundation models in general-domain benchmarks. In summary, our contributions are: • We introduce a new Bangla Medical VQA dataset containing diverse question–answer pairs across diagnostic categories for medical images. • We provide a comprehensive benchmark that systematically evaluates open and closed-source VLMs under zero-shot and Chain-of-Thought conditions. Related Work Visual Question Answering Visual Question Answering (VQA) has emerged as a challenging task at the intersection of vision and language. It was formally established by 3 , who introduced a benchmark dataset requiring fine-grained visual understanding and commonsense reasoning. 31 further advanced the field by introducing a region-focused attention mechanism, aligning image regions with question embeddings to improve performance on queries requiring localized reasoning. However, such early models exploited dataset biases, which led 10 to propose VQA v2.0, a balanced dataset with complementary image-question pairs that force models to genuinely leverage visual content. Beyond visual grounding, 30 addressed knowledge-intensive scenarios with KVQA, a dataset requiring external knowledge, particularly about named entities, to answer complex questions. Together, these works have progressively shaped VQA from simple recognition toward deeper reasoning. (a) Distribution of Clinical Conditions in the Dataset. (b) Distribution of Question Keywords in the Dataset. Figure 2: Distributions of clinical conditions and question keywords in the curated dataset. Medical Visual Question Answering 11 proposes a Latent Prompt Assist architecture for medical VQA that generates answer-constrained latent prompts and fuses them with unimodal and multimodal features. It also integrates disease–organ priors to enhance clinical relevance, and ultimately, it reported state-of-the-art gains on VQA-RAD ( 23 ) , SLAKE ( 18 ) , and VQA-2019 40 delivers a multilingual dermatology VQA dataset and benchmarks for consumer-generated images and free-text clinician responses, emphasizing real-world noise, longer responses, and multilingual evaluation metrics. 37 proposed a multi-level visual language model for MVQA, introducing a new multi-level instruction dataset (MLe-VQA), a feature alignment module, and an evaluation benchmark (MLe-Bench) to better capture recognition, details, diagnosis, knowledge, and reasoning. 8 developed rationale-guided benchmarks (R-RAD, R-SLAKE, R-Path) and a framework that integrates medical decision-making rationales into MedVQA, thereby improving both interpretability and performance. 41 introduced adaptive region-level visual prompts and a hierarchical answer generator with PEFT techniques to improve fine-grained localization and generative MedVQA. Bangla Visual Questing