SilVar-Med: A Speech-Driven Visual Language Model for Explainable Abnormality Detection in Medical Imaging
Tan-Hanh Pham Chris Ngo Trong-Duong Bui, Minh Luu Quang Tan-Huong Pham Truong-Son Hy text, Florida Institute of Technology, USA, Knovel Engineering Lab, Singapore, Vietnam Military Medical University, 108 Military Central Hospital, Vietnam, Can Tho University of Medicine and Pharmacy, Vietnam, Unive · 2025
Read SilVar-Med: A Speech-Driven Visual Language Model for Explainable Abnormality Detection in Medical Imaging by Tan-Hanh Pham Chris Ngo Trong-Duong Bui, Minh Luu Quang Tan-Huong Pham Truong-Son Hy text, Florida Institute of Technology, USA, Knovel Engineering Lab, Singapore, Vietnam Military Medical University, 108 Military Central Hospital, Vietnam, Can Tho University of Medicine and Pharmacy, Vietnam, Unive (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-10642.txt.
Introduction
The authors introduce SilVar-Med, a medical image assistant that doctors can talk to by voice instead of typing. Text prompts are awkward in settings like surgery, and most image models give answers without reasons, which weakens trust. SilVar-Med joins speech input with a vision-language model, so a clinician can ask about a scan aloud. The team also builds a reasoning dataset that pairs images with the thinking behind each call on what is abnormal.
Tests present the system as a proof of concept for reasoned, speech-driven image reading. The assistant explains the basis for each finding rather than naming a label alone. Code and data are released in public. The authors argue that voice access plus clear reasoning could make AI support more open, more interactive, and safer to use in real clinics.
Catalog record
| CREATOR | Tan-Hanh Pham Chris Ngo Trong-Duong Bui, Minh Luu Quang Tan-Huong Pham Truong-Son Hy text, Florida Institute of Technology, USA, Knovel Engineering Lab, Singapore, Vietnam Military Medical University, 108 Military Central Hospital, Vietnam, Can Tho University of Medicine and Pharmacy, Vietnam, Unive |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-10642
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SilVar-Med: A Speech-Driven Visual Language Model for Explainable Abnormality Detection in Medical Imaging Tan-Hanh Pham Chris Ngo Trong-Duong Bui, Minh Luu Quang Tan-Huong Pham Truong-Son Hy text, Florida Institute of Technology, USA, Knovel Engineering Lab, Singapore, Vietnam Military Medical University, 108 Military Central Hospital, Vietnam, Can Tho University of Medicine and Pharmacy, Vietnam, University of Alabama at Birmingham, USA text Abstract Medical Visual Language Models have shown great potential in various healthcare applications, including medical image captioning and diagnostic assistance. However, most existing models rely on text-based instructions, limiting their usability in real-world clinical environments especially in scenarios such as surgery, text-based interaction is often impractical for physicians. In addition, current medical image analysis models typically lack comprehensive reasoning behind their predictions, which reduces their reliability for clinical decision-making. Given that medical diagnosis errors can have life-changing consequences, there is a critical need for interpretable and rational medical assistance. To address these challenges, we introduce an end-to-end speech-driven medical VLM, SilVar-Med, a multimodal medical image assistant that integrates speech interaction with VLMs, pioneering the task of voice-based communication for medical image analysis. In addition, we focus on the interpretation of the reasoning behind each prediction of medical abnormalities with a proposed reasoning dataset. Through extensive experiments, we demonstrate a proof-of-concept study for reasoning-driven medical image interpretation with end-to-end speech interaction . We believe this work will advance the field of medical AI by fostering more transparent, interactive, and clinically viable diagnostic support systems. Our code and dataset are publicly available at SiVar-Med . * * footnotetext: Corresponding authors: [email protected], [email protected] ${\dagger}$ ${\dagger}$ footnotetext: Project lead 1 Introduction Recently, advancements in Visual Language Models (VLMs) have demonstrated the potential of Large Language Models (LLMs) to process both images and text at the same time [ 26 , 39 , 2 , 4 , 31 ] . In the medical domain, VLMs have gained increasing attention for their ability to facilitate intuitive human-machine interactions such as MedBLIP [ 11 ] , Med-flamingo [ 33 ] , Llava-Med [ 25 ] , improving clinical decision-making and diagnostic assistance. These models are particularly valuable for medical imaging analysis, where they can process complex radiological images — such as X-ray [ 24 , 19 ] , MRI, and CT scans [ 48 ] — and generate meaningful textual descriptions. By leveraging deep learning techniques, VLMs can assist professionals in interpreting medical images, identifying abnormalities, and supporting diagnostic workflows. Despite these advancements, most existing medical VLMs remain limited to text-based interactions, which may not be optimal in time-sensitive clinical settings or for visually impaired users. While some proprietary VLMs, such as GPT-4o [ 36 ] and Gemini [ 43 ] , support speech-driven interactions, they are not open-source, restricting fine-tuning for downstream tasks. Recently, SilVar [ 37 ] , a speech-driven multimodal model for reasoning-based visual question answering and object localization, has emerged as a pioneering effort in the field. Despite its potential applications in the medical domain, speech-based medical instruction for VLMs remains underexplored in open-source research, and existing models lack the capability to process and reason through spoken queries effectively. Furthermore, while there are some several benchmarks, have been introduced to evaluate the performance of medical VLMs such as MultiMedEval [ 41 ] , MultiMedQA [ 42 ] , OmniMedVQA [ 18 ] , existing evaluation methods primarily focus on image captioning tasks and are limited in assessing the reasoning behind predictions. In addition, commonly used datasets such as SLAKE [ 29 ] , VQA-Med [ 7 ] , VQA-RAD [ 23 ] , and PathVQA [ 17 ] primarily evaluate models using text-based instructions with short-answer responses, often without requiring deeper reasoning or justification. To address this limitation, LLaVA-Med [ 25 ] introduced a medical chat assistant capable of answering open-ended research questions, but its functionality is restricted to image-text inputs and limited by the number of supported medical image modalities. OmniMedVQA [ 18 ] , on the other hand, aggregates multiple available datasets to create a larger benchmark for multiple-choice question-answering tasks and utilizes LLMs as judges for evaluation. However, both approaches lack a structured framework for evaluating reasoning abilities in medical VLMs, particularly in the context of abnormality detection. To bridge this gap, we propose SilVar-Med, an end-to-end speech-instructed medical VLM that enables users to interact with the model verbally. Our approach not only introduces speech-driven interaction but also concentrates in reasoning abnormality detection by incorporating structured reasoning into predictions. To this end, we introduce a demonstrated dataset, designed for reasoning abnormality detection through speech instructions. Additionally, we propose a novel evaluation metric that leverages LLMs as judges to assess the reasoning capabilities of medical VLMs. Our contribution is summarized as follows: • We propose SilVar-Med, a speech-driven medical VLM that enables intuitive human-machine interaction in healthcare. • We focus on investigating the model’s reasoning abilities behind abnormality detection, addressing the limitations of predictions without explanations or short predictions. • We introduce a dataset for speech-instructed medical abnormality detection, enhancing multimodal learning in medical AI. • We propose a comprehensive reasoning evaluation metric together with LLMs as judges for medical VLMs. 2 Related Work 2.1 Medical Vision Language Models Over the past five years, there has been a rapid development of LLMs and VLMs such as Gemini and GPT-4 [ 9 , 1 ] , alongside the emergence of open-source models like the Llama family [ 44 , 45 , 15 ] , Mistral family [ 20 ] , Qwen family [ 6 , 50 ] , and Vicuna [ 56 ] . These models have significantly advanced natural language understanding, but their capabilities have been further extended by VLMs, which integrate visual and textual modalities [ 26 ] . VLMs enable models to process both images and text, enhancing applications such as visual question answering (VQA), medical image interpretation, and image captioning. There are many VLMs including Flamingo [ 2 ] , BLIP [ 30 ] , MiniGPT-v2 [ 10 ] , MiniGPT-4 [ 57 ] , LLaVA [ 31 ] , and InternVL [ 13 ] , which have demonstrated remarkable progress in general-domain visual-language tasks. Inspired by these advancements, researchers have developed domain-specific VLMs for medical applications [ 54 , 11 , 46 ] . One of the pioneering studies in this field is Med-Flamingo [ 33 ] , which extends Flamingo to the medical domain by pretraining on multimodal knowledge sources spanning various medical disciplines. Similarly, LLaVA-Med [ 25 ] filters image-text pairs from PMC-15M [ 54 ] to train a biomedical-specialized VLM leveraging LLaVA-pretrained parameters. In addition to medical image report generation and medical image captioning models [ 51 ] , MiniGPT-Med [ 3 ] and Lite-GPT [ 24 ] extend MiniGPTs [ 10 , 57 ] to generate bounding boxes along with predictions, enabling localized abnormality detection. Furthermore, Merlin [ 8 ] is one of the pioneering models for 3D VLMs, capable of processing 3D medical images alongside their corresponding textual radiology reports, along with RadFM [ 47 ] . Other notable studies, such as PubMedCLIP [ 16 ] , BiomedCLIP [ 54 ] , and BiomedGPT [ 53 ] , have also contributed to the adaptation of general-domain VLMs for medical applications. However, most