MedROV: Towards Real-Time Open-Vocabulary Detection Across Diverse Medical Imaging Modalities
Tooba Tehreem Sheikh, Jean Lahoud, Rao Muhammad Anwer, Fahad Shahbaz Khan, Salman Khan, Hisham Cholakkal, Mohamed Bin Zayed University of Artificial Intelligence (MBZUAI), {tooba.sheikh, jean.lahoud, rao.anwer, fahad.khan, salman.khan, hisham.cholakkal}, @mbzuai.ac.ae · 2025
Read MedROV: Towards Real-Time Open-Vocabulary Detection Across Diverse Medical Imaging Modalities by Tooba Tehreem Sheikh, Jean Lahoud, Rao Muhammad Anwer, Fahad Shahbaz Khan, Salman Khan, Hisham Cholakkal, Mohamed Bin Zayed University of Artificial Intelligence (MBZUAI), {tooba.sheikh, jean.lahoud, rao.anwer, fahad.khan, salman.khan, hisham.cholakkal}, @mbzuai.ac.ae (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-20650.txt.
Introduction
Tooba Tehreem Sheikh and co-authors at MBZUAI present in 2025 MedROV, a detector that finds objects in medical scans even when it was never trained on their names. Ordinary detectors only recognise a fixed list of categories. A scan showing pneumonia is missed if pneumonia was never on that list. MedROV instead takes a text description at run time and searches the image for whatever the words describe, across scan types from X-rays to microscope slides.
To teach it, the team built Omnis, a training set of 600,000 labelled examples drawn from 35 datasets covering nine kinds of medical imaging, with machine-generated labels filling gaps. MedROV beat the previous best medical foundation model by an average of 40 points on the mAP50 accuracy score. It also topped closed-set detectors by more than 3 points on the same measure, and runs at 70 frames per second, fast enough for live clinical use.
Catalog record
| CREATOR | Tooba Tehreem Sheikh, Jean Lahoud, Rao Muhammad Anwer, Fahad Shahbaz Khan, Salman Khan, Hisham Cholakkal, Mohamed Bin Zayed University of Artificial Intelligence (MBZUAI), {tooba.sheikh, jean.lahoud, rao.anwer, fahad.khan, salman.khan, hisham.cholakkal}, @mbzuai.ac.ae |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-20650
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MedROV: Towards Real-Time Open-Vocabulary Detection Across Diverse Medical Imaging Modalities Tooba Tehreem Sheikh, Jean Lahoud, Rao Muhammad Anwer, Fahad Shahbaz Khan, Salman Khan, Hisham Cholakkal, Mohamed Bin Zayed University of Artificial Intelligence (MBZUAI), {tooba.sheikh, jean.lahoud, rao.anwer, fahad.khan, salman.khan, hisham.cholakkal}, @mbzuai.ac.ae Abstract Traditional object detection models in medical imaging operate within a closed-set paradigm, limiting their ability to detect objects of novel labels. Open-vocabulary object detection (OVOD) addresses this limitation but remains underexplored in medical imaging due to dataset scarcity and weak text-image alignment. To bridge this gap, we introduce MedROV, the first Real-time Open Vocabulary detection model for medical imaging. To enable open-vocabulary learning, we curate a large-scale dataset, Omnis, with 600K detection samples across nine imaging modalities and introduce a pseudo-labeling strategy to handle missing annotations from multi-source datasets. Additionally, we enhance generalization by incorporating knowledge from a large pre-trained foundation model. By leveraging contrastive learning and cross-modal representations, MedROV effectively detects both known and novel structures. Experimental results demonstrate that MedROV outperforms the previous state-of-the-art foundation model for medical image detection with an average absolute improvement of 40 mAP50, and surpasses closed-set detectors by more than 3 mAP50, while running at 70 FPS, setting a new benchmark in medical detection. Our source code, dataset, and trained model are available at MedROV . Figure 1 : Comparison of Traditional Object Detection and MedROV: Traditional detectors are trained on a fixed set of categories and cannot recognize unseen classes. For example, the model shown here detects the left and right lungs but fails to detect pneumonia, which is present in the image. In contrast, MedROV is a Real-time Open Vocabulary detection model for medical imaging that leverages the BioMedCLIP foundation model to enable detection of both seen and unseen classes. At inference, it can detect any class described by a text prompt, if present in the image. 1 Introduction Object detection in medical imaging plays an important role in identifying abnormalities such as tumors, fractures, and diseased cells across diverse modalities, including CT scans, X-rays, MRIs, and histopathology slides. Unlike natural images, medical imaging poses unique challenges due to its multi-modal nature, where each modality has distinct visual and semantic characteristics. Traditional object detection models, including single-stage [ 40 , 54 ] and two-stage [ 41 , 17 ] detectors, have been adapted for medical tasks like tumor detection and organ localization. However, these models are limited to detecting only predefined categories (closed-set detection), making them ineffective in real-world medical scenarios where new, critical abnormalities may emerge and require immediate detection. On the other hand, Open Vocabulary Object Detection (OVOD) addresses the limitations of closed-set detection models by enabling the detection of novel objects through vision-language alignment [ 10 ] or region-level vision-language pre-training [ 26 ] . While OVOD has shown significant potential in natural images, its application to medical imaging remains largely unexplored. Adapting OVOD methods, such as YOLO-World, to the medical domain presents challenges due to the scarcity of large, diverse, and well-annotated image-text datasets necessary for learning meaningful visual-language associations. Additionally, medical images exhibit complex variations in object size and shape, overlapping objects, and imbalances in class distribution, making the application of OVOD in the medical field more challenging. To address these challenges, we introduce MedROV, the first Real-time Open Vocabulary detection model for medical imaging, designed to detect both known and novel structures across nine imaging modalities (Fig. 1 ). Trained on a large-scale dataset of over 600K samples, our model outperforms existing OVOD methods, previous state-of-the-art medical detection methods, and closed-set detectors by leveraging the BioMedCLIP [ 52 ] foundation model to enhance open vocabulary detection. In summary, our main contributions are as follows: • We introduce the first open-vocabulary object detector for medical images, capable of detecting both known and unknown structures, by curating Omnis, a large-scale dataset of over 600K detection samples across 9 imaging modalities (CT, MRI, X-ray, Ultrasound, Histopathology, Dermoscopy, Fundoscopy, Endoscopy, and Microscopy). • We adapt YOLO-World, an open vocabulary detector originally designed for natural images, to the medical domain by training it on our dataset and addressing the challenge of missing annotations when integrating multiple datasets across different modalities. • We improve the model’s detection and generalization performance by incorporating information from the BioMedCLIP foundation model, leveraging its vision-language features for improved performance. • We conduct extensive experiments comparing our method with existing OVOD approaches, the previous state-of-the-art model [ 53 ] , and closed-set detectors. Our model achieves significant improvement in zero-shot detection performance on the Omnis test set compared to the baseline YOLO-World, while maintaining comparable speed (YOLO-World: 72 FPS, Ours: 70 FPS). Additionally, it outperforms the BioMedParse foundation model by an average absolute improvement of 40 mAP50 and surpasses closed-set methods by more than 3 mAP50. 2 Related Work Recent advances in deep learning have significantly improved performance across various medical imaging tasks, including classification, segmentation, and detection. Object detection, in particular, plays a crucial role in identifying anatomical structures and pathological abnormalities within medical scans. While traditional object detectors have been successfully adapted from natural images to medical domains, they largely operate under closed-set assumptions, limiting their ability to generalize to unseen or rare categories. Meanwhile, the emergence of large-scale foundation models and open-vocabulary object detection techniques has enabled the development of more flexible and generalizable systems. In this section, we review prior work in three relevant areas: object detection in medical imaging, the application of foundation models to medical tasks, and recent progress in open-vocabulary object detection. Figure 2 : An overview of the Omnis 600K dataset. We curate a large-scale object detection dataset for medical imaging by incorporating 35 datasets with diverse modalities (represented in different colors), anatomical regions (displayed at the top), and target areas (indicated within the bars). 2.1 Object Detection in Medical Images Object detection is essential in medical imaging, allowing for the accurate detection of tumors, lesions, and other abnormalities. Deep learning-based detectors in natural images have been adapted for medical imaging. RT-DETR with multi-scale feature extraction has been applied to diabetic retinopathy detection [ 18 ] , while BGF-YOLO [ 21 ] and SOCR-YOLO [ 27 ] have introduced enhancements to the YOLO architecture for brain tumor detection and lesion detection, respectively. To further enhance detection accuracy across varying object sizes and complex backgrounds, recent work by [ 49 ] proposed a cross-scale attention and multi-layer feature fusion method based on YOLOv8 for skin disease detection. However, these approaches remain limited to closed-set object detection, restricting their ability to identify novel or previously unseen structures. 2.2 Foundation Models in Medical Images Recent foundation models have advanced medical imaging through multi-modal learning and self-supervised training. MedSAM [ 28 ] adapts SAM for medical image segmentation, while BioMedCLIP [ 52 ] , trained on the PMC15M dataset, enhances CLIP for biomedical image understanding. MEDITRON [ 9 ] , a large-scale medical language model, outperformed previous state-of-the-art models on USMLE-style questions. MedPaLM-2 [ 43 ] , an adaptation of PaLM-2 for healthcare, demonstrates strong performance in clinical question answering. Additionally, BioMedParse [ 53 ] , a biomedical foundation model for image parsing, introduces a unified approach for joint segmentation, detection, and recognition. However, most of these models remain limited to