Beyond CLIP: Knowledge-Enhanced Multimodal Transformers for Cross-Modal Alignment in Diabetic Retinopathy Diagnosis

Argha Kamal Samanta, Harshika Goyal, Vasudha Joshi, Tushar Mungle, Pabitra Mitra · 2025

Read Beyond CLIP: Knowledge-Enhanced Multimodal Transformers for Cross-Modal Alignment in Diabetic Retinopathy Diagnosis by Argha Kamal Samanta, Harshika Goyal, Vasudha Joshi, Tushar Mungle, Pabitra Mitra (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-19663.txt.

Introduction

Argha Kamal Samanta and four co-authors build a diagnostic aid for diabetic retinopathy, a leading cause of preventable blindness that damages the retina. General vision-language models like CLIP, trained on everyday images, align medical pictures and text poorly, which breaks tasks like searching scans by description. The team's framework instead trains jointly on retinal photographs, clinical narratives, and structured patient data.

Separate encoders handle each stream, a vision transformer for images, a biomedical language model for text, and a small network for patient features, fused inside one transformer. Contrastive, reconstruction, and severity-classification losses train the whole system together. On a Brazilian ophthalmology dataset, text-to-image retrieval jumps from near-useless baseline levels to near-perfect recall, severity grading stays state of the art, and an unseen second dataset confirms the gains generalize.

Catalog record

CREATORArgha Kamal Samanta, Harshika Goyal, Vasudha Joshi, Tushar Mungle, Pabitra Mitra
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-19663

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Beyond CLIP: Knowledge-Enhanced Multimodal Transformers for Cross-Modal Alignment in Diabetic Retinopathy Diagnosis Argha Kamal Samanta, Harshika Goyal, Vasudha Joshi, Tushar Mungle, Pabitra Mitra Abstract Diabetic retinopathy (DR) is a leading cause of preventable blindness worldwide, necessitating accurate automated diagnostic systems. While general-domain vision-language models like Contrastive Language-Image Pre-Training (CLIP) have demonstrated impressive capabilities in natural image tasks, they exhibit severe limitations in medical domain applications, particularly in cross-modal retrieval for ophthalmological images. This paper presents a novel knowledge-enhanced joint embedding framework that integrates retinal fundus images, clinical text, and structured patient data through a multimodal transformer architecture to address the critical gap in medical image-text alignment. Our approach employs separate encoders for each modality: a Vision Transformer (ViT-B/16) for retinal images, Bio-ClinicalBERT for clinical narratives, and a multilayer perceptron for structured demographic and clinical features. These modalities are fused through a joint transformer with modality-specific embeddings, trained using multiple objectives including contrastive losses between modality pairs, reconstruction losses for images and text, and classification losses for DR severity grading according to ICDR and SDRG schemes. Experimental results on the Brazilian Multilabel Ophthalmological Dataset (BRSET) demonstrate significant improvements over baseline models. Our framework achieves near-perfect text-to-image retrieval performance with Recall@1 of 99.94% compared to fine-tuned CLIP’s 1.29%, while maintaining state-of-the-art classification accuracy of 97.05% for SDRG and 97.97% for ICDR. Furthermore, zero-shot evaluation on the unseen DeepEyeNet dataset validates strong generalizability with 93.95% Recall@1 versus 0.22% for fine-tuned CLIP. These results demonstrate that our multimodal training approach effectively captures cross-modal relationships in the medical domain, establishing both superior retrieval capabilities and robust diagnostic performance. I Introduction Diabetic retinopathy (DR) represents one of the most prevalent microvascular complications of diabetes mellitus and stands as a leading cause of preventable blindness in working-age adults globally. With the International Diabetes Federation projecting the number of people with diabetes to reach 700 million by 2045, the burden of DR-related vision impairment continues to escalate dramatically. Early detection and timely intervention are crucial for preventing irreversible vision loss, yet traditional screening methods face significant challenges in scalability, accessibility, and cost-effectiveness. Manual examination of retinal fundus images by trained ophthalmologists remains the gold standard for DR diagnosis. However, this approach is time-consuming, expensive, and subject to inter-observer variability. More critically, the shortage of specialized ophthalmologists, particularly in low- and middle-income countries where diabetes prevalence is rising fastest, creates a substantial gap between screening demand and available resources. These limitations have motivated the development of automated screening systems powered by artificial intelligence. Recent advances in deep learning and computer vision have demonstrated remarkable success in automated medical image analysis. Vision-language models such as CLIP (Contrastive Language-Image Pre-training) have revolutionized multi-modal learning in natural image domains, achieving impressive zero-shot and few-shot capabilities by learning joint representations of images and text [ 21 ] . However, despite their success in general domains, these models face critical limitations when applied to specialized medical contexts, particularly in ophthalmology. Our preliminary investigations revealed a striking performance gap: while CLIP maintains high classification accuracy on retinal images, it catastrophically fails at cross-modal retrieval tasks that are essential for clinical decision support. Specifically, CLIP without fine-tuning achieves 0% Recall@1 in text-to-image retrieval for retinal images, and even after domain-specific fine-tuning, performance only reaches 1.29%. This severe misalignment between clinical descriptions and retinal images fundamentally limits the practical utility of such models in real-world clinical workflows where physicians need to retrieve relevant images based on clinical descriptions, generate reports from images, and integrate multi-modal patient data for comprehensive diagnosis. The fundamental challenge stems from the fact that medical images contain highly specialized visual features like microaneurysms, exudates, hemorrhages, neovascularization, that require domain-specific knowledge to align with corresponding clinical terminology. General-purpose vision-language models trained on web-scraped image-text pairs lack this medical domain knowledge and fail to capture the fine-grained pathological features critical for clinical understanding. Moreover, most existing DR detection systems focus exclusively on image-based analysis, neglecting the wealth of complementary information available in electronic health records, including structured clinical variables such as age, diabetes duration, insulin use, and patient demographics. In this work, we propose a comprehensive knowledge-enhanced joint embedding framework that addresses these limitations by integrating three distinct modalities: retinal fundus images, clinical text narratives, and structured patient data. Our approach employs modality-specific , i.e., a Vision Transformer for images, Bio-ClinicalBERT for clinical text, and a multilayer perceptron for structured features, combined with a novel joint transformer fusion mechanism that enables cross-modal attention and contextualization [ 9 , 3 ] . Through multi-objective training incorporating contrastive alignment losses, modality reconstruction losses, and supervised classification losses with learnable dynamic weighting, our model learns robust joint representations that capture complementary diagnostic signals across modalities. Evaluated on the Brazilian Multilabel Ophthalmological Dataset (BRSET) containing 16,266 fundus images from 8,524 patients [ 17 ] , our framework demonstrates dramatic improvements in cross-modal retrieval while maintaining state-of-the-art classification performance. Specifically, we achieve 99.94% Recall@1 in text-to-image retrieval compared to fine-tuned CLIP’s 1.29%, while maintaining classification accuracies of 97.05% for SDRG and 97.97% for ICDR grading schemes. Furthermore, zero-shot evaluation on the completely unseen DeepEyeNet dataset validates strong generalizability with 93.95% Recall@1, establishing the robustness of our learned representations [ 25 ] . The main contributions of this paper are: (1) a unified multimodal architecture that effectively integrates heterogeneous medical data types through modality-aware fusion; (2) empirical demonstration that general-domain vision-language models fail critically at medical cross-modal alignment tasks; (3) a multi-task learning strategy with dynamic loss weighting that jointly optimizes alignment, reconstruction, and classification objectives; (4) comprehensive evaluation demonstrating better retrieval performance without sacrificing diagnostic accuracy; and (5) strong cross-dataset generalization validating the potential for real-world clinical deployment. The remainder of this paper is organized as follows: Section II reviews related work in deep learning for DR detection, vision-language models, and multi-modal learning in healthcare. Section III describes the methodology including data preprocessing and model architecture. Section IV presents experimental results on retrieval and classification tasks. Section V concludes with discussion of findings and future directions. II Related Work II-A Deep Learning for Diabetic Retinopathy Detection The application of deep learning to automated DR screening has evolved significantly over the past decade. Early pioneering work demonstrated that deep learning algorithms could achieve sensitivity and specificity comparable to human experts in detecting DR from retinal fundus photographs [ 10 ] . Recent systems like DeepDR have advanced the field by incorporating real-time image quality assessment, lesion detection, and multi-grade classification, achieving area under the curve values ranging from 0.901 to 0.972 across different DR severity levels [ 7 ] . Convolutional neural networks, particularly architectures based on ResNet, DenseNet, Inception, and EfficientNet, have been extensively studied for DR classification tasks [ 20 ] . Transfer learning from ImageNet has proven particularly effective for addressing limited medical imaging data, with models achieving classification accuracies exceeding 97% on benchmark datasets. Advanced activation functions and data augmentation strategies have further improved performance, with recent approaches reporting accuracy as high as 99.41% on specialized datasets [ 4 ] . However, most existing approaches focus exclusively on image-based classification, neglecting complementary clinical information. II-B Vision Transformers in Medical Imaging The remarkable performance of transformer architectures in natural language processing has triggered broad interest in computer vision,

Read the complete text