DREAM: Dynamic Retinal Enhancement with Adaptive Multi-modal Fusion for Expert Precision Medical Report Generation

Nagur Shareef Shaik Teja Krishna Cherukuri, Dong Hye Ye · 2026

Read DREAM: Dynamic Retinal Enhancement with Adaptive Multi-modal Fusion for Expert Precision Medical Report Generation by Nagur Shareef Shaik Teja Krishna Cherukuri, Dong Hye Ye (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-17209.txt.

Introduction

Shaik, Cherukuri, and Ye introduce DREAM, a framework for medical report generation from retinal images that works even with limited data. Its Abstractor maps image and keyword features, with keywords curated by ophthalmologists, into one space, enriching visual data with pathology insights. Its Adaptor then fuses modalities, weighting each by parameters into one representation, while Contrastive Alignment aligns fused outputs with true reports in training. DREAM sets a new best mark on the DeepEyeNet benchmark with a BLEU score of 0.241, and generalizes well to the ROCO dataset.

Catalog record

CREATORNagur Shareef Shaik Teja Krishna Cherukuri, Dong Hye Ye
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32472
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.17209

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DREAM: Dynamic Retinal Enhancement with Adaptive Multi-modal Fusion for Expert Precision Medical Report Generation Thanks: Equal Contribution Thanks: Corresponding Author: [email protected] Nagur Shareef Shaik Teja Krishna Cherukuri, Dong Hye Ye Abstract Automating medical reports for retinal images requires a sophisticated blend of visual pattern recognition and deep clinical knowledge. Current Large Vision-Language Models (LVLMs) often struggle in specialized medical fields where data is scarce, leading to models that overfit and miss subtle but critical pathologies. To address this, we introduce DREAM (Dynamic Retinal Enhancement with Adaptive Multi-modal Fusion), a novel framework for high-fidelity medical report generation that excels even with limited data. DREAM employs a unique two-stage fusion mechanism that intelligently integrates visual data with clinical keywords curated by ophthalmologists. First, the Abstractor module maps image and keyword features into a shared space, enhancing visual data with pathology-relevant insights. Next, the Adaptor performs adaptive multi-modal fusion, dynamically weighting the importance of each modality using learnable parameters to create a unified representation. To ensure the model’s outputs are semantically grounded in clinical reality, a Contrastive Alignment module aligns these fused representations with ground-truth medical reports during training. By combining medical expertise with an efficient fusion strategy, DREAM sets a new state-of-the-art on the DeepEyeNet benchmark, achieving a BLEU-4 score of 0.241, and further demonstrates strong generalization to the ROCO dataset. I Introduction The timely and accurate interpretation of retinal images is fundamental to ophthalmology, playing a critical role in diagnosing diseases that can cause irreversible blindness such as diabetic retinopathy, age-related macular degeneration, and glaucoma [ 9 ] . Automated report generation has the potential to ease the documentation burden on clinicians, accelerate population-level screening programs, and improve access to specialized care in under-resourced regions. However, deploying such systems in real clinical environments demands a stringent combination of properties: high diagnostic accuracy, faithful grounding in the image content, interpretability for clinician trust, and computational efficiency for integration into standard hospital infrastructure. Meeting all of these simultaneously remains an open challenge. Large Vision-Language Models (LVLMs) have shown great promise in automating vision-to-text generation, yet their generalized pre-training often prevents them from capturing the nuanced details of specific ophthalmic conditions [ 2 , 11 , 9 ] . A key challenge is infusing these models with specialized domain knowledge [ 19 ] . Integrating expert-defined clinical keywords is a potent strategy, but it creates the complex problem of how to dynamically fuse these distinct data modalities in a manner sensitive to the context of each image [ 21 ] . This difficulty is magnified in ophthalmology, where datasets are small relative to natural-image corpora, and where a single misinterpreted lesion can translate into a clinically consequential error. A crucial limitation of existing retinal captioning methods is their reliance on static fusion strategies , which combine visual and textual features without the ability to dynamically arbitrate between them based on context. For instance, they cannot shift their focus to rely more on clinical keywords for an ambiguous lesion or on visual data for a clear pathology. On the other hand, generalist LVLMs such as LLaVA-Med [ 11 ] , VisionGPT [ 10 ] , and mPLUG-Owl2 [ 21 ] excel at multi-modal tasks but are often too computationally demanding for practical use in clinical settings, typically requiring billion-parameter backbones and multi-GPU inference. This gap between specialized-but-rigid retinal models and general-but-heavy LVLMs motivates the design of an architecture that is simultaneously adaptive in how it combines modalities and lightweight enough to run on commodity hardware. To address this gap, we propose DREAM (Dynamic Retinal Enhancement with Adaptive Multi-modal Fusion), a framework that advances the state-of-the-art through a novel two-stage dynamic fusion process. The Abstractor module first uses the clinical keywords to enhance pathology-relevant visual features via bidirectional cross-attention. The Adaptor then employs a learnable modality indicator to dynamically weigh each modality’s contribution and integrate them through a decoupled cross-modal attention mechanism. A Contrastive Alignment module ensures clinical fidelity by aligning the fused representation with the global semantics of ground-truth reports, and the final medical report is generated by our compact language model, LLaMA- X . This architecture enables robust, context-aware report generation from limited data with deployment-ready efficiency. The primary contributions of this work are as follows: • An Abstractor module that performs bidirectional cross-attention between image and keyword features to highlight pathology-relevant visual cues. • An Adaptor with a learnable modality indicator and decoupled cross-modal attention, enabling per-sample adaptive fusion rather than static concatenation. • A Contrastive Alignment objective that grounds the fused representation in the global semantics of ground-truth reports, mitigating factual hallucinations. II Related Work II-A Retinal Image Captioning Early attempts at medical image captioning adapted generic encoder-decoder frameworks such as Show and Tell [ 18 ] and Show, Attend and Tell [ 20 ] , which, while foundational, struggled to capture the specialized semantics of ophthalmic pathologies. Domain-specific architectures followed: DeepOpth [ 9 ] and the Deep Context-Encoding model [ 8 ] introduced retinal-specific pipelines but suffered from data variability and limited pathology coverage. Attention-based extensions, including Non-local Attention [ 7 ] and the Expert Transformer [ 19 ] , improved coherence by emphasizing salient regions. Our prior work advanced this line through lesion-focused attention in the Gated Contextual Transformer [ 15 ] , M3T [ 14 ] , and GCS-M3VLT [ 3 ] . Nevertheless, all of these approaches rely on fixed concatenation or weighted sum schemes that cannot dynamically adjust to the relative informativeness of each modality. II-B Large Vision-Language Models in Medical Imaging Recent LVLMs such as VisualGPT [ 2 ] , LLaVA-Med [ 11 ] , VisionGPT [ 10 ] , and mPLUG-Owl2 [ 21 ] leverage massive pre-training and billions of parameters to handle diverse multi-modal tasks, including radiology and ophthalmology. While these models generalize well, their adaptation to specialized domains typically requires expensive fine-tuning and substantial inference-time compute, limiting deployment in resource-constrained clinical settings. Moreover, without explicit domain priors, they remain susceptible to factual hallucinations when describing rare pathologies [ 19 ] . II-C Multi-modal Fusion Strategies Multi-modal fusion in medical imaging ranges from early concatenation schemes to cross-attention [ 17 ] and gated mechanisms. Static fusion implicitly assumes a fixed balance between modalities, which is rarely optimal in clinical settings where image quality, lesion visibility, and keyword availability vary case by case. Recent work has explored learnable gates and modality tokens to enable adaptive fusion, but these mechanisms are seldom combined with global semantic grounding through contrastive objectives. DREAM bridges this gap by integrating a learnable modality indicator, decoupled cross-modal attention, and InfoNCE-based alignment [ 12 ] in a unified framework. Fig. 1: Architecture of DREAM: The model first performs Representation Learning : a ConvBase enhanced by Guided Context Attention extracts lesion-focused visual features, while a Transformer Encoder processes keywords. In parallel, the Abstractor maps these features to a shared space, and the Adaptor fuses them using learnable modality parameters. Finally, the Language Model decodes these integrated representations to generate the final medical report. III Methodology We approach medical report generation as a conditional language modeling task, where the goal is to generate a report \hat{\mathbf{R}} given a retinal image \mathbf{V} and a set of clinical keywords \mathbf{L} . Following the DeepEyeNet protocol [

Read the complete text