Distribution-Based Masked Medical Vision-Language Model Using Structured Reports
Shreyank N Gowda, Ruichi Zhang, Xiao Gu, Ying Weng, Lu Yang · 2025
Read Distribution-Based Masked Medical Vision-Language Model Using Structured Reports by Shreyank N Gowda, Ruichi Zhang, Xiao Gu, Ying Weng, Lu Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-21794.txt.
Introduction
Teaching computers to read chest X-rays alongside their reports can power many medical tools, but real reports are messy and findings uncertain. Shreyank Gowda and co-authors build a pre-training model in this 2025 paper that expects that messiness instead of ignoring it.
A large language model first rewrites reports into a clean structure: what the disease is, where it shows, what is seen, and the verdict. The model then learns while tracking uncertainty both inside each data type and between image and text. On several follow-up tasks the approach set new best scores, which the authors credit to the structured reports and the honest handling of doubt.
Catalog record
| CREATOR | Shreyank N Gowda, Ruichi Zhang, Xiao Gu, Ying Weng, Lu Yang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-21794
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Distribution-Based Masked Medical Vision-Language Model Using Structured Reports Shreyank N Gowda, Ruichi Zhang, Xiao Gu, Ying Weng, Lu Yang Abstract Medical image-language pre-training aims to align medical images with clinically relevant text to improve model performance on various downstream tasks. However, existing models often struggle with the variability and ambiguity inherent in medical data, limiting their ability to capture nuanced clinical information and uncertainty. This work introduces an uncertainty-aware medical image-text pre-training model that enhances generalization capabilities in medical image analysis. Building on previous methods and focusing on Chest X-Rays, our approach utilizes structured text reports generated by a large language model (LLM) to augment image data with clinically relevant context. These reports begin with a definition of the disease, followed by the ‘appearance’ section to highlight critical regions of interest, and finally ‘observations’ and ‘verdicts’ that ground model predictions in clinical semantics. By modeling both inter- and intra-modal uncertainty, our framework captures the inherent ambiguity in medical images and text, yielding improved representations and performance on downstream tasks. Our model demonstrates significant advances in medical image-text pre-training, obtaining state-of-the-art performance on multiple downstream tasks. 1 Introduction With rapid advancements in deep learning, computer-aided diagnosis in medicine has seen significant progress across various model architectures. However, these models are often trained on specific anatomical or disease categories, requiring expensive data annotation and re-training when applied to new diseases, which limits their broader applicability. Although deep learning has thrived on large-scale labeled datasets from natural images [ 17 , 14 ] , annotating medical images is a much more time-intensive and costly process. A typical approach involves pre-training on extensive datasets like ImageNet [ 8 ] and fine-tuning on specialized medical datasets [ 32 ] . However, this method often struggles to achieve generalized performance due to the significant domain gap. Medical image analysis stands as a critical area in healthcare, where accurate interpretation can significantly impact clinical outcomes. Traditional methods in medical imaging rely heavily on annotated datasets [ 32 ] , which are costly and time-consuming to curate, especially for new or rare diseases. Recent advances in self-supervised pre-training methods like contrastive predictive coding [ 27 ] and masked language modeling [ 9 ] have shown promise in leveraging large, unlabeled datasets to learn robust image and text representations. While general vision-language models like CLIP [ 29 ] have achieved impressive performance on natural images, they struggle with medical data due to domain-specific language and visual features [ 33 , 13 ] . Existing medical image-text approaches like ConVIRT [ 35 ] , PRIOR [ 7 ] , M& M [ 13 ] and GLoRIA [ 19 ] often overlook the inherent uncertainties present in medical data, where variability in clinical descriptions and visual cues can lead to ambiguous interpretations. Whilst uncertainty has been explored in a wide variety of contexts [ 11 , 34 , 12 , 3 ] , to the best of our knowledge we are the first to explore this on chest X-ray image-text pre-training. We propose an uncertainty-aware pre-training model for medical image-text data, focusing on X-ray data. We leverage Distribution-based Masked Image-Language Modeling (D-MLM) to capture both inter- and intra-modal uncertainties, thus enabling more nuanced understanding and alignment between images and associated text. By treating representations as probabilistic distributions rather than deterministic points, D-MLM allows the model to capture the natural ambiguity and variability in medical data, enhancing its capacity for accurate and robust prediction. Since existing reports have semantic inconsistencies [ 33 , 13 ] , a key component of our approach involves the structured text reports generated by a large language model (LLM) [ 1 ] . We first follow M&M [ 13 ] that takes the original reports and converts them to a series of ‘Observations’ and ‘Verdicts’. To this, we add at the beginning a definition of the disease, followed by an ‘Appearance’ section to guide attention to critical regions in the image, and ending with ‘Observations’ and ‘Verdicts’ that offer conclusive insights. This structured report provides clinically relevant context that anchors the model’s predictions, ensuring that outputs align with medical semantics. We show that using such a structured report significantly improves our overall performance. We use these reports along with their corresponding images to do the pre-training. We show using our approach improves performance on multiple different downstream tasks and different benchmarks. The contributions of this work are threefold: • We introduce D-MLM to effectively model multimodal uncertainty in medical image-text data, enhancing the robustness of pre-trained representations. • We leverage structured reports to provide clinically meaningful context for model predictions, aligning with medical semantics. • Through extensive experiments on downstream tasks, we demonstrate the superiority of our method over traditional deterministic approaches, setting a new standard for multimodal pre-training in the chest X-ray domain. 2 Method This section presents our uncertainty-aware pre-training framework for medical image-text alignment. Our Distribution-based Masked Image-Language Modeling (D-MLM) approach combines LLM-generated structured reports for clinical context with probabilistic representations that capture inherent medical data ambiguity. Figure 1 provides an overview. Figure 1: Overview of our D-MLM framework for medical image-text alignment. (a) Structured Report Generation: An LLM creates standardized reports with disease definitions, appearance guidance, observations, and verdicts. (b) Distribution-Based Representation: Image and text features are encoded as probabilistic distributions with means and variances. (c) D-MLM: The model masks and predicts tokens and patches using distribution-based techniques, optimizing with KL divergence loss to enhance uncertainty modeling. 2.1 Structured Report Generation We generate structured reports for each medical image using an LLM, following a standardized format with three key components. • Definition: This section provides a concise description of the disease or clinical condition under consideration. To generate this content, we prompt the LLM with, “Define [disease name]. Give me only a single paragraph and short definition of the disease.” and the model returns a standardized response as, “Definition: [disease definition].” This provides a concise introduction to the condition, grounding the model in clinically accurate language. • Appearance: This section describes key radiographic features and diagnostic areas, by prompting the LLM with, “What are the distinguishing radiographic signs of [disease name] compared to other similar conditions?” The model then responds with “Radiographic characteristics: [disease-specific radiographic characteristics]” , guiding the model’s focus to relevant visual cues within the image data. • Observations and Verdicts: This part details visual findings and clinical verdicts, anchoring predictions in medical context. Following Masks & Manuscripts [ 13 ] , it emphasizes structured clinical reasoning for text-image alignment. By following this format, the structured reports enhance consistency in text inputs, reduce variability in clinical descriptions, and support the model in achieving precise alignment between image and text features. Details of the definitions and appearances can be found in the link: https://github.com/kini5gowda/MIMIC-CXR-text 2.2 Distribution-Based Masked Image-Language Modeling (D-MLM) Our approach centers on Distribution-based Masked Image-Language Modeling (D-MLM), which represents image and text as probabilistic distributions to capture inter-modal and intra-modal uncertainty in medical data. In D-MLM, image \mathbf{I} is encoded through ImageNet-pretrained ViT-B [ 10 ] , while text \mathbf{T} uses ClinicalBERT [ 2 ] . Both outputs are transformed into multivariate Gaussian distributions, with each token or patch represented as: h_{i}=N(\mu_{i},\sigma_{i}^{2}) (1) where \mu_{i} and \sigma_{i}^{2} are mean and