CWCD: Category-Wise Contrastive Decoding for Structured Medical Report Generation

Shantam Srivastava, Mahesh Bhosale, David Doermann, Mingchen Gao · 2026

Read CWCD: Category-Wise Contrastive Decoding for Structured Medical Report Generation by Shantam Srivastava, Mahesh Bhosale, David Doermann, Mingchen Gao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-10410.txt.

Introduction

Srivastava, Bhosale, Doermann, and Gao sharpen machine written chest X-ray reports. Single pass writing lets word habits crowd out image cues, spawning false linked findings. Their CWCD method writes one report per disease group, pitting plain scans against masked scans. Group wise visual cues steer each draft, cutting the drift toward stock phrases. Tests beat base methods on both clinical value and plain language scores.

Catalog record

CREATORShantam Srivastava, Mahesh Bhosale, David Doermann, Mingchen Gao
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41617
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.10410

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CWCD: Category-Wise Contrastive Decoding for Structured Medical Report Generation Shantam Srivastava, Mahesh Bhosale, David Doermann, Mingchen Gao Abstract Interpreting chest X-rays is inherently challenging due to the overlap between anatomical structures and the subtle presentation of many clinically significant pathologies, making accurate diagnosis time-consuming even for experienced radiologists. Recent radiology-focused foundation models, such as LLaVA-Rad and Maira-2, have positioned multi-modal large language models (MLLMs) at the forefront of automated radiology report generation (RRG). However, despite these advances, current foundation models generate reports in a single forward pass. This decoding strategy diminishes attention to visual tokens and increases reliance on language priors as generation proceeds, which in turn introduce spurious pathology co-occurrences in the generated reports. To mitigate these limitations, we propose C ategory- W ise C ontrastive D ecoding ( CWCD ), a novel and modular framework designed to enhance structured radiology report generation (SRRG). Our approach introduces category-specific parameterization and generates category-wise reports by contrasting normal X-rays with masked X-rays using category-specific visual prompts. Experimental results demonstrate that CWCD consistently outperforms baseline methods across both clinical efficacy and natural language generation metrics. An ablation study further elucidates the contribution of each architectural component to overall performance. \midlauthor The Department of Computer Science and Engineering University at Buffalo, The State University of New York, NY, USA † † volume: – 73 † † year: 2026 † † workshop: Full Paper – MIDL 2026 † † editors: Accepted for publication at MIDL 2026 keywords Radiology Report Generation, Multimodal Large Language Models, Contrastive Decoding, Chest X-rays. 1 Introduction Over the past two decades, the rapid advancement of Artificial Intelligence (AI) has significantly improved automated interpretation of medical images 58 ; 37 , particularly chest X-rays, which remain one of the most frequently performed diagnostic procedures worldwide 4 . Chest X-rays are highly valued due to their low cost, minimal radiation exposure, and ability to provide substantial clinical information. Despite these advantages, generating radiology reports remains a cognitively demanding and time-consuming task 38 . Compounding this challenge, the growing demand for interpreting chest X-rays has outpaced the supply of radiologists 10 , leaving many radiologists overworked and vulnerable to fatigue 67 . Automated Radiology Report Generation (RRG), the task of producing free-text descriptions of visual observations from a radiology image, such as a chest X-ray, has therefore emerged as an essential research direction 31 ; 43 . However, automated RRG remains fundamentally challenging: unlike natural images, chest X-rays exhibit low contrast and may contain subtle, highly localized pathologies. The requirement to generate long, unconstrained textual reports imposes additional demands on model fidelity. Unlike visual question answering, which operates within relatively short, focused outputs, comprehensive radiology findings reports may exceed 200 tokens and the model must reason jointly over multiple, often overlapping, anatomical regions. Early encoder-decoder approaches 74 ; 30 established a strong foundation and were able to generate linguistically cohesive reports, however, they often lagged in clinical efficacy 73 . The rise of Large Language Models (LLMs) 56 ; 64 and subsequently multi-modal LLMs (MLLMs) 45 ; 1 enabled the development of the first generation of radiology foundation models 70 ; 8 ; 28 ; 54 ; 69 . These models leveraged the superior language modeling and linguistic reasoning capabilities of LLMs and substantially scaled parameter counts to surpass the then state-of-the-art encoder-decoder models. They delivered remarkable improvements in clinical efficacy metrics and demonstrated stronger generalization performance on out-of-distribution datasets 54 . The second generation of radiology foundation models further advanced performance: 75 employed GPT-4 51 to refine training data by removing temporal comparisons, references to prior exams and unnecessary language variations, while 2 expanded the textual context to include indications, technique and comparison, and the visual context by including lateral and prior frontal views. Despite these advances, these foundation models remain constrained by a core limitation of MLLMs: the reduction in attention values over image tokens as more tokens are generated 18 ; 11 . Motivation. We observe that, as report generation progresses, the model’s attention increasingly relies on prior linguistic context rather than the image information. The maximum weight in multi-head attention layer 65 can be interpreted as a signal of the model’s strong confidence in the corresponding input token 66 ; 22 . Based on this insight, we define Layer-Averaged Max Attention (LAMA) , which can be computed over any subset of target tokens S (e.g., image tokens or generated text tokens). Let A^{(l,h)}_{t}\in\mathbb{R}^{N} denote the attention weights for generated token t in layer l and head h . Then the LAMA score at step t is: \text{LAMA}_{t}(S)=\frac{1}{L}\sum_{l=1}^{L}\max_{h}\left(\sum_{i\in S}A^{(l,h)}_{t}[i]\right). (1) From the MIMIC-CXR 32 dataset, we compute \text{LAMA}_{t}(S_{\text{vis}}) , where S_{\text{vis}} denotes the set of all image tokens, for 100 randomly sampled X-rays from the test set. We observe a clear downward trend in \text{LAMA}_{t}(S_{\text{vis}}) over the generation steps (Fig. ), suggesting a decay in attention to the image tokens during the generation process, accompanied by an increase in attention over the language priors. We hypothesize that this causes the model to learn spurious co-occurrences of pathology due to inherent biases in the training datasets. A typical example of such spurious pathology co-occurrence arises with cardiomegaly and pulmonary edema. In many cases, these two findings frequently appear together because both are associated with congestive heart failure 60 . As a result, when the model increasingly relies on textual priors, the presence of cardiomegaly alone serves as a language cue that strongly biases subsequent tokens toward the associated pathology (pulmonary edema in this case), even if the visual evidence is absent. Similarly, pleural effusion (fluid accumulation) can mechanically lead to some degree of rounded atelectasis (lung collapse) due to compression 48 . This statistical co-occurrence can also lead the model to generate spurious findings simply because they commonly appear together in the training distribution, rather than being grounded in the underlying image evidence. Given these observations, we introduce Category-Wise Contrastive Decoding , a novel and modular method that is designed to enhance structured findings generation in radiology foundation models. Category-Wise Contrastive Decoding aims to mitigate the problems of generating spurious co-occurrences and reduced attention on visual tokens with increase in output length in two ways: (i) Category-Specific Parametrization - We generate a findings report category-wise under eight anatomical headers, as defined by 14 : Lungs and Airways, Pleura, Cardiovascular, Hila and Mediastinum, Tubes, Catheters, and Support Devices, Musculoskeletal and Chest Wall, Abdominal, and Other. Henceforth, we refer to these anatomical headers as categories of a structured radiology report. (ii) Masked Contrastive Decoding - An inference time strategy, where instead of normal greedy decoding, we sample from a contrasted distribution obtained by masking the X-ray using category-specific visual prompts. Introducing a contrastive objective at inference time prevents hallucinations arising from prior language bias learned during training. 2 Methods Vision Language Modeling. Large language models (LLMs) process sequences of text tokens to generate textual output in an autoregressive manner. This mechanism can be extended to images by adding a vision encoder that extracts visual features, which are then projected into the text embedding space so they can be fed to the language model (LM) as additional input tokens. In practice, this is done by using a pre-trained vision backbone (e.g., ViT 17 or

Read the complete text