DiA-gnostic VLVAE: Disentangled Alignment-Constrained Vision Language Variational AutoEncoder for Robust Radiology Reporting with Missing Modalities

Nagur Shareef Shaik, Teja Krishna Cherukuri, Adnan Masood, Dong Hye Ye · 2025

Read DiA-gnostic VLVAE: Disentangled Alignment-Constrained Vision Language Variational AutoEncoder for Robust Radiology Reporting with Missing Modalities by Nagur Shareef Shaik, Teja Krishna Cherukuri, Adnan Masood, Dong Hye Ye (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-05968.txt.

Introduction

Nagur Shareef Shaik and co-authors write radiology reports from medical images even when parts of the clinical picture are missing. Real records often lack a modality, like the patient's context notes. Models also tangle shared and modality-specific features, which yields fused reports with invented findings.

Their model, built on a vision-language variational autoencoder with a mixture of experts, separates shared features from modality-specific ones. A constrained objective keeps those features aligned yet orthogonal, blocking bad fusion. A compact decoder then writes the report from the separated features. On two chest X-ray data sets the scores were competitive with far heavier systems. The authors stress resilience to missing inputs as the design's main virtue.

Catalog record

CREATORNagur Shareef Shaik, Teja Krishna Cherukuri, Adnan Masood, Dong Hye Ye
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-05968

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DiA-gnostic VLVAE: Disentangled Alignment-Constrained Vision Language Variational AutoEncoder for Robust Radiology Reporting with Missing Modalities Nagur Shareef Shaik, Teja Krishna Cherukuri, Adnan Masood, Dong Hye Ye Abstract The integration of medical images with clinical context is essential for generating accurate and clinically interpretable radiology reports. However, current automated methods often rely on resource-heavy Large Language Models (LLMs) or static knowledge graphs and struggle with two fundamental challenges in real-world clinical data: (1) missing modalities , such as incomplete clinical context , and (2) feature entanglement , where mixed modality-specific and shared information leads to suboptimal fusion and clinically unfaithful hallucinated findings. To address these challenges, we propose the DiA-gnostic VLVAE , which achieves robust radiology reporting through Di sentangled A lignment. Our framework is designed to be resilient to missing modalities by disentangling shared and modality-specific features using a Mixture-of-Experts (MoE) based Vision-Language Variational Autoencoder (VLVAE). A constrained optimization objective enforces orthogonality and alignment between these latent representations to prevent suboptimal fusion. A compact LLaMA- X decoder then uses these disentangled representations to generate reports efficiently. On the IU X-Ray and MIMIC-CXR datasets, DiA has achieved competetive BLEU@4 scores of 0.266 and 0.134, respectively. Experimental results show that the proposed method significantly outperforms state-of-the-art models. Introduction Radiology report generation (RRG) is a critical task in medical imaging that aims to produce accurate and comprehensive reports from scans, which can help lessen the burden on radiologists. Despite progress in computer vision and natural language processing, RRG remains a significant challenge due to the need for precise clinical insight and coherent report synthesis. This is often complicated by imbalanced datasets where rare conditions are underrepresented, which can compromise diagnostic reliability ( 58 ) . Early models, such as R2Gen ( 8 ) and CvT2Dis ( 31 ) , relied exclusively on image features, using transformers and contrastive learning to refine visual representations. However, this image-centric approach has difficulty capturing nuanced diseases and integrating clinical reasoning. Subsequent efforts focused on improving vision-language integration. For example, XProNet utilized cross-modal prototypes for alignment ( 49 ) , while METransformer used multiple learnable expert tokens to enhance textual consistency ( 50 ) . Still, these models’ reliance on image-centric patterns can lead to semantic discrepancies and clinical errors, especially when radiographic features of different diseases overlap, due to a lack of contextual grounding. To address these limitations, recent models have begun to incorporate diagnostic context, such as disease pseudo-labels, knowledge graphs, or prior findings. Knowledge-driven approaches like MKSG ( 54 ) and M2KT ( 53 ) use medical knowledge graphs to improve factual accuracy. Context-aware models such as KiUT ( 14 ) , DCL ( 20 ) , EKAGen ( 5 ) , and PromptMRG ( 16 ) have also integrated expert knowledge and prior reports through graphs and prompts. While these methods enhance the clinical relevance of the generated reports, they have several technical constraints. For instance, they often lack explicit disentanglement, making it difficult to separate modality-specific knowledge from shared information. Consequently, the absence of context can lead to incomplete reports due to inefficient multi-modal alignment. Additionally, prompt-based models often depend on templates constructed from pseudo-diagnoses, which limits their adaptability and can significantly increase computational overhead due to their use of Large Language Models (LLMs). Retrieval-augmented methods like SEI ( 24 ) have advanced this area by extracting “factual entities” from a study, retrieving similar past cases, and using them to guide a cross-modal fusion decoder. However, this approach has its own issues. The entity-extraction and retrieval stages can be brittle, and the fusion network does not enforce explicit modality disentanglement or probabilistic feature gating. This leaves the model vulnerable to feature interference within what the authors term an “unstable fusion space”. Furthermore, when contextual information is missing, these models often fall back on deterministic rules instead of a principled probabilistic strategy, which can cause errors from earlier stages to propagate. To tackle these challenges, we introduce the DiA-gnostic VLVAE , designed for robust radiology reporting by leveraging the principle of Disentangled Alignment . To handle missing modalities and dynamic patient states, the framework uses real-time clinical data, including demographics, symptoms, and prior history, as dynamic context. Its core is a Vision-Language Variational Autoencoder ( 27 ) that disentangles modality-specific and shared latent representations, ensuring consistent vision-language alignment even when context is incomplete. This is supported by a Vision-Language Representation Learning module using Guided Context Attention ( 9 ) and a Modality Abstractor ( 47 ) for effective cross-modal feature fusion. Finally, a compact and efficient LLaMA- X decoder generates clinically precise reports, avoiding the template rigidity of prompt-based models ( 16 ) while outperforming more resource-intensive alternatives in adaptability and computational efficiency. Related Work Fusion of Heterogeneous Medical Data Fusing heterogeneous medical data, such as EHR, clinical notes, and various medical imaging types ( 48 ; 30 ) , has shown significant potential for improving clinical tasks like prognosis prediction ( 18 ; 6 ) , phenotyping ( 11 ) , and medical image segmentation ( 13 ) . This integration of diverse data sources is a clear trend aimed at building more comprehensive and accurate clinical models ( 12 ) . Handling Missing Modality In practice, some clinical data modalities are inevitably missing ( 12 ) . A common solution is late fusion, where predictions from independently modeled modalities are aggregated at the decision level ( 57 ; 43 ) . However, this approach can be suboptimal as it fails to capture the interactions between modalities ( 12 ) . More recent research has explored generative methods to impute or reconstruct missing data at the feature or instance level ( 26 ; 59 ; 38 ) . These techniques may use a Bayesian meta-learning framework ( 26 ) or impute features in the latent space with auxiliary information ( 59 ) . Despite these advances, results from generated data may not be robust ( 19 ; 55 ) , and handling missing data in highly heterogeneous settings like image-and-text fusion remains an open challenge ( 55 ) . Disentangled Representation Learning A promising approach for handling both missing data and modal inconsistency is to disentangle shared and modality-specific information ( 55 ; 22 ; 35 ) . The goal is to learn representations that separate common, patient-related information from unique, modality-specific details ( 35 ) . This is often achieved by imposing explicit constraints on the latent space. Common techniques include enforcing orthogonality between shared and specific representations to minimize redundancy ( 3 ; 55 ) or minimizing their mutual information, often via an adversarial objective ( 37 ; 22 ; 35 ) . Concurrently, the alignment of shared representations is enforced using methods like Jensen-Shannon divergence (JSD) ( 55 ) or contrastive objectives ( 35 ) . While most prior work focused on more homogeneous modalities like different MRI scans ( 7 ; 39 ) , DiA introduces a probabilistic tri‑factor decomposition that leverages a Vision–Language VAE with a shared‑gate Mixture‑of‑Experts and a unified Disentangled‑Alignment constraint, enabling robust radiology reporting from highly heterogeneous inputs with missing modalities. Figure 1: Architecture of DiA: Extracts vision features using EfficientNetB 0 with Guided Context Attention and language features via a Transformer Encoder , fused

Read the complete text