Clinically-aligned Multi-modal Chest X-ray Classification

Phillip Sloan, Edwin Simpson, Majid Mirmehdi · 2025

Read Clinically-aligned Multi-modal Chest X-ray Classification by Phillip Sloan, Edwin Simpson, Majid Mirmehdi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-09581.txt.

Introduction

Phillip Sloan and two co-authors classify chest X-rays the way radiologists actually work. Several images are read together with the patient's medical context. Most automated systems read one image with no background, discarding structured information available at reporting time. Rising demand and radiologist shortages make better support tools valuable.

Their framework, CaMCheX, encodes frontal and lateral radiographs with view-specific ConvNeXt networks. A transformer fusion module merges image features with the medical indication, patient history, and vital signs. The resulting context-aware representations mirror medical reasoning more closely. The authors report results beyond the state of the art on the original MIMIC-CXR data set. They also lead on the newer CXR-LT benchmarks, supporting clinically grounded multimodal design.

Catalog record

CREATORPhillip Sloan, Edwin Simpson, Majid Mirmehdi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-09581

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Clinically-aligned Multi-modal Chest X-ray Classification Phillip Sloan, Edwin Simpson, Majid Mirmehdi Abstract Radiology is essential to modern healthcare, yet rising demand and staffing shortages continue to pose major challenges. Recent advances in artificial intelligence have the potential to support radiologists and help address these challenges. Given its widespread use and clinical importance, chest X-ray classification is well suited to augment radiologists workflows. However, most existing approaches rely solely on single-view, image-level inputs, ignoring the structured clinical information and multi-image studies available at the time of reporting. In this work, we introduce CaMCheX \xspace , a multimodal transformer-based framework that aligns multi-view chest X-ray studies with structured clinical data to better reflect how clinicians make diagnostic decisions. Our architecture employs view-specific ConvNeXt encoders for frontal and lateral chest radiographs, whose features are fused with clinical indications, history and vital signs using a transformer fusion module. This design enables the model to generate context-aware representations that mirror the reasoning in clinical practice. Our results exceed the state of the art for both the original MIMIC-CXR dataset and the more recent CXR-LT benchmarks, and highlight the value of clinically grounded multimodal alignment for advancing chest X-ray classification. † † volume: 297 † † year: 2025 † † workshop: Machine Learning for Health (ML4H) 2025 Data and Code Availability The primary dataset used in this study was MIMIC-CXR ( 15 ) with its CXR-LT 2023/2024 extensions ( 9 ; 24 ) . For pretraining, we incorporated additional public datasets, including CheXpert ( 11 ) , ChestXray14 ( 49 ) , and VinDR ( 28 ) , while MIMIC-IV-ED ( 16 ) was leveraged to provide vital signs. Code is available here: https://github.com/phillipSloan/CaMCheX . Institutional Review Board (IRB) This work used public, deidentified imaging datasets with no direct human subject involvement; as such, IRB approval was not required. 1 Introduction Figure 1: Conceptual visualisation of CaMCheX \xspace ’s Approach – CaMCheX \xspace mirrors clinical decision-making by incorporating imaging data with complementary information such as clinical indications from radiology request forms and vital signs from A&E triage, enabling accurate and clinically relevant chest X-ray classification. Radiology plays a vital role in modern healthcare, functioning as a key diagnostic tool for identifying diseases, tracking treatment response, and informing patient outcomes. Among radiological procedures, the chest X-ray is the most frequently performed examination and is crucial for the early detection of serious conditions such as pneumonia, lung cancer, and congestive heart failure. However, the increasing gap between radiology workforce capacity and service demand is a growing concern, with nearly all (99%) clinical directors expressing worries about its impact on the safety and quality of care ( 35 ) . Fatigue-related errors are an additional concern, potentially compromising diagnostic accuracy ( 45 ) . In light of these challenges, there is an urgent need for innovation, and AI-based approaches to chest X-ray classification have been identified as a means to augment radiologists’ reporting ( 34 ) , with multimodal approaches highlighted as essential to maximise impact ( 2 ) . Medical image classification presents a distinct set of challenges that set it apart from conventional image classification tasks. For instance, chest X-ray studies often comprise multiple views, including frontal (posteroanterior or anteroposterior) and lateral (left or right) projections, each offering distinct but complementary diagnostic insights. Despite this inherent multi-view structure, the majority of classification methods, such as those proposed in 29 ; 14 ; 31 ; 47 ; 8 ; 48 , approach the task at single image level, treating each chest X-ray independently. While this has lead to measurable progress, it overlooks the clinical reality that radiologists interpret typically at the study level, where multiple views are reviewed together. Ignoring such anatomical and pathological context across views at study-level may result in missed findings and inaccurate diagnoses. Medical imaging tasks such as stroke outcome prediction 40 and stroke lesion segmentation 39 integrate spatial information across multiple planes or slices. These volumetric methods share conceptual parallels with the multi-view setting, as both combine multiple observations from the same patient. The distinction is that multi-view approaches operate on 2D chest X-rays rather than 3D volumes. Images may include frontal and lateral projections, which provide spatial context, or multiple images from the same perspective, each offering complementary diagnostic information. Another limitation in current chest X-ray classification methods is the underuse of complementary clinical information by most existing approaches ( 20 ; 29 ; 14 ) . Clinical indications are readily available at the time of reporting and have been shown to enhance diagnostic accuracy in radiological practice ( 5 ) , and vital signs provide essential physiological context ( 4 ; 33 ) , offering cues that can inform image interpretation. For example, an elevated temperature may prompt radiologists to focus more closely on features suggestive of pneumonia. Such structured data have been successfully incorporated into deep learning models in other medical domains to support outcome prediction ( 37 ; 44 ) . Although 13 incorporate clinical indications, their method remains limited to image-level classification, without study-level, multi-view reasoning or the inclusion of vital signs. To our knowledge, no existing chest X-ray classification approach jointly models both multi-view radiographs and structured clinical information, as found in datasets such as MIMIC-CXR ( 15 ) and challenge benchmarks like CXR-LT 2023 ( 9 ) and CXR-LT 2024 ( 24 ) . Other long-standing challenges, such as class imbalance and the multi-label nature of chest X-ray datasets, are well recognised in the field ( 20 ; 9 ) . These are commonly addressed using custom loss functions ( 20 ; 31 ) or model ensembling ( 29 ; 14 ; 10 ) . We adopt a weighted asymmetric loss function to account for these factors, however, our primary focus is on enhancing multi-view integration and aligning model design more closely with clinical reasoning. In this paper, we address these limitations by introducing our CaMCheX \xspace (Clinical-aligned Multi-modal Chest X-ray), a model that simultaneously integrates clinical indications and vital sign data with multi-view radiological images at the study level, advancing multimodal learning for chest X-ray classification. CaMCheX \xspace incorporates a multi-view image backbone with separate, view-specific ConvNeXt encoders that independently process frontal and lateral images, enabling each encoder to specialise in the distinct characteristics of its respective view and extract complementary features. These radiographic features are then integrated with structured clinical data through a transformer-based fusion module, facilitating context-informed fusion that improves the accuracy and robustness of classification. Figure 1 presents a conceptual overview of our approach. The design of our framework is grounded in feedback from five clinicians who confirmed they routinely review clinical indications and observations alongside available imaging. Our contributions can be summarised as follows: (i) We propose CaMCheX \xspace which integrates multi-view chest X-ray images with structured clinical data, including clinical indications and vital signs, in a manner that more closely reflects the clinical process followed by clinicians ( 2 ; 27 ) . (ii) The proposed method is capable of handling any number of frontal and lateral images using separate, view-specific encoders, and conditionally integrates clinical indications and vital signs when available, enabling robust performance across incomplete or heterogeneous inputs such as those in the MIMIC-CXR dataset. (iii) We perform

Read the complete text