Multi-Plane Vision Transformer for Hemorrhage Classification Using Axial and Sagittal MRI Data
Badhan Kumar Das, Gengyan Zhao, Boris Mailhe, Thomas J. Re, Dorin Comaniciu, Eli Gibson, Andreas Maier · 2025
Read Multi-Plane Vision Transformer for Hemorrhage Classification Using Axial and Sagittal MRI Data by Badhan Kumar Das, Gengyan Zhao, Boris Mailhe, Thomas J. Re, Dorin Comaniciu, Eli Gibson, Andreas Maier (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-07349.txt.
Introduction
Badhan Kumar Das and co-authors present a transformer model that classifies brain bleeding from MRI scans taken in two planes, in a 2025 paper. Scans vary in contrast and orientation, which makes the task hard for neural networks. Resampling images to one fixed plane can lose information.
Their model uses two encoders, one for top-view scans and one for side-view scans, and joins them with cross-attention. A marker vector tells the model when a contrast is missing. Tests on a real clinical data set support the design. The work aims to help clinicians spot bleeding reliably despite varied scan setups.
Catalog record
| CREATOR | Badhan Kumar Das, Gengyan Zhao, Boris Mailhe, Thomas J. Re, Dorin Comaniciu, Eli Gibson, Andreas Maier |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-07349
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Multi-Plane Vision Transformer for Hemorrhage Classification Using Axial and Sagittal MRI Data Badhan Kumar Das, Gengyan Zhao, Boris Mailhe, Thomas J. Re, Dorin Comaniciu, Eli Gibson, Andreas Maier Abstract Identifying brain hemorrhages from magnetic resonance imaging (MRI) is a critical task for healthcare professionals. The diverse nature of MRI acquisitions with varying contrasts and orientation introduce complexity in identifying hemorrhage using neural networks. For acquisitions with varying orientations, traditional methods often involve resampling images to a fixed plane, which can lead to information loss. To address this, we propose a 3D multi-plane vision transformer (MP-ViT) for hemorrhage classification with varying orientation data. It employs two separate transformer encoders for axial and sagittal contrasts, using cross-attention to integrate information across orientations. MP-ViT also includes a modality indication vector to provide missing contrast information to the model. The effectiveness of the proposed model is demonstrated with extensive experiments on real world clinical dataset consists of 10,084 training, 1,289 validation and 1,496 test subjects. MP-ViT achieved substantial improvement in area under the curve (AUC), outperforming the vision transformer (ViT) by 5.5% and CNN-based architectures by 1.8%. These results highlight the potential of MP-ViT in improving performance for hemorrhage detection when different orientation contrasts are needed. keywords Vision Transformer, Hemorrhage classification, Cross attention Introduction Intracranial hemorrhage (ICH) is a critical medical condition which, on suspicion, requires rapid and accurate detection and characterization. The long-time reference standard technique for its detection is computed tomography (CT). Magnetic resonance imaging (MRI) has been shown to be at least as sensitive to CT in detecting ICH while offering more detailed characterization of this finding, particularly of its chronicity (acute vs subacute vs chronic nature), while avoiding patient exposure to ionizing radiation [ 1 ] . Furthermore, MRI offers superior detection of other critical intracranial findings, in particular, ischemia [ 2 ] . Still, MRI has not replaced CT for most imaging of patients with suspected ICH. This is due, in part to issues of availability and longer scan times of MRI over CT, an issue being actively addressed through new technologies. A further issue impeding MRI’s adaptation in ICH diagnosis is the higher complexity involved with interpreting MRI results over those of CT. The application of machine learning and deep learning methods can be used to significantly reduce the complexity of such interpretation while enhancing accuracy and efficiency in this context. In the rapidly evolving field of deep learning and computer vision, the application of deep learning techniques for medical image processing have made substantial strides, revolutionizing tasks such as image classification [ 3 , 4 ] , object detection [ 5 , 6 ] , and segmentation [ 7 , 8 ] . Recently, the successes of vision transformers [ 9 ] on natural images, have encouraged researchers to query further the application of self-attention in medical imaging in order to capture long-range dependencies between pixels. Additionally, transformers have achieved comparable performance to state-of-the-art convolutional neural networks (CNNs) on medical image classification [ 10 , 11 ] , detection [ 12 , 13 ] , segmentation [ 14 , 15 , 16 ] and reconstruction [ 17 , 18 ] . In clinical practice, MRI scans are often performed with relatively thick slices to speed up the imaging, leading to anisotropic images with fewer slices that may not be sufficient to detect abnormalities. To enhance the diagnostic accuracy, clinicians frequently use multiple planes (axial, sagittal, and coronal) to get a comprehensive view of anatomical structures. Different planes reveal various aspects of anatomical structures, and some abnormalities or hemorrhages might be more visible in a particular plane due to their orientation. However, in both CNNs and vision transformer models, the input images are needed to be processed to a specific plane due to architectural constraints. Consistent alignment allows the neural network to effectively learn spatial relationships and integrate information from different channels, which is essential for accurate analysis of volumetric data. Despite the widespread adoption of using CNN and ViT for medical image classification tasks, concerns linger about potential information loss or distortion, especially when resampling MRI images from one orientation to another. For anisotropic images this change in orientation may need extensive down sampling/up sampling which can lead to significant information loss. Deep learning models have proven effective for classifying hemorrhages, such as in CT brain hemorrhage [ 19 ] and intestinal hemorrhage [ 20 ] . Recently, transformer-based architecture has been used by Adel et al. [ 21 ] for intracranial hemorrhage classification and detection. Sharma et al [ 22 ] , used vision transformer with residual in self-attention for brain tumor classification using MR images. ScopeFormer [ 23 ] used a hybrid CNN-Transformer architecture for intracranial hemorrhage detection. In another study, multi-stage transformer fusion is used for efficient hemorrhage classification [ 24 ] . However, all these studies do not address how to handle differences in image orientation. Existing methods require resizing the anisotropic input images from different orientations to the same resolution, which may introduce unnecessary resolution reduction and inaccurate interpolation. Another issue with real-world clinical datasets is that they often have inconsistent contrast across cases. This motivates the exploration of innovative methodologies to alleviate the necessity for changing orientations and optimizing the performance and robustness of deep learning models. Building on the successes of vision transformers in medical imaging, we extend this paradigm to propose a novel approach to hemorrhage classification using the multi-plane vision transformer (MP-ViT) architecture with axial and sagittal brain MRI data. Traditionally, medical imaging analyses have focused on individual planes, limiting a comprehensive understanding. Our approach consolidates information from multiple imaging perspectives, providing a more comprehensive representation of hemorrhagic regions and enhancing the model’s discriminative power. We also introduce a modality indication vector inspired by dynamic filter ideas from ModDrop++ [ 25 ] to address the challenge of missing contrasts in the input data. This vector serves as a guide for the Transformer model, enabling it to better handle variations in imaging modalities and effectively compensate for the absence of certain contrasts. By incorporating this modality indication vector, our approach ensures that the model maintains its performance and accuracy even when faced with incomplete or heterogeneous input data, ultimately leading to more robust and reliable hemorrhage classification results. Methods Dataset and Pre-processing This retrospective study is compliant with the Health Insurance Portability and Accountability Act (HIPAA). The paper describes scientific research using retrospectively acquired anonymized data. The anonymization was performed in accordance with applicable laws and regulations before secure transfer was made to Siemens Healthineers for the study. Use of the data followed all the applicable license terms. The study does not involve any clinical or human subject research component to it. As such, IRB approval and informed consent are not applicable because there are no data privacy issues and no patients were impacted by the research (did not affect treatment or diagnosis). The research was conducted conforming to the appropriate scientific practices and in accordance with the relevant guidelines and regulations of the institution conducting the experiments. Our dataset consists of 10,084 training, 1289 validation, and 1496 test subjects of 3D MRI data. The dataset was acquired from scanners produced by three manufacturers (GE, Philips, and Siemens Healthineers).