Self Pre-training with Adaptive Mask Autoencoders for Variable-Contrast 3D Medical Imaging

Badhan Kumar Das Gengyan Zhao Han Liu Thomas J. Re Dorin Comaniciu Eli Gibson Andreas Maier · 2025

Read Self Pre-training with Adaptive Mask Autoencoders for Variable-Contrast 3D Medical Imaging by Badhan Kumar Das Gengyan Zhao Han Liu Thomas J. Re Dorin Comaniciu Eli Gibson Andreas Maier (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-09096.txt.

Introduction

Badhan Kumar Das and co-authors pre-train a medical vision model that copes with however many scans a patient actually has in a 2025 paper. Masked autoencoders learn by hiding parts of an image and predicting them, a trick that teaches rich context. Real brain studies break the trick's assumptions: one patient brings two scan contrasts, the next brings five.

Their adaptive design takes a variable number of three-dimensional contrasts per person. Pre-training drank in scans from over forty-five thousand subjects; a smaller labeled set then fine-tuned the model for stroke lesion segmentation. Self pre-training lifted segmentation scores by a few points across vision transformer backbones. The authors frame flexibility as the point: hospital archives are messy, and a model that demands uniform inputs will sit unused.

Catalog record

CREATORBadhan Kumar Das Gengyan Zhao Han Liu Thomas J. Re Dorin Comaniciu Eli Gibson Andreas Maier
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE16 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-09096

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Self Pre-training with Adaptive Mask Autoencoders for Variable-Contrast 3D Medical Imaging Badhan Kumar Das Gengyan Zhao Han Liu Thomas J. Re Dorin Comaniciu Eli Gibson Andreas Maier Abstract The Masked Autoencoder (MAE) has recently demonstrated effectiveness in pre-training Vision Transformers (ViT) for analyzing natural images. By reconstructing complete images from partially masked inputs, the ViT encoder gathers contextual information to predict the missing regions. This capability to aggregate context is especially important in medical imaging, where anatomical structures are functionally and mechanically linked to surrounding regions. However, current methods do not consider variations in the number of input images, which is typically the case in real-world Magnetic Resonance (MR) studies. To address this limitation, we propose a 3D Adaptive Masked Autoencoders (AMAE) architecture that accommodates a variable number of 3D input contrasts per subject. A magnetic resonance imaging (MRI) dataset of 45,364 subjects was used for pretraining and a subset of 1648 training, 193 validation and 215 test subjects were used for finetuning. The performance demonstrates that self pre-training of this adaptive masked autoencoders can enhance the infarct segmentation performance by 2.8%-3.7% for ViT-based segmentation models. † † address: ‡ Siemens Healthineers AG † Siemens Medical Solutions USA, Inc. § FAU Erlangen-Nuremberg ©20XX IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. 1 Introduction In recent years, significant advancements have been made in applying deep learning methodologies to medical imaging tasks such as classification [ 1 , 2 ] , segmentation [ 3 , 4 ] , and object detection [ 5 , 6 ] . With the introduction of the Vision Transformer [ 7 ] and its success in computer vision tasks, researchers are now exploring different methodologies and applications in medical image analysis by leveraging transformers [ 8 , 9 , 10 ] . In UNETR [ 10 ] , the authors used a ViT encoder and a UNet [ 11 ] shaped decoder for medical image segmentation. Similarly, with the introduction of hierarchical Swin transformers [ 12 ] , researchers propsoed SwinUNETR [ 13 ] for segmentation tasks. Zhou, Lei, et al [ 14 ] applied ViT-based masked autoencoders for self pre-training on medical image classification and segmentation tasks. A self-supervised pre-training approach can be used to enhance the downstream performance of ViT based architectures. However, medical imaging presents unique challenges due to the complexity and variability of the data. Real world medical images often come in the form of multiple 3D scans, with varying number of contrasts for each acquisition, making it difficult to develop models that can handle this heterogeneity. Current models typically require the number of input images to be fixed. These approaches fall short of meeting the practical needs of medical diagnosis, where the number and types of input images can vary significantly from acquisition to acquisition depending on the diagnostic task at hand and the protocols of different clinical sites. To address these challenges, this work introduces a new flexible framework for medical imaging that enables the following: • The ability to handle a variable number of input images during both training and testing, improving the model’s flexibility and adaptability to real world clinical data with varying contrasts. • Self pre-training with variable number of 3D input contrasts and maximize data availability. This framework aims to bridge the gap between current models and the complex demands of real-world medical diagnostics. 2 Methodology Figure 1: Adaptive MAE self-supervised pretraining architecture which can handle N number of modalities. 2.1 Data and Pre-processing For pretraining we used 45,374 MR studies. For each case, a variable number of MRI contrasts acquired were used, including a set of seven contrasts: Apparent Diffusion Coefficient (ADC), Trace-weighted , T2-weighted, Gradient Echo (GRE), Susceptibility-Weighted , T1-weighted, and Fluid-Attenuated Inversion Recovery (FLAIR), or a subset of these when available. The images were acquired on scanners from Siemens Healthineers AG and, GE Medical Systems. For finetuning, We used a subset with 1648 training, 193 validation and 215 test subjects where acute/subacute brain infarct regions were manually segmented. The manual segmentation of acute and subacute infarct lesions was performed on Axial Trace-weighted (AxTrace) contrast image series by a radiologist (T.J.R., 10 years of experience). The radiologist used the medical image segmentation software ITK-SNAP version 3.8.0. The AxTrace image series and corresponding ADC image map were loaded into the software and reviewed by the radiologist. Areas, within the brain parenchyma, of hyperintensity in the TraceW image series with hypo or iso-intensity in the ADC map were considered positive for recent (acute to subacute) infarct by the radiologist and delineated as such in an image mask using the software tool. The inclusion criteria were studies where at least AxTrace and AxADC contrasts are present and where the radiologist identified at least one infarct lesion by aforementioned method. Additionally, Axial T2-weighted contrast is used if it is available. 2.2 Adaptive Masked Autoencoders The proposed adaptive framework employs a masked autoencoder approach for self-supervised learning (SSL) as shown in Figure 1 . The process begins with random masking of a percentage of 3D patches in each modality’s original 3D volume. Inspired by the Dynamic Filter Network used in ModDrop++ [ 15 ] , we propose a 3D Dynamic Convolution Tokenizer (DCT), which is then used to encode unmasked 3D patches into tokens. The DCT is designed to dynamically adapt to different input modalities. Each modality is assigned a unique modality vector \mathbf{m}_{i} of length l , where i represents the i -th modality: \mathbf{m}_{i}\in\mathbb{R}^{l} The modality vector \mathbf{m}_{i} is passed through a learnable linear projector and then split to generate a weight vector \mathbf{w}_{conv} and a bias vector \mathbf{b}_{conv} . The weight vector \mathbf{w}_{conv} and bias vector \mathbf{b}_{conv} are used to update the weights and biases of the dynamic convolutional layer. Let \mathbf{W} and \mathbf{B} denote the original weights and biases of the convolutional layer. The updated weights \mathbf{W}_{updated} and biases \mathbf{B}_{updated} are: \mathbf{W}_{updated}=\mathbf{W}\cdot\mathbf{w}_{conv} \mathbf{B}_{updated}=\mathbf{B}\cdot\mathbf{b}_{conv} This dynamic convolutional layer allows the tokenizer to extract modality-specific features from each 3D image. It also creates non overlapping tokens for the transformer by using patch size as kernel and stride size of the convolution and output dimension of the convolution as embedding dimension of the transformer. Let \mathbf{{X}_{i}}\in\mathbb{R}^{H\times W\times D} represent one 3D input image (or image tensor), where H,W,D are the height, width, and depth of the 3D image, respectively. The convolution operation with the updated weights and biases is: \mathbf{{Y}_{i}}=\text{Conv}(\mathbf{X},\mathbf{W}_{updated},\mathbf{B}_{updated}) where \mathbf{{Y}_{i}}\in\mathbb{R}^{\left(\frac{H\times W\times D}{\text{Patchsize}^{3}}\right)\times\text{EmbeddingDim}} is the output tokens produced by the dynamic convolutional layer for one input image. The unmasked tokens from all modalities are then concatenated into a long sequence and then fed into a transformer encoder. Each token receives a positional embedding (either sinusoidal or learnable) based on its 3D patch’s location in the 3D volume as transformer is position-agnostic. Two tokens from different modalities but the same patch location will have the same positional embedding. These embeddings help preserve spatial relationships of the tokens. During pretrain, the

Read the complete text