Lost in Distortion: Uncovering the Domain Gap Between Computer Vision and Brain Imaging - A Study on Pretraining for Age Prediction
Yanteng Zhang, Songheng Li, Zeyu Shen, Qizhen Lan, Lipei Zhang, Yang Liu, Vince Calhoun · 2025
Read Lost in Distortion: Uncovering the Domain Gap Between Computer Vision and Brain Imaging - A Study on Pretraining for Age Prediction by Yanteng Zhang, Songheng Li, Zeyu Shen, Qizhen Lan, Lipei Zhang, Yang Liu, Vince Calhoun (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-01310.txt.
Introduction
Yanteng Zhang and co-authors ask if models trained on photos help with brain scans. Big brain scan sets could train strong base models. But brain scans vary a lot in quality. Some are crisp, while others are warped or cut off.
Photo data from daily life is far more clean and even. That gap can hurt a model moved from photos to brains. The team studies the gap by training models for age guesses from scans. Their work shows what breaks in the move. It also points to steps that make base models for brain data more sound.
Catalog record
| CREATOR | Yanteng Zhang, Songheng Li, Zeyu Shen, Qizhen Lan, Lipei Zhang, Yang Liu, Vince Calhoun |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 16 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-01310
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Lost in Distortion: Uncovering the Domain Gap Between Computer Vision and Brain Imaging - A Study on Pretraining for Age Prediction Yanteng Zhang, Songheng Li, Zeyu Shen, Qizhen Lan, Lipei Zhang, Yang Liu, Vince Calhoun Abstract Large-scale brain imaging datasets provide unprecedented opportunities for developing domain foundation models through pretraining. However, unlike natural image datasets in computer vision, these neuroimaging data often exhibit high heterogeneity in quality, ranging from well-structured scans to severely distorted or incomplete brain volumes. This raises a fundamental question: can noise or low-quality scans contribute meaningfully to pretraining, or do they instead hinder model learning? In this study, we systematically explore the role of data quality level in pretraining and its impact on downstream tasks. Specifically, we perform pretraining on datasets with different quality levels and perform fine-tuning for brain age prediction on external cohorts. Our results show significant performance differences across quality levels, revealing both opportunities and limitations. We further discuss the gap between computer vision practices and clinical neuroimaging standards, emphasizing the necessity of domain-aware curation to ensure trusted and generalizable domain-specific foundation models. † † address: 1 Center for Translational Research in Neuroimaging and Data Science (GSU, GATech, Emory), USA; 2 College of Artificial Intelligence, Chengdu University of Information Technology, Chengdu, China; 3 Department of Applied Mathematics and Statistics, Johns Hopkins University, Baltimore, USA; 4 McWilliams School of Biomedical Informatics, University of Texas Health Center at Houston, USA; 5 Department of Applied Mathematics and Theoretical Physics, University of Cambridge, UK; 6 School of Biomedical Engineering and Imaging Sciences, King’s College London, UK 1 Introduction The advent of foundation models has profoundly reshaped machine learning, with large-scale pretraining driving major breakthroughs in natural image understanding. Frameworks such as MAE [ 5 ] , DINO [ 12 ] , and SimCLR [ 1 ] have demonstrated that general purpose visual representations can be learned from massive unlabeled datasets, leading to strong transferability and data efficiency in a variety of downstream tasks [ 13 ] . Inspired by this paradigm, the medical imaging community has begun to explore self-supervised pretraining in modalities such as MRI and CT to reduce dependence on annotated data and enhance model generalization for applications including disease diagnosis, brain age prediction, and structural abnormality detection [ 3 ] [ 8 ] . However, directly transferring such strategies from natural image analysis to neuroimaging presents unique challenges. Unlike natural image datasets that are typically clean and semantically diverse, neuroimaging repositories often contain scans with artifacts, intensity distortions, or even missing brain regions caused by acquisition variability across scanners, sites, and protocols. This inherent heterogeneity raises a critical question: does incorporating low-quality scans enrich pretraining by increasing data diversity, or does it simply introduce noise that undermines downstream generalization? Addressing this question is essential for understanding whether scaling up heterogeneous neuroimaging data truly benefits self-supervised learning or compromises representation consistency. In addition, brain imaging exhibits considerable complexity due to differences in imaging mechanisms, anatomical constraints, and multi-center acquisition settings [ 4 ] [ 2 ] . The non-standardization and heterogeneity introduced by various scanners and preprocessing pipelines make it challenging to build stable and transferable self-supervised representations. Figure 1: Overview of the two-stage pretraining and fine-tuning procedure for brain age prediction. To systematically investigate this issue, this study employs the large-scale FOMO [ 10 ] brain MRI dataset as a representative example to examine how data characteristics influence self-supervised pretraining and downstream performance. Specifically, we perform self-supervised pretraining using a 3D Masked Autoencoder (MAE). The pretrained encoder is subsequently transferred and fine-tuned on our in-house labeled dataset for brain age prediction. The main contributions of this work including: (1) Systematic investigation of data quality factors contributing to brain MRI self-supervised pretraining performance. (2) Empirical validation through the brain age prediction task. (3) Analysis and visionary outlook on the evolving paradigms of brain imaging. 2 Methodology Our task follows a two-stage framework, as illustrated in Fig.1. (1) A self-supervised pretraining stage based on a 3D Masked Autoencoder (MAE) that learns anatomical representations from unlabeled brain MRI scans; (2) A downstream fine-tuning stage for subject-level brain age prediction. This framework is designed to systematically evaluate how pretraining data quality affects downstream task performance. Figure 2: Several representative brain MRI samples intentionally selected to reflect quality differences across the three pretraining datasets used in this study; each sample is labeled with its original data index for reference and review. 2.1 MAE Pretraining We adopt a 3D MAE architecture composed of a convolutional encoder and decoder, which effectively captures local anatomical continuity while maintaining computational efficiency for volumetric sMRI data. The encoder consists of sequential 3D convolutional blocks with channel dimensions of 16, 16, 32, 32, 64, 64, 128, and 128. Each block includes a 3×3×3 convolution, followed by 3D batch normalization and ReLU activation. During pretraining, a voxel-level random masking strategy is applied, where 50% of voxels in each MRI scan are randomly masked, and only the remaining visible voxels are provided to the decoder for reconstruction. The decoder takes the latent features generated by the encoder and reconstructs the entire MRI volume, including masked regions, through a symmetric 3D deconvolutional network. It contains four transposed convolutional layers with progressively reduced channel sizes (128→64→32→16→1). Each layer is followed by 3D batch normalization and ReLU activation, except for the final layer, which uses a sigmoid function to normalize voxel intensities to the range [0,1] . Finally, the reconstructed output is upsampled using trilinear interpolation to restore the original spatial resolution, allowing recovery of fine-grained anatomical details. The reconstruction objective is defined as the mean squared error (MSE) between the reconstructed and the input images. 2.2 Fine-tuning for Brain Age Prediction After self-supervised pretraining, the encoder weights are transferred to a downstream CNN regression model for brain age prediction. The fine-tuning model retains the same CNN backbone as the pretrained encoder to ensure fair comparison. An attention block is introduced after the CNN encoder to highlight related brain regions [ 9 ] , followed by a fully connected (FC) regression layer that produces the predicted age. The network is optimized using MSE loss between the predicted and ground-truth ages. Table 1: Summary of downstream brain age prediction results. To clarify, the upper section (rows labeled as Raw and Filtered) corresponds to models pretrained leveraging the FOMO dataset, while the lower section presents the models pretrained leveraging the ADNI dataset. Model denotes the fine-tuning network (CNN is ACNN without the attention block); Data source specifies the pretraining dataset (‘–’ indicates no pretraining stage); Quantity represents the number of samples in the pretraining dataset; Image size is the resolution of MRI scans after resizing operation in pretraining, due to lots of raw brain scans in FOMO have one or more spatial dimensions with a resolution smaller than 20 voxels, it is infeasible to perform 3D CNN pretraining or using center-cropped (CC) operation; Frozen specifies whether encoder parameters were frozen during fine-tuning. In each section, the top two results are highlighted in bold to provide a clearer overall comparison. Model Data source Quantity Image size Frozen Dataset I Dataset II CC ↑ RMSE ↓ MAE(yrs) ↓ CC ↑ RMSE ↓ MAE(yrs) ↓ ACNN Raw 8017 121×145×121 \sqrt{} 0.467 6.824 4.847 0.621 5.551