BrainAnytime: Anatomy-Aware Cross-Modal Pretraining for Brain Image Analysis with Arbitrary Modality Availability

Guangqian Yang*, Tong Ding*, Wenlong Hou, Yue Xun, Ye Du, Qian Niu 🖂, Shujun Wang 🖂, for the Alzheimer’s Disease Neuroimaging Initiative · 2026

Read BrainAnytime: Anatomy-Aware Cross-Modal Pretraining for Brain Image Analysis with Arbitrary Modality Availability by Guangqian Yang*, Tong Ding*, Wenlong Hou, Yue Xun, Ye Du, Qian Niu 🖂, Shujun Wang 🖂, for the Alzheimer’s Disease Neuroimaging Initiative (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-13059.txt.

Introduction

Yang and six co-authors present BrainAnytime, a pretraining framework for brain image analysis that takes whatever scans are on hand, from a lone T1 scan to a full multimodal workup. Most models assume fixed scan types, yet patients arrive with mixed, often incomplete scan sets. Trained on 34,899 3D brain scans, it learns MRI-PET links via cross-modal distillation and spotlights disease-vulnerable anatomy via atlas-guided masking. Across four tasks it largely beats single-scan and missing-scan rivals.

Catalog record

CREATORGuangqian Yang*, Tong Ding*, Wenlong Hou, Yue Xun, Ye Du, Qian Niu 🖂, Shujun Wang 🖂, for the Alzheimer’s Disease Neuroimaging Initiative
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25956
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.13059

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

BrainAnytime: Anatomy-Aware Cross-Modal Pretraining for Brain Image Analysis with Arbitrary Modality Availability Guangqian Yang*, Tong Ding*, Wenlong Hou, Yue Xun, Ye Du, Qian Niu 🖂, Shujun Wang 🖂, for the Alzheimer’s Disease Neuroimaging Initiative Abstract Clinical diagnostic workups typically follow a modality escalation pathway: after initial clinical evaluation, clinicians begin with routine structural imaging ( e.g. , MRI), selectively add sequences such as FLAIR or T2 to refine the differential, and reserve molecular imaging ( e.g. , amyloid-PET) for cases that remain uncertain after standard evaluation. Consequently, patients are observed with heterogeneous and often incomplete modality subsets. However, most current AI models assume fixed data modalities as the model inputs. In this paper, we present BrainAnytime, a unified pretraining framework pretrained on 34,899 3D brain scans from five datasets that support brain image analysis under arbitrary modality availability spanning multi-sequence MRI and amyloid-PET. A single model accepts whatever imaging is available, from a lone T1 scan to a full multimodal workup. Pretraining learns structural-molecular correspondences between MRI and PET via cross-modal distillation (RCMD) and prioritizes disease-vulnerable anatomy via atlas-guided curriculum masking (PACM), all within a shared 3D masked autoencoder (Multi-MAE3D). Across four downstream tasks and five clinically motivated modality settings, BrainAnytime largely outperforms modality-specific models, missing-modality baselines, and large-scale brain MRI pretrained foundation models on most modality settings. Notably, it surpasses the strongest missing-modality baselines with relative improvements of 6.2% and 7.0% in average accuracy on CN vs. AD and CN vs. MCI classification, respectively. Code is available at https://github.com/SDH-Lab/BrainAnytime . 1 Introduction When a patient presents with suspected cognitive decline, the diagnostic workup typically begins with clinical history review and neuropsychological screening [ 13 ] . Once these initial evaluations warrant neuroimaging, clinicians rarely order every modality upfront. Instead, imaging follows a staged escalation of evidence: T1-weighted MRI is acquired first, additional MRI sequences ( e.g. , FLAIR and T2) are added as needed, and amyloid-PET, which measures \beta -amyloid plaque burden and serves as a core biomarker of AD pathology, is often reserved for cases where structural imaging alone is insufficient [ 3 ] . We refer to this sequential acquisition logic as the clinical modality escalation pathway . Consequently, modality availability is inherently incomplete and stage dependent: in major AD cohorts, only a small fraction of subjects have a complete set of all imaging modalities (Fig. 1 and Table 1 ), while most are observed through partial and heterogeneous modality subsets. These practice patterns motivate a different pretraining paradigm. Rather than training separate models for fixed input configurations, we seek a unified pretraining model that supports brain image analysis under arbitrary modality availability spanning multi-sequence MRI and amyloid-PET. Such a framework should satisfy three requirements. (i) Modality-flexible inference. A single model should accept any subset of available modalities at test time without architectural switching or retraining. (ii) Cross-modal structural-molecular correspondence. Pretraining should encourage shared representations that capture correspondences between structural MRI and amyloid-PET, instead of learning each modality in isolation [ 4 ] . (iii) Anatomy-aware learning. Because AD exhibits selective regional vulnerability [ 12 ] , pretraining should emphasize disease-salient anatomy rather than treating all locations uniformly. However, existing brain imaging pretraining paradigms do not meet these requirements in the AD setting. Recent large-scale self-supervised models pretrained on brain neuroimaging show strong transfer across downstream tasks [ 19 , 5 , 17 , 22 , 6 , 23 , 8 , 27 ] . Despite this progress, three limitations remain. First, most are MRI-only and therefore do not learn an explicit bridge to amyloid-PET, even though PET is a key source of pathological evidence along the escalation pathway. Second, pretraining and deployment commonly assume a fixed modality (or a fixed combination), mismatching real-world, stage-dependent missingness and hindering inference under arbitrary modality subsets. Third, standard MAE pretraining [ 11 ] relies on spatially uniform masking , providing no mechanism to prioritize anatomically vulnerable regions that are most informative for AD. These gaps motivate a cross-modal, modality-flexible, and anatomy-aware pretraining framework tailored to clinically realistic modality availability. We present BrainAnytime , an anatomy-aware cross-modal pretraining framework for brain image analysis with arbitrary modality availability . BrainAnytime is pretrained on 34,899 3D brain scans from five large-scale datasets spanning multi-sequence MRI and amyloid-PET, by randomly sampling modality subsets to mimic clinical missingness. BrainAnytime integrates three components: Multi-MAE3D , a 3D multi-modal masked autoencoder with a shared Transformer encoder for any modality subset; Reciprocal Cross-Modal Distillation (RCMD) , an EMA-teacher distillation objective that aligns MRI and PET representations; and Pathology-Aware Curriculum Masking (PACM) , an atlas-guided curriculum masking strategy that emphasizes AD-relevant neuroanatomy during reconstruction. Together, these designs yield a unified pretrained model that is cross-modal, robust to missing modalities, and explicitly anatomy-aware. 2 Method Figure 1: Overall framework of BrainAnytime. We propose Anatomy-Aware Cross-Modal Pretraining , a self-supervised framework for learning unified representations from heterogeneous 3D neuroimaging data with arbitrary missing modalities. As shown in Fig. 1 , our framework consists of three components: (i) a 3D multi-modal masked autoencoder ( Multi-MAE3D ) that encodes any subset of modalities with a single Transformer; (ii) Reciprocal Cross-Modal Distillation (RCMD) that learns MRI–PET correspondences via an EMA teacher; and (iii) Pathology-Aware Curriculum Masking (PACM) that progressively focuses reconstruction on disease-relevant brain regions. 2.1 Multi-MAE3D: 3D Multi-Modal Masked Autoencoder Given a set of 3D volumes \{\mathbf{x}_{m}\}_{m\in\mathcal{M}} where \mathcal{M}\subseteq\{\text{T1, T2, FLAIR, PET}\} denotes the available modalities, each volume \mathbf{x}_{m}\in\mathbb{R}^{D\times H\times W} is partitioned into N non-overlapping patches of size p^{3} . Per-modality tokenization. Each modality has a modality-specific patch embedding f_{m} with a 3D convolutional layer, that maps raw patches to d -dimensional tokens. Following PACM (Sec. 2.3 ), only a subset of visible tokens per modality is retained. Missing modalities are fully masked, and the visible-token budget is redistributed among observed modalities via \text{Dir}(\alpha) [ 8 ] under a global mask ratio r , keeping the total token count constant regardless of modality availability. Shared encoder. The visible tokens from all available modalities are concatenated with a learnable [CLS] token, augmented with 3D sinusoidal positional embeddings, and processed by a shared student encoder. An attention mask blocks tokens from missing modalities (set to -\infty ), ensuring the encoder gracefully handles any modality subset at both training and inference time. Per-modality decoders. Each modality has a lightweight decoder that takes the encoded visible tokens and learnable mask tokens as input. Cross-attention between mask tokens and encoder output is followed by self-attention blocks. The decoder reconstructs all N patches per modality; the training loss is computed only on masked patches using mean squared error with per-patch normalization [ 11 ] : \mathcal{L}_{\text{MAE}}=\frac{1}{|\mathcal{M}|}\sum_{m\in\mathcal{M}}\frac{1}{|\mathcal{S}_{m}|}\sum_{i\in\mathcal{S}_{m}}\left\|\hat{\mathbf{p}}_{i}^{m}-\bar{\mathbf{p}}_{i}^{m}\right\|_{2}^{2}, (1) where \mathcal{S}_{m} is the set of masked patch indices for modality m , \hat{\mathbf{p}}_{i}^{m} is the predicted patch, and \bar{\mathbf{p}}_{i}^{m} is the per-patch normalized ground truth. 2.2 Reciprocal Cross-Modal Distillation (RCMD) To explicitly learn MRI–PET correspondences, we introduce RCMD, a cross-modal prediction objective based on an exponential moving average (EMA) cross-modal teacher. The teacher shares the encoder architecture and is updated as \theta_{T}\leftarrow\mu\,\theta_{T}+(1{-}\mu)\,\theta_{S} , where the momentum \mu follows a cosine schedule from \mu_{0} to 1.0 . For paired samples containing both MRI and PET, we compute group-level representations by average-pooling the encoded tokens within two groups: \mathbf{z}_{\text{MRI}} (T1+T2+FLAIR tokens) and

Read the complete text