VolTA-3D: Self-Supervised Learning for Brain MRI using 3D Volumetric Token Alignment

Amy Makawana 1 , Abhijeet Parida 2 , Marius George Linguraru 2,3, Julia Ive 1 , Syed Muhammad Anwar 2,3* · 2026

Read VolTA-3D: Self-Supervised Learning for Brain MRI using 3D Volumetric Token Alignment by Amy Makawana 1 , Abhijeet Parida 2 , Marius George Linguraru 2,3, Julia Ive 1 , Syed Muhammad Anwar 2,3* (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-16775.txt.

Introduction

Makawana and colleagues present VolTA-3D, a self-taught 3D Vision Transformer system for brain MRI that learns shared volume features. Most 3D brain models focus on one task, limiting transfer across datasets, protocols, and downstream uses. VolTA-3D jointly aligns global class-style tokens and local patch tokens in a student-teacher setup and adds fine-grained shape rebuild. On out-of-distribution tasks including memory-region mapping and sorting of sex and memory loss versus healthy controls, it beats fresh baselines and shows stronger transfer under domain shift.

Catalog record

CREATORAmy Makawana 1 , Abhijeet Parida 2 , Marius George Linguraru 2,3, Julia Ive 1 , Syed Muhammad Anwar 2,3*
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE27702
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.16775

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

VolTA-3D: Self-Supervised Learning for Brain MRI using 3D Volumetric Token Alignment Amy Makawana 1 , Abhijeet Parida 2 , Marius George Linguraru 2,3, Julia Ive 1 , Syed Muhammad Anwar 2,3* Abstract Self-supervised learning (SSL) has advanced medical image analysis by enabling representation learning from large unlabelled data. However, in brain magnetic resonance imaging (MRI), most 3D models remain specialised for either segmentation or classification, limiting their ability to generalise across datasets, imaging protocols, and downstream tasks. This lack of transferability constrains the clinical utility of 3D MRI models, despite the availability of unlabelled volumetric data. We present VolTA-3D , a self-supervised 3D Vision Transformer framework designed to learn transferable volumetric representations. VolTA-3D jointly aligns global class-style tokens and local patch tokens within a student–teacher paradigm and enforces fine-grained structural reconstruction. This combined global–local alignment addresses the limited semantic diversity and subtle anatomical variation characteristic of brain MRI, which challenge existing SSL approaches. We evaluate VolTA-3D on multiple out-of-distribution downstream tasks, including hippocampal segmentation and classification of sex and Alzheimer’s disease versus healthy controls. Across all tasks, representations learned by VolTA-3D outperform randomly initialised baselines, demonstrating improved transferability and robustness under domain shift. Hence jointly enforcing global semantic consistency and local structural learning during pretraining enables broader concept learning from unlabelled brain MRI data. Overall VolTA-3D supports effective multi-task downstream performance without task-specific pretraining, a step towards generalisable and clinically viable 3D models. 1 Institute of Health Informatics, University College London, London, UK 2 Sheikh Zayed Institute for Pediatric Surgical Innovation, Children’s National Hospital, Washington, DC USA 3 School of Medicine and Health Sciences, George Washington University, Washington, DC, USA * [email protected] I Introduction Deep learning is increasingly applied to brain magnetic resonance imaging (MRI) for disease classification and structural segmentation, where manual interpretation remains slow and variable [ 19 ] . These tasks are essential for observing anatomical changes to understand disease progression, planning surgical intervention, and delineating structures and regions, among other uses [ 7 ] . However, most existing brain MRI models are highly task-specific, limiting their ability to generalise across datasets, imaging protocols, and downstream tasks—an essential requirement for reliable clinical deployment [ 14 ] . A central factor limiting generalisability is the quality of learned representations. Brain anatomy and pathology are inherently 3D (three-dimensional), and models that fail to exploit volumetric structure risk learning acquisition-specific or slice-dependent features that do not transfer well across domains [ 10 ] . While 2D (two-dimensional) models are more widely adopted due to lower computational cost, they lose critical inter-slice context and global anatomical structure, which are more stable across scanners and protocols. As a result, 3D models consistently outperform 2D approaches in capturing anatomically meaningful features and improving cross-domain robustness, even in reduced data settings [ 1 ] . Despite their representational advantages, supervised training of 3D brain MRI models is constrained by the lack of high-quality and diverse annotated medical imaging data, and where more diverse data is available, model training architectures are not always built to leverage these effectively [ 23 ] . Manual annotation of volumetric MRI is time-consuming and expensive, resulting in a far greater availability of unlabelled MRI data than reliably annotated datasets [ 7 ] . 3D models additionally have the advantage of greater global context learning which can be further enhanced using vision transformers (ViTs) [ 18 ] . ViT models excel in capturing global structure, outperforming the more traditionally used convolutional neural networks (CNNs), but they often require substantial data for training, and their application to brain MRI is limited [ 16 ] . Self-supervised learning (SSL) offers a promising solution to the lack of labelled medical data by enabling representation learning from the more abundant unlabelled MRI data collections [ 14 ] . However, SSL for brain MRI remains challenging due to the limited semantic diversity of brain anatomy and the subtle nature of clinically relevant variations [ 4 ] . This motivates development of novel 3D SSL training strategies that can learn transferable volumetric representations from heterogeneous datasets, where scanner and protocol variation hinder generalisation to out of domain samples. We propose the novel Vit-based VolTA-3D (volumetric token alignment for 3D imaging) pretraining methodology to leverage diverse concept learning from 3D MRI sourced from various centres and scanners. Utilising both global and local token alignment across varying views of an MRI, alongside Gaussian-mixture masked-modelling (GMML), we show enhanced performance compared to randomly initialised baselines across both segmentation and classification tasks. Our results demonstrate the potential of this methodology to adapt to diverse downstream tasks, paving the way for larger-scale pretraining. Our main contributions are: • We introduce VolTA-3D , the first 3D extension of the Diverse Concept Modelling (DiCoM) framework [ 21 ] , enabling joint class-style token and patch-token alignment, with group masked model learning, for volumetric MRI. This integrates global semantic alignment with local structural reconstruction within a unified 3D ViT student–teacher paradigm. • We demonstrate that VolTA-3D learns transferable 3D MRI representations, outperforming randomly initialised 3D ViT, Swin Transformer, and CNN baselines on out-of-distribution segmentation and classification tasks, demonstrating robustness to domain shift; a key challenge in clinical deployment. • We further demonstrate VolTA-3D outperforms baselines when trained on reduced data subsets, indicating utility in limited-data settings. Fig. 1: VolTA-3D pretraining pipeline. A T1 MRI is geometrically augmented to produce two global crops with mild intensity for the teacher and stronger intensity with masking for the student—which also receives clean local crops. Both networks share the wrapped ViTAutoEnc encoder with a projection head, and the student additionally performs reconstruction. II Methods II-A Data and Preprocessing To pretrain VolTA-3D, we used 2,002 diverse 3D T1/T1n MRIs: 773 from the Alzheimer’s Disease Neuro-imaging Initiative (ADNI) dataset [ 13 ] spanning cognitively normal, significant memory concern, early mild cognitive impairment, late mild cognitive impairment, and Alzheimer’s disease (AD) patient groups, and 1,229 from the Brain Tumor Segmentation (BraTS) dataset [ 6 ] from adult glioma, adult metastasis and peadiatric glioma. For downstream tasks, HarP-labelled ADNI scans (135 MRIs) [ 2 ] were used for hippocampus segmentation (all patients excluded from ADNI pretraining). Sex classification used 619 healthy T1 MRIs (342 male and 277 female) from IXI 1 1 1 https://brain-development.org/ixi-dataset/ , and AD classification used 420 T1 MRIs (347 healthy and 73 AD) from the AIBL dataset [ 8 ] . To match BraTS preprocessing [ 17 ] , ADNI 2D DICOMs (Digital Imaging and Communications in Medicine files) were converted to NIfTI (Neuroimaging Informatics Technology Initiative files), re-oriented to right-anterior-superior orientation, resampled to 1mm 3 isotropic resolution. All MRI were resized/padded to 144\times 192\times 192 with intensities scaled to [0,1] . II-B VolTA-3D Framework and pretraining In general, SSL aims to extract information from data without ground truth labels to minimise human bias. We extend the SSL DiCoM [6] framework into 3D for VolTA-3D, incorporating: 1) GMML, 2) local and global alignment, and 3) knowledge distillation. The high-level VoLTA-3D pipeline is presented in Fig. 1 . VolTA-3D is built around MONAI’s ViTAutoEnc, a 3D Vision Transformer autoencoder designed for volumetric inputs. The model partitions each MRI volume into 3D patches, embeds them as a sequence of tokens, and processes these tokens with Transformer encoder blocks to capture

Read the complete text