Stochastic Siamese MAE Pretraining for Longitudinal Medical Images

Taha Emre, Arunava Chakravarty, Thomas Pinetz, Dmitrii Lachinov, Martin J. Menten, Hendrik Scholl, Sobha Sivaprasad, Daniel Rueckert, Andrew Lotery, Stefan Sacu, Ursula Schmidt-Erfurth, Hrvoje Bogunović, for the PINNACLE Consortium, for the Alzheimer’s Disease Neuroimaging Initiative (ADNI) · 2025

Read Stochastic Siamese MAE Pretraining for Longitudinal Medical Images by Taha Emre, Arunava Chakravarty, Thomas Pinetz, Dmitrii Lachinov, Martin J. Menten, Hendrik Scholl, Sobha Sivaprasad, Daniel Rueckert, Andrew Lotery, Stefan Sacu, Ursula Schmidt-Erfurth, Hrvoje Bogunović, for the PINNACLE Consortium, for the Alzheimer’s Disease Neuroimaging Initiative (ADNI) (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-23441.txt.

Introduction

Taha Emre and a twelve-author team present STAMP, a pretraining method that teaches image models how diseases evolve over time. Medical datasets follow patients across visits, and capturing progression in 3D scans is crucial, yet the strongest self-supervised method, masked autoencoding, treats each scan as timeless. Temporal extensions compare scans as fixed, ignoring that disease evolution is uncertain.

STAMP wraps masked autoencoding in a Siamese design. Two volumes from different visits are encoded together, conditioned on the time gap between them. The loss becomes a conditional variational objective, so the model learns a distribution over futures, not one fixed outcome. Pretrained this way, vision transformers beat temporal masked autoencoders and foundation models at predicting progression of late-stage age-related macular degeneration and Alzheimer's disease. Tests span two retinal OCT datasets and one MRI dataset with visits per patient.

Catalog record

CREATORTaha Emre, Arunava Chakravarty, Thomas Pinetz, Dmitrii Lachinov, Martin J. Menten, Hendrik Scholl, Sobha Sivaprasad, Daniel Rueckert, Andrew Lotery, Stefan Sacu, Ursula Schmidt-Erfurth, Hrvoje Bogunović, for the PINNACLE Consortium, for the Alzheimer’s Disease Neuroimaging Initiative (ADNI)
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-23441

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Stochastic Siamese MAE Pretraining for Longitudinal Medical Images Taha Emre, Arunava Chakravarty, Thomas Pinetz, Dmitrii Lachinov, Martin J. Menten, Hendrik Scholl, Sobha Sivaprasad, Daniel Rueckert, Andrew Lotery, Stefan Sacu, Ursula Schmidt-Erfurth, Hrvoje Bogunović, for the PINNACLE Consortium, for the Alzheimer’s Disease Neuroimaging Initiative (ADNI) Abstract Temporally aware image representations are crucial for capturing disease progression in 3D volumes of longitudinal medical datasets. However, recent state‑of‑the‑art self‑supervised learning approaches like Masked Autoencoding (MAE), despite their strong representation learning capabilities, lack temporal awareness. In this paper, we propose STAMP ( S tochastic T emporal A utoencoder with M asked P retraining), a Siamese MAE framework that encodes temporal information through a stochastic process by conditioning on the time difference between the 2 input volumes. Unlike deterministic Siamese approaches, which compare scans from different time points but fail to account for the inherent uncertainty in disease evolution, STAMP learns temporal dynamics stochastically by reframing the MAE reconstruction loss as a conditional variational inference objective. We evaluated STAMP on two OCT and one MRI datasets with multiple visits per patient. STAMP pretrained ViT models outperformed both existing temporal MAE methods and foundation models on different late stage Age‑Related Macular Degeneration and Alzheimer’s Disease progression prediction which require models to learn the underlying non‑deterministic temporal dynamics of the diseases. I Introduction Longitudinal medical images are routinely acquired in hospitals across multiple visits for each patient to track degenerative disease progression and inform clinical decisions. In ophthalmology, Age-related macular degeneration (AMD), a major contributor to vision loss and blindness [ 7 ] , causes irreversible tissue damage with the severity steadily worsening over time, and is diagnosed using 3D optical coherence tomography (OCT) scans. Intermediate stage AMD (iAMD) may follow one of two pathways: neovascular AMD (wet-AMD) and Geographic Atrophy (GA) (Fig. 1 - 1st & 2nd rows), indicating multiple prognostic paths [ 25 ] . Moreover, progression to late stage wet-AMD can occur suddenly (Fig. 1 - 1st row), contributing to the uncertainty. Similarly, Alzheimer’s disease (AD) is an irreversible, progressive neurodegenerative disorder and the leading cause of dementia. With approximately 50 million people affected worldwide [ 6 ] , early and accurate detection is critical. While current treatments only address symptoms, volumetric brain MRI is widely used in clinical settings to identify early structural changes in patients with cognitively normal (CN) and mild cognitive impairment (MCI) diagnosis to make informed clinical decisions and understand the onset of dementia [ 8 ] (Fig. 1 - 3rd row). The ability to forecast the future trajectory of disease progression from the current patient scan is crucial for personalized treatment [ 4 ] . It can help prioritize patients at increased risk of converting to the advanced stage through timely treatment and frequent monitoring [ 53 , 1 ] . However, manual identification of high-risk patients is often subjective, tedious, and error-prone. Moreover, the high cost of diagnostic labeling in longitudinal data often leaves them in forecasting. This underscores the need to develop automated methods that can utilize unlabeled longitudinal medical datasets. Self-supervised learning (SSL) aims to leverage available unlabeled data to learn meaningful features about disease. In the longitudinal setting, SSL methods learn to model temporal relationships by comparing images from different time points without relying on any human supervision. Although most existing longitudinal SSL methods make use of multiple visits as input for future forecasting, they ultimately learn to extract deterministic temporal features. However, this is limiting because the rate of disease progression and the resulting anatomical changes vary significantly across patients. Therefore, it becomes crucial to model multiple possible future outcomes as a stochastic variable, allowing models to predict a distribution over a range of plausible disease progression trajectories. Leveraging large longitudinal medical datasets, we can model these temporal dynamics in an unsupervised manner so that future progression can be estimated from a single visit, reducing hospital load and enabling timely treatment planning. In this work, we propose STAMP (Stochastic Temporal Autoencoder with Masked Pretraining), a novel SSL approach that extends masked auto-encoding to longitudinal medical imaging data within a Siamese framework. We hypothesize that latent factors driving temporal evolution can be inferred without explicit supervision by contrasting a patient’s two visits via a variational process. To account for the inherent uncertainty arising from these latent factors, we sample temporal feature tokens from a learned prior probability distribution conditioned on the time interval and the current input scan. STAMP is therefore able to capture long-term non-deterministic temporal dependencies using only two visits. Moreover, our SSL does not require spatio-temporal attention to learn temporal dynamics [ 5 ] , making it model agnostic and compatible with standard Vision Transformer backbones. Our key contributions are 1 1 1 Source code is available at https://github.com/EmreTaha/STAMP : Fig. 1: Top : HARBOR dataset for wet-AMD conversion from iAMD, arrow pointing the subretinal fluid. Middle : PINNACLE dataset for GA conversion from iAMD, arrow pointing increasing light transmission due to atrophy. Bottom : ADNI dataset, CN to MCI to AD with difference map, red/green means decrease/increase in pixel intensity. • Time Awareness: We introduce a learnable encoding of time difference between input images, improving temporal image features over longer intervals. • Time‑Conditioned Stochasticity: We introduce a stochastic component into future image representations by sampling from a learned time-conditioned probability distribution, taking the uncertainty of the future into account. • 3D Volumetric Extension: We employ a fully 3D approach to learn representations of volumetric scans at each time point, marking the first adaptation of the Siamese MAE framework to 3D medical scans. • Extensive Evaluation: We evaluate STAMP in two different modalities (OCT and MRI) on various prognostic tasks with different predictive time windows. II Background And Related work Self-supervised learning is a pretraining approach that extracts meaningful representations from data without manual labels, thereby improving downstream performance when annotations are scarce. In medical imaging, SSL has been used for various downstream tasks, such as segmentation, disease classification, or disease forecasting [ 28 ] . Longitudinal medical datasets offer a unique opportunity to learn long-term relations with SSL. Previously, various longitudinal SSL methods have been developed using contrastive learning (temporal sensitivity [ 20 ] , age as metadata [ 17 ] ), and pretext learning (time difference prediction [ 43 ] , patient-level trajectory modeling [ 42 ] ). Although these models achieved considerable results, they are limited by metadata availability and large batch size dependency of contrastive learning [ 11 ] , which makes scaling to longitudinal 3D volumes computationally inefficient. However, with the advent of Vision Transformer (ViT) architectures, Masked Auto-encoding (MAE) [ 26 ] has emerged as the state-of-the-art SSL method for pretraining on unlabeled data. It addresses ViTs’ lack of inductive biases such as spatial locality and translation equivariance by adapting an image inpainting task [ 36 ] to the patch‑based attention mechanism of ViTs. After dividing each input image into non‑overlapping patches ( tokens ), MAE randomly masks up to 90% of them, and only the visible patches are processed by a ViT encoder to extract strong high‑ and low‑level feature embeddings [ 32 ] . A lightweight ViT‑based decoder then reconstructs the masked patches from the visible patch embeddings combined with learnable mask tokens and

Read the complete text