An Open Multi-Center Whole-Body FDG PET/CT Foundation Model for Tumor Segmentation

Xiaofeng Liu, Qianru Zhang, Thibault Marin, Menghua Xia, Chi Liu · 2026

Read An Open Multi-Center Whole-Body FDG PET/CT Foundation Model for Tumor Segmentation by Xiaofeng Liu, Qianru Zhang, Thibault Marin, Menghua Xia, Chi Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-21835.txt.

Introduction

Xiaofeng Liu and coauthors present an open-source, multi-center, whole-body FDG PET/CT foundation model for tumor segmentation. Methods stay narrow, train on single-center cohorts, or fuse branches late, delaying PET-CT mixing. It uses hierarchical UNet models with early channel-wise concatenation, letting anatomical and metabolic data mix from the first layer. A masked autoencoding with zero-mean imputation avoids false intensity jumps at mask edges. With ten percent of labeled data, the models match full-dataset scratch models on AutoPET lesion segmentation.

Catalog record

CREATORXiaofeng Liu, Qianru Zhang, Thibault Marin, Menghua Xia, Chi Liu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33222
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.21835

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

An Open Multi-Center Whole-Body FDG PET/CT Foundation Model for Tumor Segmentation Thanks: Xiaofeng Liu, Qianru Zhang, Thibault Marin, Menghua Xia, Chi Liu, Georges El Fakhri, and Jinsong Ouyang are with the Department of Radiology and Biomedical Imaging, Yale Biomedical Imaging Institute, Yale University, New Haven, CT, USA. Thanks: Corresponding author: Jinsong Ouyang (email: [email protected]). PubID: pubid: 0000–0000/00$00.00 © 2021 IEEE Xiaofeng Liu, Qianru Zhang, Thibault Marin, Menghua Xia, Chi Liu Abstract The synergistic interpretation of anatomical information from computed tomography (CT) and metabolic information from positron emission tomography (PET) is important to oncologic imaging. However, existing deep learning methods for PET/CT remain largely task-specific, are often trained on single-center cohorts, or adopt dual-branch fusion schemes that delay cross-modal interaction and underutilize early spatial correspondence between PET and CT. To address these limitations, we present an open-source, multi-center, whole-body FDG PET/CT foundation model utilizing 4,997 harmonized scans from four public datasets. Our framework employs hierarchical UNet-shaped backbones with early channel-wise concatenation, enabling anatomical and metabolic features to interact from the first embedding layer onward. We further introduce a masked autoencoding objective based on zero-mean imputation, combined with a weighted global reconstruction loss. This design avoids non-physical intensity discontinuities at masked-region boundaries that arise from learnable mask tokens. On downstream AutoPET lesion segmentation, the proposed models demonstrate strong label efficiency: with only 10% of the labeled training data, they achieve performance comparable to models trained from scratch on the full dataset. Under extreme 5-shot linear probing, joint PET/CT pretraining also achieves higher Dice scores than separated-modality pretraining. This multi-center foundation model demonstrates label efficiency and cross-modality representation learning for PET/CT tumor segmentation. It provides a robust, open-source basis for advancing automated oncologic imaging, significantly reducing the need for large-scale manual annotations in clinical practice. I Introduction The complementary use of positron emission tomography (PET) and computed tomography (CT) has become a mainstay of oncologic imaging because it jointly characterizes anatomy and metabolism, enabling lesion detection, staging, and treatment monitoring [ 6 , 23 ] . In addition to visual interpretation, accurate tumor delineation is essential for quantifying disease burden, supporting staging, guiding treatment planning, and enabling longitudinal assessment of therapeutic response. In routine clinical practice, however, PET/CT interpretation remains labor-intensive and error-prone. Radiologists often need to review hundreds of slices per study while distinguishing malignant uptake from normal physiologic FDG activity and other confounding patterns. This process is further complicated by substantial inter-reader and intra-reader variability, as lesion identification and boundary delineation can vary across radiologists and even for the same reader at different times [ 15 ] . Recent advances in deep learning have substantially improved automated PET/CT analysis, particularly for lesion segmentation and detection [ 26 , 4 ] . Despite this progress, most existing methods have limited clinical generalizability. A large fraction of prior work is designed for a single downstream task and trained on relatively small, single-center cohorts [ 19 ] . Such approaches depend heavily on scarce lesion-level annotations and often exhibit reduced robustness across institutions, scanner vendors, acquisition protocols, and patient populations [ 24 , 16 , 2 ] . Moreover, models trained with a relatively small number of studies may experience substantial performance degradation when applied to out-of-distribution cases. Foundation models trained on large-scale and diverse datasets, often comprising tens of thousands of medical imaging volumes, offer significant advantages over conventional task-specific models trained on limited cohorts of hundreds or fewer studies. By learning generalizable representations without being optimized for a single downstream task, such models can substantially reduce the amount of labeled data required for a given application. In addition, they tend to be less sensitive to domain shifts arising from inter-institutional variations, scanner vendors, and acquisition protocols, and are more robust when applied to out-of-distribution cases. From an optimization perspective, foundation models often enable faster and more stable training during downstream fine-tuning, and provide a flexible framework for modular domain adaptation across different clinical tasks and datasets. Early efforts focused mainly on 2D modalities such as chest X-rays [ 11 ] , whereas more recent work has shifted toward 3D volumetric data. Tak et al. [ 22 ] introduced BrainIAC, a UNet-shaped brain MRI foundation model, demonstrating that large-scale masked pre-training can substantially improve both full fine-tuning and few-shot linear probing. For dense voxel-wise downstream tasks, masked reconstruction objectives such as masked autoencoder (MAE) [ 9 ] are often better aligned than contrastive approaches such as SimCLR [ 3 ] , because reconstruction encourages retention of fine spatial detail that contrastive invariance may suppress. This paradigm has recently been extended to PET/CT. Oh et al. [ 19 , 20 ] proposed FratMAE, a PET/CT foundation model based on separate PET and CT encoders. While this approach leverages prior advances in single-modality representation learning, it treats PET and CT as isolated inputs and delays cross-modal interaction. In addition, its decoder reconstructs feature tokens rather than voxel intensities, limiting compatibility with dense voxel-level tasks such as lesion segmentation. The absence of hierarchical skip connections further constrains the model’s ability to recover high-resolution details critical for detecting small lesions. Another major challenge in PET/CT representation learning is the scale and diversity of available data. Public PET/CT datasets remain small compared with natural-image datasets [ 7 ] , and prior PET/CT pre-training has largely relied on single-center cohorts [ 19 ] . This lack of diversity limits robustness to domain shifts arising from differences in scanners, acquisition protocols, and patient populations. To address this issue, we aggregate multiple public FDG PET/CT datasets and resampled them into a unified physical voxel space (Table I ), mitigating cross-dataset heterogeneity while preserving anatomically and metabolically meaningful image characteristics. Masked autoencoders [ 9 ] have become a standard approach for visual pre-training, but their conventional implementation is not fully compatible with hierarchical encoder-decoder architectures widely used in medical imaging. In ViT-based MAE, masked patches are replaced with learnable tokens [ 19 ] , which can introduce artificial discontinuities. In UNet-like architectures, such discontinuities may destabilize convolutional processing in skip connections and degrade high-frequency structural information. This issue is particularly critical for PET/CT, where accurate delineation of small lesions depends on preserving fine spatial details. To address these limitations, we introduce a multi-center whole-body FDG PET/CT foundation model pre-trained on 4,797 harmonized scans from four public datasets, with an additional 200 scans for leave-out evaluation. Instead of using separate modality-specific encoders, we adopt early channel-wise concatenation within hierarchical UNet-shaped backbones, enabling joint learning of anatomical and metabolic features from the first embedding layer onward. We further design a masked autoencoding objective tailored to encoder-decoder architectures by replacing learnable mask tokens with dual-channel randomized zero-mean imputation and optimizing a weighted global reconstruction loss. This design preserves stable skip-connection behavior while emphasizing masked-region recovery. TABLE I: Dataset scale, spatial extent, and demographic summary. {\dagger} Only the FDG-PET subset of DeepPSMA is used; PSMA-PET scans are excluded. * Eleven known anomalous SPADE samples containing only a single slice were excluded. Dataset Valid FDG PET/CT Scans Z Min Z Max Y Min Y Max X Min X Max Site(s) Scanner Vendors Female Age AutoPET IV [ 7 ] 1014 200 661 122 235 179 251 Germany SIEMENS 44.2% 59.7 \pm 15.9 DeepPSMA {\dagger} [ 13 ]

Read the complete text