Raptor: Scalable Train-Free Embeddings for 3D Medical Volumes Leveraging Pretrained 2D Foundation Models

Ulzee An, Moonseong Jeong, Simon A. Lee, Aditya Gorla, Yuzhe Yang, Sriram Sankararaman · 2025

Read Raptor: Scalable Train-Free Embeddings for 3D Medical Volumes Leveraging Pretrained 2D Foundation Models by Ulzee An, Moonseong Jeong, Simon A. Lee, Aditya Gorla, Yuzhe Yang, Sriram Sankararaman (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-08254.txt.

Introduction

Ulzee An and five co-authors present in 2025 Raptor, a training-free method that turns 3D medical volumes into rich embeddings using a frozen 2D foundation model. Training volume models from scratch is costly and data-hungry. Flat image models, by contrast, are plentiful and strong.

Slices of each volume pass through the frozen model to yield visual tokens. Random projections then compress those tokens while keeping their meaning. Ten diverse medical volume tasks grade the result. Raptor beats rivals that were trained on medical volumes alone, by margins from three to fourteen percent. No training cost is incurred at all. The authors pitch slice-based reuse as a fast foundation for volume AI in clinics.

Catalog record

CREATORUlzee An, Moonseong Jeong, Simon A. Lee, Aditya Gorla, Yuzhe Yang, Sriram Sankararaman
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-08254

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Raptor: Scalable Train-Free Embeddings for 3D Medical Volumes Leveraging Pretrained 2D Foundation Models Ulzee An, Moonseong Jeong, Simon A. Lee, Aditya Gorla, Yuzhe Yang, Sriram Sankararaman Abstract Current challenges in developing foundational models for volumetric imaging data, such as magnetic resonance imaging (MRI), stem from the computational complexity of training state-of-the-art architectures in high dimensions and curating sufficiently large datasets of volumes. To address these challenges, we introduce Raptor ( Ra ndom P lanar T ens o r R eduction), a train-free method for generating semantically rich embeddings for volumetric data. Raptor leverages a frozen 2D foundation model, pretrained on natural images, to extract visual tokens from individual cross-sections of medical volumes. These tokens are then spatially compressed using random projections, significantly reducing computational complexity while retaining semantic information. Extensive experiments on ten diverse medical volume tasks verify the superior performance of Raptor over state-of-the-art methods, including those pretrained exclusively on medical volumes ( + 3 % +3\% SuPreM, + 6 % +6\% MISFM, + 10 % +10\% Merlin, + 13 % +13\% VoCo, and + 14 % +14\% SLIViT), while entirely bypassing the need for costly training. Our results highlight the effectiveness and versatility of Raptor as a foundation for advancing deep learning-based methods for medical volumes (code: github.com/sriramlab/raptor ). † † affiliationnotice: Equal contribution Figure 1: Comparisons between Raptor and state-of-the-art methods on diverse medical volume tasks. Raptor achieves superior performance across classification (AUROC \uparrow ) and regression (mean r^{2} \uparrow , indicated with + ) tasks while remaining entirely train-free. 1 Introduction Understanding volumetric data is essential in a variety of applications, including healthcare ( Milletari et al. 2016 ) , human-computer interaction ( Jiang et al. 2023 ) , and autonomous systems ( Huang et al. 2025 ) . However, learning from volumetric modalities, represented as 3-dimensional voxels, meshes or point clouds, continues to pose logistical challenges compared with 2-D images. For instance, adapting convolutional ( Yang et al. 2021 ) or transformer architectures designed for images ( Dosovitskiy et al. 2021 ) to volumes remains computationally challenging as both classes of architectures incur cubic or higher order costs to learn or infer with volumes. Overcoming these hurdles generally requires the development of novel training objectives, efficient architectures, or access to large-scale compute infrastructure (though such infrastructure remains out of reach in most research settings). Furthermore, significant challenges remain in acquiring large-scale 3D training datasets relevant to the task of interest: the largest 3D medical datasets (160K volumes, Wu et al. 2024b ) remain several orders of magnitude smaller than the 2D image datasets (1.2B images, Oquab et al. 2023 ) that have allowed researchers to scale large image models for current state-of-the-art 2D vision applications. As a result, few works to date have experimentally demonstrated the benefits of scaling large models for medical volumes ( Zhu et al. 2023 ; Ma et al. 2024 ; Wu et al. 2024a ) , while large models for natural images and their capabilities are well-established ( Radford et al. 2021 ; Caron et al. 2021 ; Zhou et al. 2022 ; Saharia et al. 2022 ; Rombach et al. 2021 ; Ranftl et al. 2022 ; Kirillov et al. 2023 ; Liu et al. 2024a ; Oquab et al. 2023 ) . In this work we propose extracting semantically rich, low-dimensional embeddings from large 3D volumes by leveraging foundation models trained on generic 2D images without requiring training on the volumes . The straightforward approach involving a direct application of an image foundation model (DINOv2-L, Oquab et al. 2023 ) to infer tokens over orthogonal cross-sections of a volume, however, leads to embeddings that are substantially larger than the volumes. To this end, we introduce Raptor : Ra ndom P lanar T ens o r R eduction, a train-free technique that drastically compresses the dimensionality of features over volumes by using random projections . Random projection is a class of scalable stochastic dimensionality reduction methods that map high-dimensional data into a lower-dimensional space by multiplying with a random matrix, with well-studied theoretical/empirical guarantees such as approximate preservation of pairwise distances and low-rank signals ( Xie et al. 2017 ) . We use random projections to compress tokens inferred by a frozen image foundation model over orthogonal cross-sections of a volume and then reduce major spatial dimensions to significantly compress the final representation. Our method reliably yields embeddings that have \sim 99\% smaller footprint than that of raw voxels ( 256^{3} , compressed). Despite its substantial size reduction, we show that Raptor embeddings retain volumetric information by demonstrating higher accuracies on downstream tasks than current state-of-the-art (SOTA) models, including those that leverage extensive pretraining (Figure 1 ). We demonstrate these capabilities without fine-tuning the underlying vision model or fitting any additional 3D architecture (i.e., train-free ), making Raptor a particularly appealing choice in data-scarce and compute-constrained settings that are common in biomedical applications. Our work enables researchers with limited computational resources to utilize powerful pretrained models for high-dimensional volumetric analyses, thus broadening access to deep-learning-based 3D methods and fostering more inclusive innovation in machine learning. Our contributions can be summarized as follows: • Raptor is highly data-efficient – it is the first method that achieves state-of-the-art performance without requiring access to large datasets of medical volumes. • Raptor embeddings are train-free – we eliminate the need for costly 3-dimensional model training, significantly reducing computational overhead. • Raptor embeddings are scalable - they are sub-cubic with respect to input volume size, ensuring efficiency even for high-resolution medical volumes. • Raptor is model-agnostic – it is designed to seamlessly integrate with and benefit from future advancements in image foundation models. 2 Related Works Table 1: Current state-of-the-art models that leverage pretraining with medical volumes. We state the scale of each method’s medical volume pretraining data, its target medical domain, and its backbone architecture ( ∗ we note that Raptor is agnostic to any downstream domain, medical or not). Methods Medical Pretraining Data Domain Architecture SLIViT ( Avram et al. 2024 ) 14M images (ImageNet) + 108K OCT images Optical 2D Conv + 3D Transformer SuPreM ( Li et al. 2024 ) 5K CT volumes General 3D SwinUNETR, 3D ResNet Merlin ( Blankemeier et al. 2024 ) 15K volumes Chest 3D ResNet MISFM ( Wang et al. 2023 ) 110K CT volumes General 3D Conv + 3D Transformer VoCo ( Wu et al. 2024a ) 160K CT volumes General 3D SwinUNETR Raptor (Ours) None (uses 2D foundation model) General ∗ 2D ViT + 3D Compression Table 2: Sizes of comparable medical volume models. We compare the parameter count of each medical volume model and the size of their latents. The latent was determined as the smallest bottleneck layer without pooling. Methods Param. count Latent size SLIViT 48.4M 768\times 64\times 8\times 8 SuPreM 5.1M 128\times 12\times 12\times 12 Merlin 124.7M 2048\times 14\times 7\times 7 MISFM 46.2M 100\times 16\times 16\times 16 VoCo 294.9M 3072\times 3\times 3\times 3 Raptor (Ours) 304.4M (DINOv2-L) 3\times 100\times 16\times 16 ( K=100 ) Volumetric data span modalities such as point clouds ( Caesar et al. 2020 ) , meshes ( Chang et al. 2015 ) , and voxels ( Yang et al. 2023 ) . Each modality presents unique

Read the complete text