The Missing Piece: A Case for Pre-Training in 3D Medical Object Detection
Katharina Eckstein, Constantin Ulrich, Michael Baumgartner, Jessica Kächele, Dimitrios Bounias, Tassilo Wald, Ralf Floca, Klaus H. Maier-Hein · 2025
Read The Missing Piece: A Case for Pre-Training in 3D Medical Object Detection by Katharina Eckstein, Constantin Ulrich, Michael Baumgartner, Jessica Kächele, Dimitrios Bounias, Tassilo Wald, Ralf Floca, Klaus H. Maier-Hein (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-15947.txt.
Introduction
Katharina Eckstein leads a large team that tests pre-training for finding objects in 3D medical scans. Spotting tumors and organs in volumes is a key part of computer aided diagnosis. Pre-training has helped image cutting a lot, yet its use for 3D detection stayed underexplored. Past attempts leaned on flat scans or photos of the natural world.
Theirs is the first systematic study of how pre-training fits top detection designs, both classic and transformer based. Pre-training lifts detection scores across tasks and data sets in a steady way. Self taught pre-training based on rebuilding inputs beats supervised pre-training. Contrast based pre-training gives no clear gain here. Code is public for reuse.
Catalog record
| CREATOR | Katharina Eckstein, Constantin Ulrich, Michael Baumgartner, Jessica Kächele, Dimitrios Bounias, Tassilo Wald, Ralf Floca, Klaus H. Maier-Hein |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-15947
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Missing Piece: A Case for Pre-Training in 3D Medical Object Detection Katharina Eckstein, Constantin Ulrich, Michael Baumgartner, Jessica Kächele, Dimitrios Bounias, Tassilo Wald, Ralf Floca, Klaus H. Maier-Hein Abstract Large-scale pre-training holds the promise to advance 3D medical object detection, a crucial component of accurate computer-aided diagnosis. Yet, it remains underexplored compared to segmentation, where pre-training has already demonstrated significant benefits. Existing pre-training approaches for 3D object detection rely on 2D medical data or natural image pre-training, failing to fully leverage 3D volumetric information. In this work, we present the first systematic study of how existing pre-training methods can be integrated into state-of-the-art detection architectures, covering both CNNs and Transformers. Our results show that pre-training consistently improves detection performance across various tasks and datasets. Notably, reconstruction-based self-supervised pre-training outperforms supervised pre-training, while contrastive pre-training provides no clear benefit for 3D medical object detection. Our code is publicly available at: https://github.com/MIC-DKFZ/nnDetection-finetuning . 1 1 footnotetext: Equal contribution; co-first author order may be adjusted for individual use. 2 2 footnotetext: Work done while at DKFZ, now at Siemens Healthineers. 1 Introduction Accurate detection of anatomical structures and abnormalities in 3D medical imaging is crucial for reliable diagnosis and clinical decision-making. Unlike segmentation, which provides detailed structural delineation, detection focuses on localizing clinically relevant objects. Critically, detection excels in clinically relevant metrics, especially in high-stakes scenarios where completely missing an object can have far more severe consequences than minor inaccuracies in pixel-wise delineation [ 73 ] . Despite its clinical importance, research on 3D object detection has received significantly less attention than segmentation, as evidenced by medical image analysis challenges predominantly emphasizing segmentation tasks [ 72 ] . Recent advancements in 3D medical image segmentation have spurred interest in large-scale pre-training. For instance, Ulrich et al. introduced Multitalent [ 98 ] , a framework that enables supervised training across multiple segmentation datasets. Moreover, self-supervised pre-training strategies [ 100 , 111 , 103 , 97 , 30 ] have demonstrated promising results for segmentation applications. Likewise, detection models might particularly benefit from pre-training due to the typically small size of annotated datasets and their tendency to over focus on local image features, rather than leveraging broader contextual information. However, despite the advancements in pre-training for segmentation, the impact of purely 3D large-scale pre-training remains unexplored for 3D object detection. This gap was also acknowledged in one of the most recent and comprehensive studies on 3D medical object detection by Baumgartner et al., who extensively revised the nnDetection framework [ 11 , 10 ] . While their work made significant contributions to the field, it did not address the potential role of large-scale pre-training. Yet, beyond their work, research on pre-training strategies for medical object detection is virtually nonexistent, with only a handful of studies even touching upon this direction. Existing pre-training strategies for medical object detection have predominantly focused on 2D data, utilizing either natural image pre-training [ 104 ] or 2D medical data [ 61 , 21 , 79 , 12 , 62 ] . This is largely due to the scarcity of publicly available 3D object detection datasets with sufficient cases for effective pre-training. To partially capture 3D context, some methods extend pre-trained 2D models by integrating adjacent slices. This includes using ImageNet-pretrained backbones with 3D context slices added at the downstream stage [ 101 , 106 ] or pseudo-3D approaches that treat image channels (e.g., RGB) as separate slices during pre-training [ 109 ] . Another strategy relies on video-based pre-training, where adjacent frames are used to simulate the sequential nature of medical slices [ 4 ] . Notably, no prior study has systematically explored large-scale 3D pre-training for 3D medical object detection. To bridge this gap, we present a comprehensive study evaluating the impact of different large-scale pre-training strategies on 3D medical object detection. Specifically, our key contributions include: 1. The First Comprehensive Study on Pre-Training Paradigms for 3D Object Detection to analyze the impact of both supervised and self-supervised large-scale pre-training for 3D medical object detection across eight diverse downstream detection datasets. 2. Evaluation Across Detection Architectures: Model performance varies widely depending on dataset characteristics and annotation types, such as bounding boxes or segmentation masks, as demonstrated by Baumgartner et al. [ 10 ] . To assess the generalization of pre-training strategies, we examine their transferability to two state-of-the-art detection models, Retina U-Net [ 36 , 10 ] and Deformable DETR [ 112 , 10 ] , covering both CNN and Transformer. 3. Comparison of Pre-Training Architectures for Pre-training: ResEncL, a state-of-the-art model for semantic segmentation [ 35 ] that has shown improved downstream performance with self-supervised pre-training [ 100 ] , and an adapted version of Retina U-Net, allowing both segmentation pre-training as well as downstream 3D object detection fine-tuning. 2 Methods In this study, we evaluate the impact of large-scale pre-training on 3D medical object detection using two state-of-the-art architectures: Retina U-Net and Deformable DETR. Notably, both architectures are specifically designed for detection and cannot be directly applied to other tasks without modifications. Therefore, for pre-training, we adapt Retina U-Net for supervised segmentation and employ the state-of-the-art ResEncL model for both supervised and self-supervised learning [ 100 , 35 ] . We then transfer only the pre-trained backbone from these models to the detection networks for downstream fine-tuning, as visualized in fig. 1 . Our experimental setup involves five development datasets and three independent testing datasets. The development datasets are employed to systematically investigate various fine-tuning strategies, enabling us to identify optimal approaches for adapting pre-trained models to 3D medical object detection. Figure 1: A cross-framework bridge between nnDetection and its pre-training counterparts : Different pre-training paradigms (supervised and self-supervised), pre-training architectures (Retina U-Net and ResEncL), and detection-specific models (Retina U-Net and Deformable DETR) can be combined like puzzle pieces, offering a flexible and integrative approach to optimizing detection performance. 2.1 3D Object Detection Retina U-Net [ 36 ] is a single-stage, anchor-based object detector enhanced with semantic segmentation supervision. Its architecture extends the Feature Pyramid Network (FPN) of RetinaNet with additional high-resolution levels in the FPN’s top-down pathway to support an auxiliary segmentation task, creating a U-Net-like symmetric structure (U-FPN), as visualized in fig. 1 . The detection head, applied to the final four or five resolution levels, consists of a classification and a regression branch. The regression branch uses smooth L1 loss, while the classification branch employs binary focal loss. Segmentation is supervised with a combined cross-entropy and batch Dice loss function. Deformable DETR [ 112 ] is a two-stage transformer-based detection architecture. In contrast to traditional DEtection TRansformer (DETR), Deformable DETR replaces global self-attention with a sparse deformable attention mechanism, significantly reducing computational complexity and enhancing efficiency by focusing on a small set of queries per attention operation. Additionally, Deformable DETR introduces iterative bounding box refinement, progressively updating the bounding boxes instead of predicting them from scratch. As visualized in fig. 1 , Deformable DETR contains an encoder network as a first component to extract a feature representation from the input image. A point-wise convolution is applied to this feature representation to reduce the channel dimensionality. The extracted feature maps are flattened into a sequence of spatial tokens, with positional encodings added to