nnActive: A Framework for Evaluation of Active Learning in 3D Biomedical Segmentation
Carsten T. Lüth, Jeremias Traub, Kim-Celine Kahl, Till Bungert, Lukas Klein, Lars Kraemer, Paul F. Jaeger, Fabian Isensee, Klaus Maier-Hein · 2025
Read nnActive: A Framework for Evaluation of Active Learning in 3D Biomedical Segmentation by Carsten T. Lüth, Jeremias Traub, Kim-Celine Kahl, Till Bungert, Lukas Klein, Lars Kraemer, Paul F. Jaeger, Fabian Isensee, Klaus Maier-Hein (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-19183.txt.
Introduction
Carsten T. Lüth and eight co-authors build nnActive, a framework for testing active learning in three-dimensional medical image segmentation. Labeling scans by hand is slow and costly, so active learning picks only the most useful cases to label. In 3D biomedical imaging, though, earlier studies disagreed on whether it truly beats random sampling.
The authors name four flaws in past evaluations, from too few data sets to counting cost only in voxels. nnActive fixes them with a large study, partial annotations inside the nnU-Net tool, foreground-aware random baselines, and a foreground efficiency metric. The setup gives a fairer test of when active learning saves effort.
Catalog record
| CREATOR | Carsten T. Lüth, Jeremias Traub, Kim-Celine Kahl, Till Bungert, Lukas Klein, Lars Kraemer, Paul F. Jaeger, Fabian Isensee, Klaus Maier-Hein |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 133 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-19183
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
nnActive: A Framework for Evaluation of Active Learning in 3D Biomedical Segmentation Carsten T. Lüth, Jeremias Traub, Kim-Celine Kahl, Till Bungert, Lukas Klein, Lars Kraemer, Paul F. Jaeger, Fabian Isensee, Klaus Maier-Hein Abstract Semantic segmentation is crucial for various biomedical applications, yet its reliance on large annotated datasets presents a significant bottleneck due to the high cost and specialized expertise required for manual labeling. Active Learning (AL) aims to mitigate this challenge by selectively querying the most informative samples, thereby reducing annotation effort. However, in the domain of 3D biomedical imaging, there remains no consensus on whether AL consistently outperforms Random sampling strategies. Current methodological assessment is hindered by the wide-spread occurrence of four pitfalls with respect to AL method evaluation. These are (1) restriction to too few datasets and annotation budgets, (2) training 2D models on 3D images and not incorporating partial annotations, (3) Random baseline not being adapted to the task, and (4) measuring annotation cost only in voxels. In this work, we introduce nnActive, an open-source AL framework that systematically overcomes the aforementioned pitfalls by (1) means of a large scale study evaluating 8 Query Methods on four biomedical imaging datasets and three label regimes, accompanied by four large-scale ablation studies, (2) extending the state-of-the-art 3D medical segmentation method nnU-Net by using partial annotations for training with 3D patch-based query selection, (3) proposing Foreground Aware Random sampling strategies tackling the foreground-background class imbalance commonly encountered in 3D medical images and (4) propose the foreground efficiency metric, which captures that the annotation cost for background- compared to foreground-regions is very low. We reveal the following key findings: (A) while all AL methods outperform standard Random sampling, none reliably surpasses an improved Foreground Aware Random sampling; (B) the benefits of AL depend on task specific parameters like number of classes and their locations; (C) Predictive Entropy is overall the best performing AL method, but likely requires the most annotation effort; (D) AL performance can be improved with more compute intensive design choices like longer training and smaller query sizes. As a holistic, open-source framework, nnActive has the potential to act as a catalyst for research and application of AL in 3D biomedical imaging. Code is at: https://github.com/MIC-DKFZ/nnActive 1 Introduction Semantic segmentation is vital for numerous biomedical applications, including the delineation and detection of structures and pathologies in computed tomography scans (CT), magnetic resonance imaging (MRI), and microscopy images. With the advent of deep learning, training-based approaches that require large annotated datasets have become the de facto standard for semantic segmentation. This trend is particularly evident in 3D medical imaging, where U-Net-like architectures like nnU-Net ( 19 ) have received significant emphasis. While there is an increasing number of medical scans created each day and stored in large databases ( 37 ) , the cost of data annotation remains one of the main obstacles when solving specific 3D biomedical tasks, due to the high manual effort required to create segmentation masks. This is particularly drastic for biomedical images, as specialized personnel have to carry out the annotation. The promise of Active Learning (AL) is to reduce this cost of annotation by only querying the most informative samples to be annotated for the task. This reduction in annotation cost needs to be weighed against an increase in computational and setup costs stemming from the use of AL. Therefore, to justify a general recommendation for an AL method, it needs to reliably bring performance benefits over computationally cheap annotation strategies like Random sampling in multiple ’realistic scenarios’ that are substantial enough to ensure amortization of its cost increases during application ( 25 ; 30 ; 28 ) . While there are various studies on AL for 2D image and video segmentation ( 28 ; 27 ) , many open questions remain about its effectiveness in the 3D biomedical domain, largely due to fundamental differences between the 2D and 3D domains. Most importantly, the high annotation cost and the high redundancy in 3D image data, e.g. the similarity of neighboring areas, necessitate more efficient annotation strategies such as partial annotations, in the form of slices or patches. In addition, 3D biomedical images commonly have a background class that occupies most of the image, standing in stark contrast to the dense multiclass masks frequent in 2D natural image semantic segmentation tasks ( 8 ) . The AL community lacks a common benchmark for the 3D biomedical domain as it is highly scattered in terms of evaluation practices (see table 1 ), making it nearly impossible to directly compare results across papers. Most importantly, there is no consensus on whether employing AL methods leads to reliable performance improvements over Random sampling. Many studies indicate that AL methods do not always outperform Random sampling ( 31 ; 11 ; 12 ; 10 ; 38 ; 7 ) and also commonly emphasize that it remains a surprisingly strong baseline ( 31 ; 7 ) . Further, 7 concluded that AL methods do not reliably outperform advanced Random sampling strategies where the sampling is adapted to the 3D structure of the data. Crucially, this is the only work investigating improved Random baselines. When taking further into account that most studies neither make use of standardized segmentation models, proven to bring state-of-the-art performance, nor 3D models making explicit use of partial annotations during training. With both of these points potentially substantially reducing overall model performance, it is apparent that the current evaluation protocol does not allow for making practically relevant and generalizing assessments based on which a practitioner can make an informed decision whether to employ AL or not. In response, this work introduces a novel framework for performance evaluation of 3D biomedical AL for semantic segmentation. It systematically addresses shortcomings in prior work, formalized as four pitfalls, by employing best practices for general AL evaluation. These practices are extended to account for the specific properties of 3D biomedical segmentation, enabling potential practitioners and developers to better assess the expected performance improvements when utilizing AL in a close-to-production scenario. Concretely, our contributions are: 1. We provide nnActive, a highly configurable AL extension for nnU-Net using partial annotations in the form of 3D patches that ensure state-of-the-art segmentation performance and out-of-the-box adaptation to new segmentation tasks. 2. We introduce Foreground Aware Random sampling as a stronger, more realistic baseline, which tackles the class imbalance typically encountered in 3D images. 3. We perform the largest study to date of AL methods with a specific focus on uncertainty based Query Methods (QMs), encompassing over 7500 nnU-Net trainings on 12 dataset-settings from four different datasets with three respective Label Regimes for each dataset, alongside numerous ablations to allow a holistic view of AL performance benefits. 4. We propose Foreground Efficiency (FG-Eff), a novel metric measuring annotation efficiency which takes into account that annotating background has a negligible annotation effort compared to foreground, setting it apart from other metrics using voxels as a proxy for annotation effort. 2 Requirements of Active Learning Evaluation In its very foundation, AL represents a wager where an expected reduction in annotation cost is weighed against the additional experimental setup and compute costs induced by it. The expected annotation cost reduction can be estimated from the annotation cost and