Prompting with the human-touch: evaluating model-sensitivity of foundation models for musculoskeletal CT segmentation
Caroline Magg, Maaike A. ter Wee, Johannes G.G. Dobbe, Geert J. Streekstra, Leendert Blankevoort, Clara I. Sánchez, Hoel Kervadec · 2026
Read Prompting with the human-touch: evaluating model-sensitivity of foundation models for musculoskeletal CT segmentation by Caroline Magg, Maaike A. ter Wee, Johannes G.G. Dobbe, Geert J. Streekstra, Leendert Blankevoort, Clara I. Sánchez, Hoel Kervadec (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-10541.txt.
Introduction
Caroline Magg and co-authors benchmark promptable foundation models for musculoskeletal CT segmentation, asking how sensitive they are to the human touch in prompts. Eleven models were tested with 2D and 3D prompting on private and public data, covering bone and implant tasks in four regions. The best 2D and 3D models differed, and scores varied widely. Human prompts from an observer study lowered performance versus ideal prompts drawn from labels, and every model reacted to prompt changes. Choosing a model for human-driven use, they conclude, stays hard.
Catalog record
| CREATOR | Caroline Magg, Maaike A. ter Wee, Johannes G.G. Dobbe, Geert J. Streekstra, Leendert Blankevoort, Clara I. Sánchez, Hoel Kervadec |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 106443 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.10541
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Prompting with the human-touch: evaluating model-sensitivity of foundation models for musculoskeletal CT segmentation Caroline Magg, Maaike A. ter Wee, Johannes G.G. Dobbe, Geert J. Streekstra, Leendert Blankevoort, Clara I. Sánchez, Hoel Kervadec Abstract Promptable Foundation Models (FMs), initially introduced for natural image segmentation, have also revolutionized medical image segmentation. The increasing number of models, along with evaluations varying in datasets, metrics, and compared models, makes direct performance comparison between models difficult and complicates the selection of the most suitable model for specific clinical tasks. In our study, 11 promptable FMs are tested using non-iterative 2D and 3D prompting strategies on a private and public dataset focusing on bone and implant segmentation in four anatomical regions (wrist, shoulder, hip and lower leg). The Pareto-optimal models are identified and further analyzed using human prompts collected through a dedicated observer study. Our findings are: 1) The segmentation performance varies a lot between FMs and prompting strategies; 2) The Pareto-optimal models in 2D are SAM and SAM2.1, in 3D nnInteractive and Med-SAM2; 3) Localization accuracy and rater consistency vary with anatomical structures, with higher consistency for simple structures (wrist bones) and lower consistency for complex structures (pelvis, tibia, implants); 4) The segmentation performance drops using human prompts, suggesting that performance reported on “ideal” prompts extracted from reference labels might overestimate the performance in a human-driven setting; 5) All models were sensitive to prompt variations. While two models demonstrated intra-rater robustness, it did not scale to inter-rater settings. We conclude that the selection of the most optimal FM for a human-driven setting remains challenging, with even high-performing FMs being sensitive to variations in human input prompts. Our code base for prompt extraction and model inference is available: https://github.com/CarolineMagg/segmentation-FM-benchmark/ 1 Introduction Foundation models (FMs) for medical image segmentation have gained significant attention as a promising paradigm for developing promptable methods, which allow users to guide segmentation through simple interactions such as selected points, bounding boxes or scribbles (i.e. prompts). Inspired by the Segment Anything Model (SAM) [ 1 ] , a growing number of medical variants tried to transfer the benefits of broad generalization and promptable inference to the medical domain. These models aim to reduce annotation burden, accelerate data curation, and enhance clinical usability by human-guided interactions and refinements. Despite extensive interest and many evaluation efforts, most studies rely on synthetic or algorithmically generated prompts based on reference segmentation masks. While these represent “ideal” prompts, they fail to account for the inherent variability of human annotations. This results in a limited understanding of real-world prompting behavior , where prompts are not “perfect” but still correct, provided by humans with varying levels of expertise and experience. Consequently, a significant gap exists in analyzing how human-generated prompt variability impacts the final segmentation performance. To address this gap, we conducted an observer study to collect and analyze human prompts. This allowed us to quantify intra- and inter-rater variability and, more importantly, to evaluate model sensitivity to input prompt variations . In addition to addressing the limited understanding of real-world prompt behavior, this study incorporates specific experimental design choices to overcome three critical challenges in the current evaluation landscape: Challenge 1 – Scalability of multi-rater evaluations The growing number of promptable FMs makes exhaustive benchmarking across all available architectures computationally demanding, especially when accounting for multiple human-annotator prompt sets. Furthermore, analyzing the sensitivity of under-performing models offers limited insight. Solution: We implemented a two-stage evaluation strategy. First, 11 models were compared using standardized “ideal” prompts to identify the Pareto-optimal models with the least model parameters, offering the best trade-offs between segmentation performance and parameter efficiency. Second, analysis with human prompts was focused exclusively on these top-performing models, ensuring that our sensitivity evaluation targeted the most relevant candidates. Challenge 2 – Benchmarking fairness and data contamination While public datasets drive progress in the field, many medical FMs are trained on publicly available datasets, making it difficult to compare models fairly when the same data cannot be reused for testing. As a result, ensuring fair, unbiased evaluation often depends on private datasets, as these provide the necessary independence. At the same time, private datasets hinder reproducibility of a study, creating an inherent tension between the need for unbiased assessment and the desire for open, community-verifiable research. Solution: We utilize a hybrid data strategy. By combining private, task-specific data for independent assessment with public data, our study maintains a balance between independent validation and scientific reproducibility. Challenge 3 – Disconnect between benchmark dataset diversity and clinical requirements While recent large-scale benchmarks aim to showcase the generalization of FMs across diverse datasets [ 2 ] , they often lack the depth required to validate performance on specialized clinical tasks. In clinical practice, the integration of a model depends on its performance for specific tasks, such as wrist bone segmentation for osteoarthritis assessment [ 3 ] , tibia and implant segmentation for loosening quantification [ 4 ] , or shoulder joint analysis for humeral head positioning [ 5 ] . Evaluating models across heterogeneous tasks can dilute the focus on these task-specific requirements. Solution: Rather than distributing efforts across many modalities and heterogeneous tasks, we performed a targeted, task-focused investigation on musculoskeletal (MSK) CT scans for bone and implant segmentation. Interested in FM performance in human-driven settings, our work contributes an extensive evaluation of FMs in bone and implant segmentation that moves beyond idealized simulations. By (1) integrating the human-in-the-loop variability, (2) using both public and private data, and (3) focusing on clinically relevant MSK tasks, we provide a more realistic assessment of FM performance. We make our code base for prompt extraction and model inference publicly available 1 1 1 https://github.com/CarolineMagg/segmentation-FM-benchmark/ . 2 Related Work Segment Anything Model The Segment Anything Model (SAM) [ 1 ] enables image segmentation from sparse or dense prompts (bounding boxes, positive and negative points, masks) using three components: an image encoder, a prompt encoder, and a lightweight mask decoder that fuses image and prompt embeddings into a binary mask. Applied to 3D medical scans, SAM operates slice-by-slice and requires a prompt for each slice. SAM2 [ 6 ] extends SAM to video by replacing the image encoder backbone and adding a memory attention module that merges image embeddings, prompt encodings, and predicted masks into a joint representation. Stored in a first-in, first-out (FIFO) memory bank, this representation is queried when segmenting neighboring video frames. Treating CT slices as video frames allows SAM2 to propagate information across slices, enabling volumetric segmentation without prompting each slice individually. Recently, SAM3 [ 7 ] was released as concept-driven foundation model that unifies image, video, and volumetric segmentation and object tracking by using short noun phrases or image examples (i.e., concept) instead of geometric prompts as used in SAM and SAM2. Medical Foundation Models A wide range of geometric prompt-based interactive FMs have been proposed for medical image segmentation: Spanning from fine-tuned 2D SAM variants (Med-SAM [ 8 ] , SAM-Med2D [ 9 ] , ScribblePrompt [ 10 ] , MedicoSAM [ 11 ] ) and fine-tuned 3D SAM2-based models (Medical-SAM2 [ 12 ] , Med-SAM2 [ 13 ] ) to SAM-based extensions to 3D (SAM-Med3D [ 14 ] , SegVol [ 15 ] ), as well as non-SAM CNN-based methods like