Show and Segment: Universal Medical Image Segmentation via In-Context Learning
Yunhe Gao, Di Liu, Zhuowei Li, Yunsheng Li, Dongdong Chen, Mu Zhou, Dimitris N. Metaxas · 2025
Read Show and Segment: Universal Medical Image Segmentation via In-Context Learning by Yunhe Gao, Di Liu, Zhuowei Li, Yunsheng Li, Dongdong Chen, Mu Zhou, Dimitris N. Metaxas (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-19359.txt.
Introduction
Yunhe Gao and co-authors present Iris, a system that outlines structures in medical images after seeing just a few examples. Their 2025 paper tackles the wide range of bodies, scan types, and outline tasks in medicine. Older systems need fresh training for each new task or class. Iris instead learns from reference pairs, where each pair shows an image and its correct outline. A small module squeezes the task lesson from those pairs into a compact code.
That code then guides the outlining of new images, with no retraining. Tests span a dozen data sets across many scan kinds. Iris matches or nears task-built models on familiar tasks. On seven held-out sets with new classes, it generalizes better than rivals. The code also picks up links between body parts across sets on its own. Such links may teach anatomy without any direct labels.
Catalog record
| CREATOR | Yunhe Gao, Di Liu, Zhuowei Li, Yunsheng Li, Dongdong Chen, Mu Zhou, Dimitris N. Metaxas |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-19359
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Show and Segment: Universal Medical Image Segmentation via In-Context Learning Yunhe Gao, Di Liu, Zhuowei Li, Yunsheng Li, Dongdong Chen, Mu Zhou, Dimitris N. Metaxas Abstract Medical image segmentation remains challenging due to the vast diversity of anatomical structures, imaging modalities, and segmentation tasks. While deep learning has made significant advances, current approaches struggle to generalize as they require task-specific training or fine-tuning on unseen classes. We present Iris , a novel I n-context R eference I mage guided S egmentation framework that enables flexible adaptation to novel tasks through the use of reference examples without fine-tuning. At its core, Iris features a lightweight context task encoding module that distills task-specific information from reference context image-label pairs. This rich context embedding information is used to guide the segmentation of target objects. By decoupling task encoding from inference, Iris supports diverse strategies from one-shot inference and context example ensemble to object-level context example retrieval and in-context tuning. Through comprehensive evaluation across twelve datasets, we demonstrate that Iris performs strongly compared to task-specific models on in-distribution tasks. On seven held-out datasets, Iris shows superior generalization to out-of-distribution data and unseen classes. Further, Iris’s task encoding module can automatically discover anatomical relationships across datasets and modalities, offering insights into medical objects without explicit anatomical supervision. 1 Introduction The accurate segmentation of anatomical structures in medical images is fundamental for clinical practice and biomedical research, enabling precise diagnosis De Fauw et al. 2018 ; Shen et al. 2015 and treatment planning Nestle et al. 2005 . While deep learning has demonstrated remarkable success Liu et al. 2021d ; Wang et al. 2022 , the vast diversity of anatomical structures, imaging modalities, and clinical tasks poses long-standing challenges for developing truly generalizable solutions. Current efforts typically focus on disease-specific tasks or a limited set of anatomical structures Isensee et al. 2021 ; Liu et al. 2021b ; Liu et al. 2022 ; Chang et al. 2022 ; Gao et al. 2022 ; Gao 2024 ; He et al. 2023 ; Zhangli et al. 2022 ; Liu et al. 2021a , struggling to handle the heterogeneous landscape of medical imaging that spans diverse modalities, body regions, and diseases Yoon et al. 2023 ; Niu et al. 2024 . Figure 1 : Comparison of medical image segmentation approaches. A) Task-specific models require training separate models for each task, limiting their flexibility and scalability. B) Multi-task universal models can handle diverse tasks and imaging modalities, but fail on novel classes. C) SAM-based foundation models enable flexible segmentation through user interactions, but impractical for high-throughput automated processing. D) Our proposed Iris combines automatic processing with flexible adaptation via in-context learning, enabling both seen and unseen task segmentation without any manual interaction or retraining. These task-specific methods show critical limitations compared to human experts’ capabilities. First, existing models often perform poorly on out-of-distribution examples Zhou et al. 2022 —a common scenario in medical imaging where variations arise from different imaging centers, patient populations, and acquisition protocols. Second, traditional segmentation models, while achieving a high accuracy on their trained tasks, lack the adaptability to handle novel classes without extensive retraining or fine-tuning Zhang et al. 2023b . This dilemma fundamentally limits task-specific models’ applicability in dynamic clinical settings and research environments, where new segmentation tasks continue to emerge over the course of real-world practice. Recent research has explored several directions to address these challenges (Figure 1 ). Universal medical segmentation models Zhang et al. 2021 ; Liu et al. 2023a ; Ye et al. 2023 ; Ulrich et al. 2023 attempt to leverage synergy among multiple tasks across diverse datasets to learn robust representations, yet struggling with unseen classes and requiring fine-tuning. Foundation models with interactive capabilities, such as SAM Kirillov et al. 2023 and its medical variants Zhang et al. 2024 ; Ma et al. 2024 ; Cheng et al. 2023 ; Wang et al. 2024 , offer flexibility via user prompts. But they require multiple interactions for optimal segmentation results, especially for complex 3D structures, and lack the efficiency for large-scale automated analysis. In addition, in-context learning (ICL) methods Butoi et al. 2023 ; Rakic et al. 2024 show promise in automatically handling arbitrary new tasks through a few reference examples, but current methods exhibit suboptimal performance compared to task-specific models and suffer from computational inefficiencies, requiring expensive reference encoding during each inference step. To address these fundamental challenges, we present Iris framework for universal medical image segmentation via in-context learning. At its core, Iris features a lightweight task encoding module that efficiently distills task-specific information from reference image-label pairs into compact task embeddings, which then guide the segmentation of target objects. Unlike existing ICL methods Butoi et al. 2023 ; Rakic et al. 2024 , Iris decouples the task definition from query image inference, eliminating redundant context encoding while enabling flexible inference strategies, all coming with high computational efficiency. Our main contributions include: • A novel in-context learning framework for 3D medical images, enabling a strong adaptation to arbitrary new segmentation tasks without model retraining or fine-tuning. • A lightweight task encoding module that captures task-specific information from reference examples, handling medical objects of varying sizes and shapes. • Multiple flexible inference strategies suitable for different practical scenarios, including one-shot inference, context ensemble, object-level context retrieval, and in-context tuning. • Comprehensive experiments on 19 datasets demonstrate Iris’s superior performance across both in-distribution and challenging scenarios, particularly on held-out domains and novel anatomical structures. It extends to reveal the capability of automatically discovering meaningful anatomical relationships across datasets and modalities. 2 Related Work Medical Universal Models. Universal medical image segmentation models aim to address the data heterogeneity across tasks and modalities while learning generalizable feature representations. Early works include multi-dataset learning through label conditioning Dmitriev and Kaufman 2019 , organ size constraints Zhou et al. 2019 , and pseudo-label co-training Huang et al. 2020 . Recent works are placed on sophisticated task encoding strategies. DoDNet Zhang et al. 2021 pioneered one-hot task vectors with an extension into TransDoDNet Xie et al. 2023 using transformer backbones. Advances include CLIP-driven models using semantic encodings Liu et al. 2023a , task-specific heads in MultiTalent Ulrich et al. 2023 , and modality priors in Hermes Gao 2024 . UniSeg Ye et al. 2023 introduced learnable task prompts and MedUniseg Ye et al. 2024 unified 2D/3D image handling. Despite of substantial efforts, these universal models all require fine-tuning when assessing unseen classes. In contrast, Iris enables the segmentation of unseen classes only through a single reference image-label pair without any model finetuning. SAM-based Interactive Models. Segment Anything Model (SAM) Kirillov et al. 2023 emerges as a shifting paradigm of interactive segmentation via its prompt-based architecture and large-scale training. SAM’s success has inspired a range of medical variants. Major examples include MedSAM Ma et al. 2024 with 1.5M image-mask pairs for 2D segmentation, SAM-Med2D Cheng et al. 2023 trained on 4.6M images, and SAM-Med3D Wang et al. 2024 extending to volumetric data with 22K 3D images. These efforts all require multiple prompts and interactive refinements, especially for analyzing complex objects in 3D scenarios. This interaction-dependent design becomes a bottleneck in high-throughput scenarios that an automated processing of large-scale datasets is much desired. As comparison, Iris