In-context learning for medical image segmentation

Eichi Takaya, Shinnosuke Yamamoto · 2024

Read In-context learning for medical image segmentation by Eichi Takaya, Shinnosuke Yamamoto (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-13299.txt.

Introduction

Eichi Takaya and Shinnosuke Yamamoto teach an image model to segment a whole scan volume from a whisper of annotation. Outlining organs slice by slice is essential for judging treatment and planning radiotherapy, yet hand annotation is the bottleneck strangling medical AI. Their method, In-context Cascade Segmentation, extends a few-shot framework that segments from example images without retraining.

Each slice's fresh segmentation joins the support set, and the cascade sweeps forward and backward through the volume, keeping neighboring slices consistent. On a cardiac dataset covering eight regions it significantly improved accuracy. Annotation-light segmentation could finally scale to the archive sizes hospitals actually hold.

Catalog record

CREATOREichi Takaya, Shinnosuke Yamamoto
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE27 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-13299

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

In-context learning for medical image segmentation Thanks: [email protected] Eichi Takaya, Shinnosuke Yamamoto Abstract Accurate annotation of medical images, such as MRI and CT scans, is critical for assessing treatment efficacy and planning radiotherapy. However, the substantial time and effort required of medical professionals limit the volume of annotated data, thereby constraining AI applications in medical imaging. To address this challenge, we propose in-context cascade segmentation (ICS), a novel method that minimizes annotation needs while maintaining high segmentation accuracy for sequential medical images. ICS extends the UniverSeg framework, which performs few-shot segmentation based on support images without additional training. By iteratively incorporating the inference results of each slice into the support set, ICS propagates information forward and backward through the sequence, ensuring inter-slice consistency. We evaluate ICS on the HVSMR dataset encompassing eight cardiac regions. Experimental results demonstrate that ICS significantly outperforms baseline methods, particularly in complex anatomical regions where maintaining boundary consistency across slices is crucial. Furthermore, we show that the number and placement of initial support slices can markedly influence segmentation accuracy. ICS thus provides a promising solution for reducing annotation burdens while delivering robust segmentation results, paving the way for broader implementation in both clinical and research settings. K eywords Medical Image Segmentation \cdot In-context Learning \cdot Semi-supervised Learning 1 Introduction Research on AI applications in healthcare, particularly automated segmentation in medical imaging, has accelerated in recent years in tandem with deep learning advances [ 1 ] [ 2 ] . Segmentation of sequential imaging data, such as CT and MRI scans, is essential for surgical and radiotherapy planning [ 3 ] [ 4 ] , as well as for evaluating cancer treatment efficacy [ 5 ] . Given the high manual workload and associated costs, there is a pressing need to automate these processes. In addition, building AI-assisted diagnostic systems frequently requires defining regions of interest (ROIs) for lesions within training datasets. For instance, ROIs are used for image cropping in classification tasks [ 6 ] and for producing radiomics features [ 7 ] . Furthermore, recent findings suggest that performing pixel-level lesion classification, differentiating benign from malignant conditions using segmentation methods, can improve diagnostic accuracy [ hooper_case_nodate ] . A range of deep neural network architectures has been introduced, starting with U-Net [ 8 ] , More recently, vision-transformer-based models have gained popularity [ 9 ] . Large-scale pretraining on datasets such as ImageNet [ 10 ] and RadImageNet [ 11 ] [ 12 ] , alongside foundation models capable of zero-shot inference [ 13 ] [ 14 ] , has further enhanced generalization performance across diverse segmentation tasks. Nonetherless, domain shifts (differences in patient populations or imaging protocols), modality shifts (variations between imaging modalities such as CT and MRI), and site shifts (differences in imaging equipment or procedures across institutions) remain challenging [ 15 ] [ 16 ] [ 17 ] . Additionally, these approaches often exhibit limited generalizability to rare diseases [ 18 ] , frequently necessitating the creation of new, task-specific training sets. Reducing the burden of annotating entire image sequences therefore remains a critical objective. Our study aims to develop a method that minimizes the annotation load for sequential imaging datasets. We introduce a segmentation approach based on in-context learning, specifically leveraging UniverSeg [ 19 ] , a framework designed for few-shot medical image segmentation. By iteratively incorporating minimal initial labels (“support” images) and augmenting the support set with predicted masks, we implement an in-context cascade segmentation (ICS) strategy. We evaluated ICS on eight cardiac segmentation tasks in the HVSMR dataset [ 20 ] , highlighting its performance and characteristics. The remainder of this paper is organized as follows. Section 2 reviews related work on medical image segmentation, including in-context and semi-supervised learning approaches. Section 3 describes our problem setting and details the proposed method. Section 4 outlines the experimental setup and reports results on HVSMR data. Section 5 discusses the findings, and Section 6 concludes with recommendations for future research directions and open challenges. 2 Related Works 2.1 Medical Image Segmentation Prior to the widespread adoption of deep learning, segmentation in medical imaging was predominantly approached through traditional machine learning methods employing both unsupervised and supervised strategies [ 21 ] . In unsupervised methods, clustering and region-based segmentation algorithms, such as K-means and watershed, were extensively utilized [ 22 ] [ 23 ] . These approaches partitioned images based on the statistical properties of pixels or voxels, eliminating the need for prior labeling. Supervised techniques typically relied on engineered features and models such as support vector machines (SVMs) and random forests for pixel-level classification [ 24 ] [ 25 ] . Although these methods succeeded in certain tasks, they generally lacked accuracy and scalability. Following the work by Ciresan et al. [ 26 ] , deep neural networks enabled automatic feature extraction from images, thereby facilitating segmentation. Subsequently, U-Net [ 8 ] emerged as a standard architecture, leveraging an encoder–decoder design and skip connections to capture both local and global features for high-precision segmentation. In recent years, transformer-based models and 3D segmentation techniques have further advanced the state of the art by tackling complex anatomical structures and diverse imaging modalities [ 9 ] [ 27 ] [ 28 ] . Large-scale segmentation frameworks, such as MedSAM [ 13 ] [ 14 ] built on Segment anything model (SAM) [ 29 ] [ ravi_sam_nodate ] , have also shown promise in achieving high accuracy from minimal annotations. The UniverSeg [ 19 ] model used in our study belongs to this class of large-scale medical image segmentation frameworks. 2.2 In-context Learning In-context learning [ 30 ] [ 31 ] was initially introduced in large language models (LLMs), wherein a pre-trained model can adapt to new tasks given only a few examples without requiring additional training. In GPT-3 [ 32 ] , which first popularized the term, it was demonstrated that just a small number of examples in the prompt can enable the model to generalize effectively. Moreover, theoretical work by Oswald et al. [ 33 ] suggests that the underlying mechanism in transformers performing in-context learning parallels gradient descent. The concept of in-context learning has been extended to image recognition [ 34 ] . In UniverSeg [ 19 ] , for example, labeled "support" images guide the segmentation of new query images. In this study, we further develop this approach of in-context learning and apply it as ICS, where inference results are sequentially utilized to improve segmentation consistency across sequential slices. 2.3 Semi-supervised Learning Semi-supervised learning seeks to leverage unlabeled data when labeled datasets are scarce, making it especially relevant in medical image segmentation, where annotation is both expensive and time-consuming [ 35 ] [ 36 ] . Common approaches include self-training [ 37 ] , co-training [ 38 ] , and graph-based methods [ 39 ] . Self-training re-trains the model using pseudo-labels generated from unlabeled data, while co-training employs two distinct classifiers that iteratively learn from each other’s pseudo-labels. In graph-based methods, data are represented as a graph, and labels are propagated to unlabeled nodes. Among these techniques, sequential semi-supervised segmentation (4S) [ 40 ] is particularly related to our approach. It begins with a small number of consecutively labeled slices

Read the complete text