MultiverSeg: Scalable Interactive Segmentation of Biomedical Imaging Datasets with In-Context Guidance

Hallee E. Wong, Jose Javier Gonzalez Ortiz, John Guttag, Adrian V. Dalca · 2024

Read MultiverSeg: Scalable Interactive Segmentation of Biomedical Imaging Datasets with In-Context Guidance by Hallee E. Wong, Jose Javier Gonzalez Ortiz, John Guttag, Adrian V. Dalca (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-15058.txt.

Introduction

Hallee Wong and co-authors present MultiverSeg, a tool that lets researchers outline structures across a new biomedical image set without starting over on each picture, in a 2024 paper. Segmenting a novel dataset normally means hand-tracing every image or having labeled examples from the same task already, which new studies lack. MultiverSeg takes the user's clicks, boxes, or scribbles on the current image, plus every image labeled in the session, as context.

As the context grows, each image needs fewer interactions to reach an accurate outline. Against a state-of-the-art interactive segmenter working image by image, MultiverSeg cut clicks by thirty-six percent and scribbles by twenty-five percent at the same quality bar. The tasks were new to the model. The authors release code and model weights. They pitch it at labs that face a fresh segmentation task every project.

Catalog record

CREATORHallee E. Wong, Jose Javier Gonzalez Ortiz, John Guttag, Adrian V. Dalca
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE73 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-15058

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MultiverSeg: Scalable Interactive Segmentation of Biomedical Imaging Datasets with In-Context Guidance Hallee E. Wong, Jose Javier Gonzalez Ortiz, John Guttag, Adrian V. Dalca Abstract Medical researchers and clinicians often need to perform novel segmentation tasks on a set of related images. Existing methods for segmenting a new dataset are either interactive, requiring substantial human effort for each image, or require an existing set of previously labeled images. We introduce a system, MultiverSeg , that enables practitioners to rapidly segment an entire new dataset without requiring access to any existing labeled data from that task or domain. Along with the image to segment, the model takes user interactions such as clicks, bounding boxes or scribbles as input, and predicts a segmentation. As the user segments more images, those images and segmentations become additional inputs to the model, providing context. As the context set of labeled images grows, the number of interactions required to segment each new image decreases. We demonstrate that MultiverSeg enables users to interactively segment new datasets efficiently, by amortizing the number of interactions per image to achieve an accurate segmentation. Compared to using a state-of-the-art interactive segmentation method, MultiverSeg reduced the total number of clicks by 36% and scribble steps by 25% to achieve 90% Dice on sets of images from unseen tasks. We release code and model weights at https://multiverseg.csail.mit.edu . Figure 1 : MultiverSeg enables users to rapidly segment new datasets . The MultiverSeg network takes as input an image to segment, user interactions, and a context set of previously segmented image-segmentation pairs ( left ). As the user completes more segmentations, those images and segmentations become additional inputs to the model, populating the context set. As the context set of labeled images grows, the number of interactions required to achieve an accurate segmentation decreases ( right ). 1 Introduction Segmentation is an important step in biomedical image analysis pipelines. Biomedical and clinical researchers often acquire novel images types or identify new regions of interest, and need to perform new segmentation tasks. Typically, scientists want to segment the same region of interest in many similar images from a new dataset. Manually segmenting images is labor-intensive and requires domain expertise. Interactive segmentation systems, in which a user provides a few clicks or scribbles on an image to produce a predicted segmentation, help to speed up the annotation of individual images. But with existing interactive segmentation systems, the user must independently repeat the same process for each image [ 61 , 130 , 88 , 85 , 19 , 137 ] . Ideally, a system should be able to learn from experience, becoming more accurate as the user completes more segmentations from the same task. We propose MultiverSeg , a new interactive system that, as more images are segmented, progressively reduces the number of user interactions needed to predict accurate segmentations ( Fig. 1 ). MultiverSeg takes as input user interactions for a new image, along with a context set , of example (previously segmented) image-segmentation pairs. To segment a new dataset, the user begins by interactively segmenting the first image. Once completed, the example becomes an input to MultiverSeg, providing context for the segmentation of subsequent examples. The user interactively segments the next image with MultiverSeg using bounding boxes, clicks or scribbles. As the user labels more images, the context set grows and the number of interactions required to achieve the desired segmentation of subsequent images decreases, often to zero. Unlike existing interactive segmentations systems [ 130 , 61 , 85 , 88 , 140 , 127 , 78 , 79 , 135 , 111 , 126 ] where the work required to segment a dataset is linear in the number of images, MultiverSeg enables users to rapidly segment entire datasets. This paper • Presents MultiverSeg, a new interactive segmentation framework that progressively reduces the amount of user interaction needed to predict accurate segmentations, as more images are segmented in a particular task. • Introduces a model that segments an image given user prompts (bounding boxes, clicks and/or scribbles) and a variably-sized context set of previously labeled example images and segmentations. This network enables scalable segmentation of datasets by performing interactive segmentation in context . • Demonstrates that MultiverSeg can dramatically reduce the total number of user interactions needed to segment a collection of medical images. 2 Related Work Interactive Segmentation. Recent interactive segmentation models can generalize to new segmentation tasks in medical [ 130 , 88 , 136 ] and natural images [ 61 , 106 ] . While these methods are effective for segmenting single images, they require prohibitively extensive human interaction when segmenting large datasets. To incorporate new information, they must be retrained or fine-tuned. In contrast, at inference time MultiverSeg can be conditioned on example segmentations from the new task, dramatically reducing the human effort required for accurate segmentation. Many works have sought to improve task-specific interactive segmentation performance by fine-tuning foundation models, either through full fine-tuning [ 58 ] or more efficient adaption techniques [ 133 , 98 , 75 , 124 , 48 , 138 , 105 ] . The fine-tuning must be repeated for each new task or group of tasks. It requires many relevant annotated images and the substantial computational resources needed to train a large model. In-Context Learning. Recent in-context learning approaches to segmentation [ 16 , 104 , 128 , 132 , 22 , 47 ] enable users to perform new tasks by providing a set of labeled examples to the model at inference time. These methods often need existing large context sets to achieve adequate performance, and provide no mechanism for correcting predicted segmentations. A few works have explored in-context segmentation using a context set of example images with user annotations on those example images. OnePrompt [ 132 ] segments a medical image given exactly one context example with click, scribble, bounding box or mask annotation. In contrast, MultiverSeg has mechanisms to facilitate a variable number of context set entries and enables users to incorporate interactions on the target image along with the context of previous segmentations, enabling substantially richer use cases. LabelAnything [ 25 ] is a few-shot framework that enables multi-label segmentation of a target natural image given a small context set of example images with click, box, or mask annotations on the examples. In contrast, MultiverSeg can leverage large context sets, and enables users to provide interactions on the target image and corrections to refine the prediction to get the desired result. Continual Learning. One approach to segmenting a new dataset involves manually labeling a large number of images, and then training an automatic task-specific model [ 53 ] to segment the rest. For example, MonaiLabel [ 28 ] is an open-source tool that packages this process. In contrast, MultiverSeg can be adapted at inference time using new labels (collected manually or interactively) without the need to re-train. Annotation-Efficient Learning. Another approach to segmenting a new dataset is to collect sparse annotations on many images and train an automatic segmentation model using these annotations for supervision. The annotations can be bounding boxes [ 67 , 121 ] , clicks [ 77 , 81 , 110 ] or scribbles [ 73 , 72 ,

Read the complete text