Continuous Online Adaptation Driven by User Interaction for Medical Image Segmentation
Wentian Xu, Ziyun Liang, Harry Anthony, Yasin Ibrahim, Felix Cohen, Guang Yang, Daniel Whitehouse, David Menon, Virginia Newcombe, Konstantinos Kamnitsas · 2025
Read Continuous Online Adaptation Driven by User Interaction for Medical Image Segmentation by Wentian Xu, Ziyun Liang, Harry Anthony, Yasin Ibrahim, Felix Cohen, Guang Yang, Daniel Whitehouse, David Menon, Virginia Newcombe, Konstantinos Kamnitsas (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-06717.txt.
Introduction
Wentian Xu and co-authors let a medical image model keep learning from the doctor using it, in a 2025 paper. Their work starts from interactive segmentation, where a clinician clicks to correct an outline the model drew. Hospital scans differ from training data, and those clicks carry exactly the information needed to adapt.
The team first trains models to respond sharply to clicks. Then, during use, the corrected outlines act as teaching labels, and the model updates itself image by image as new scans arrive. This lean online adaptation needs no extra annotation campaign or retraining run. Tests on medical scans with shifted distributions show the model recovers accuracy as the session proceeds. The idea turns everyday corrections into free training, a practical route to keeping clinical AI reliable when it travels between hospitals.
Catalog record
| CREATOR | Wentian Xu, Ziyun Liang, Harry Anthony, Yasin Ibrahim, Felix Cohen, Guang Yang, Daniel Whitehouse, David Menon, Virginia Newcombe, Konstantinos Kamnitsas |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 24 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-06717
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Continuous Online Adaptation Driven by User Interaction for Medical Image Segmentation Wentian Xu, Ziyun Liang, Harry Anthony, Yasin Ibrahim, Felix Cohen, Guang Yang, Daniel Whitehouse, David Menon, Virginia Newcombe, Konstantinos Kamnitsas Abstract Interactive segmentation models use real-time user interactions, such as mouse clicks, as extra inputs to dynamically refine the model predictions. After model deployment, user corrections of model predictions could be used to adapt the model to the post-deployment data distribution, countering distribution-shift and enhancing reliability. Motivated by this, we introduce an online adaptation framework that enables an interactive segmentation model to continuously learn from user interaction and improve its performance on new data distributions, as it processes a sequence of test images. We introduce the Gaussian Point Loss function to train the model how to leverage user clicks, along with a two-stage online optimization method that adapts the model using the corrected predictions generated via user interactions. We demonstrate that this simple and therefore practical approach is very effective. Experiments on 5 fundus and 4 brain MRI databases demonstrate that our method outperforms existing approaches under various data distribution shifts, including segmentation of image modalities and pathologies not seen during training. Code and models will be released upon publication. 1 Introduction Medical image segmentation facilitates disease analysis, diagnosis and treatment. Deep learning methods have driven notable advances in automated segmentation but performance can still fall short of expert-level accuracy, particularly when applied to data that differ significantly from the training data. In many workflows, after a model is deployed, clinicians can provide valuable feedback while using it, to guide segmentation results towards expert-level accuracy. Interactive segmentation methods leverage this by incorporating user prompts—such as clicks, scribbles or bounding boxes—to inform model predictions. A common approach for interactive segmentation modeling is to incorporate user prompts as input channels to a Convolutional Network, with DeepIGeoS [ 19 ] and Interactive FCNN [ 15 ] as representative examples. Other models such as SAM [ 6 ] , MedSAM [ 12 ] , Med-SA [ 22 ] encode user-prompts via a Transformer. They have achieved promising performance on natural and medical image benchmarks, demonstrating the benefit of incorporating user prompts. Standard interactive models, such as the above, optimize parameters on training data without further adaptation to the data processed after model deployment – commonly called the test data distribution . This distribution shift between test and training data, such as due to acquisition with different scanners, can hinder model performance. Various methods were developed to alleviate this. For this purpose, a category of interactive methods fine-tune the model for the specific image they currently segment [ 18 , 16 ] . This process is independent for each image, restoring the original model parameters before processing the next image. Thus these models do not learn the overall test data distribution. To account for data shift more holistically and adapt the model to the overall data distribution processed after deployment, Continuous Adaptation methods were developed for interactive segmentation. These methods use Online Learning [ 5 ] to update model parameters after each test image is processed, continuously optimizing model performance as they process the sequence of data. User prompts are exploited to define an optimization objective 1 1 1 Continuous Adaptation in this context optimizes for the test distribution, thus it does not aim to tackle Catastrophic Forgetting of training data as Continual Learning [ 20 ] . . Prior work on Continual Online Learning for interactive medical image segmentation is rather limited. An early related method is IA+SA [ 7 ] , originally developed for natural images, that combines independent image-level adaptation (IA) and image sequence adaptation (SA). A related prior method applied on medical imaging is TSCA, which leverages the teacher-student framework for adaptation [ 1 ] . These methods leverage user corrections to optimize the model to the test distribution, using a sparse cross-entropy or focal loss defined only on the pixels clicked by the user to update the model. This focuses optimization on the few clicked pixels, neglecting surrounding regions. Hence these methods require additional regularizers to enforce spatial coherence of segmentation, increasing method complexity and hyper-parameters that require tuning. This study proposes a novel Online Adaptation for Interactive Medical-image Segmentation (OAIMS) method that learns from the sequence of data processed after deployment. Guided by the following design principles, the study makes corresponding technical contributions to meet them: a) We focus on user interactions via clicks, because they are simple to use for various tasks, such as segmenting tissues with multiple, irregular components. To train a model leverage user clicks effectively, we design the Gaussian Point Loss (GPL) that enforces coherent, correct segmentation of regions around clicks. b) Adaptation of interactive models should bolster both the ability to segment and to react after user clicks to correct errors. We designed a two-stage procedure for this: Stage-1 optimizes ability to segment with one localization click; Stage-2 optimizes use of multiple correction clicks. Importantly, we leverage segmentations corrected via user interactions as supervision to learn the new distribution. c) We strive for simplicity, as adaptation must be reliable on varying new distributions in clinical workflows, without oversight by technical experts. We demonstrate this no-frills method outperforms prior approaches on distribution shifts across 5 fundus and 4 brain MRI databases with varying modalities and diseases. 2 Methods The overview of our method is shown in Fig. 1 . To enhance the segmentation performance , the model is fine-tuned with a pseudo ground truth. This pseudo ground truth is the prediction mask of the base interactive model given T user clicks. In stage 1 , the model is fine-tuned with only a single localization click as input. In stage 2 , the output is fine-tuned with additional generated pseudo points as inputs. To improve the model’s ability to utilize clicks under distribution shift , a Gaussian Point Loss (GPL) is proposed. We below describe our method for binary segmentation but it can be extended to multi-class settings, as demonstrated in the Experiments section. Figure 1: Overview of proposed method. Stage 1: Fine-tune with one localization point as input on final Predicted mask. Stage 2: Fine-tune with multiple correction points and enhance the model’s ability to utilize clicks. 2.1 Interactive Model We define the interactive model as f(I,C,\theta) , where I is the input image, C is the set of user-provided clicks, and \theta represents the model parameters. The user sets C to the foreground or background class when clicking. During training, the model is trained to predict given C . During online adaptation, we receive a sequence of images, {I_{1},I_{2},\dots,I_{n}} . For each image, at each iteration t , a new click c_{t} based on the prediction f(I_{n},C_{t-1},\theta) is added to the click set C_{t}=C_{t-1}\cup\{c_{t}\} , until it reaches C_{T} after T iterations. The final prediction for an image I_{n} is given by P_{nT}=f(I_{n},C_{T},\theta) . We then apply the softmax and argmax functions to obtain the final binary segmentation mask, P_{n\_{final}} , for image I_{n} . For simplicity, we omit the index n in the subsequent representations. 2.2 Online Adaptation Strategy When adapting interactive segmentation models to a new distribution, it is important to strengthen both their ability to segment and to leverage user