Prompt learning with bounding box constraints for medical image segmentation
Mélanie Gaillochet, Mehrdad Noori, Sahar Dastani, Christian Desrosiers, Hervé Lombaert · 2025
Read Prompt learning with bounding box constraints for medical image segmentation by Mélanie Gaillochet, Mehrdad Noori, Sahar Dastani, Christian Desrosiers, Hervé Lombaert (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-02743.txt.
Introduction
Five authors, Mélanie Gaillochet, Mehrdad Noori, Sahar Dastani, Christian Desrosiers, and Hervé Lombaert, wrote this 2025 paper on cheaper training for medical image segmentation. Tracing organs pixel by pixel takes expert time and costs a lot. Box outlines around a target are far quicker to draw, so methods that learn from boxes are in demand.
Vision foundation models can segment well when a user gives a prompt, such as a point or a box. Prompt learning adapts such models to a task and cuts user input. Past prompt learning, though, still needs full pixel masks for training. The new frame joins the two ideas: it learns to make its own prompts for a foundation model using only box outlines as training labels. That pairing could bring strong segmentation to clinics that cannot afford dense annotation.
Catalog record
| CREATOR | Mélanie Gaillochet, Mehrdad Noori, Sahar Dastani, Christian Desrosiers, Hervé Lombaert |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-02743
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Prompt learning with bounding box constraints for medical image segmentation Mélanie Gaillochet, Mehrdad Noori, Sahar Dastani, Christian Desrosiers, Hervé Lombaert Abstract Pixel-wise annotations are notoriously labourious and costly to obtain in the medical domain. To mitigate this burden, weakly supervised approaches based on bounding box annotations —much easier to acquire—offer a practical alternative . Vision foundation models have recently shown noteworthy segmentation performance when provided with prompts such as points or bounding boxes. Prompt learning exploits these models by adapting them to downstream tasks and automating segmentation , thereby reducing user intervention . However, existing prompt learning approaches depend on fully annotated segmentation masks . This paper proposes a novel framework that combines the representational power of foundation models with the annotation efficiency of weakly supervised segmentation . More specifically, our approach automates prompt generation for foundation models using only bounding box annotations . Our proposed optimization scheme integrates multiple constraints derived from box annotations with pseudo-labels generated by the prompted foundation model. Extensive experiments across multi-modal datasets reveal that our weakly supervised method achieves an average Dice score of 84.90 % 84.90\% in a limited data setting, outperforming existing fully-supervised and weakly-supervised approaches . The code is available at https://github.com/Minimel/box-prompt-learning-VFM.git . K eywords Bounding box \cdot Constraints \cdot Foundation model \cdot Medical \cdot Prompt \cdot Segmentation \cdot Weakly supervised 1 Introduction Figure 1: Differences between a) Interactive SAM and b-d) Various adaptive methods for prompt automation: b) Using SAM’s prompt encoder with physical prompts generated by a trained prompt module, c) Directly generating dense and/or sparse prompt embeddings ( dark yellow ) with a prompt module trained with ground truth (GT) masks, and d) Our method, which generates prompt embeddings from the image using only bounding box annotations during training. The precise delineation of regions of interest is a crucial step in clinical decision-making, affecting the diagnosis, treatment planning and patient outcome. However, manual annotation of medical images remains a labour-intensive and time-consuming process. With an ever-growing volume of medical imaging data, developing efficient and accurate segmentation methods has become a major challenge. Automatic segmentation algorithms [ 1 ] have been introduced to alleviate the burden of manual annotation and reduce the impact of expert subjectivity and inadvertent errors . In medical image analysis, segmentation methods based on variants of the UNet [ 2 ] and lately of the vision transformer architecture [ 3 , 4 ] , have achieved state-of-the-art performance on multiple tasks and datasets [ 5 , 6 , 7 ] . However, these models depend on large, fully-annotated datasets for training, and their performance typically deteriorates when training data is scarce. Yet , the prohibitive cost of manually labeling medical images makes the collection of such well-annotated datasets difficult, hampering the development of these data-hungry models. Recently, large promptable vision models [ 8 , 9 , 10 ] have gained considerable attention for their flexibility and generalization abilities on multiple natural imaging benchmarks. Unlike specialized models trained on domain-specific datasets, these foundation models can generate, for any new image, informative representations which can be leveraged to produce a segmentation mask . Notably, the Segment Anything Model (SAM) [ 8 ] has showcased impressive zero-shot capabilities by generating accurate segmentation masks for tasks and classes unseen during training. Current foundation models require user-provided prompts, typically manually crafted, to tailor the segmentation outputs to specific tasks . The zero-shot performance of these models heavily depends on the quality of the user prompt [ 11 , 12 , 13 , 10 ] . However, prompts can be ambiguous, particularly in medical images where object boundaries are often subtle , leading to poor predictions (see Fig. 2 ). To fully exploit the potential of foundation models and improve scalability, recent efforts have focused on automating prompt generation. In particular, prompt tuning [ 14 , 15 , 16 , 17 , 12 ] adapts large models for downstream tasks by optimizing a small set of trainable prompt embeddings, rather than updating millions of model parameters . Prompt tuning searches for the best input prompt to satisfy the target task. Visual prompt tuning [ 18 , 19 ] has demonstrated remarkable adaptation performance with minimal trainable parameters, and several studies have applied this approach to SAM [ 14 , 15 , 17 , 20 , 13 , 21 ] . Existing methods for medical image segmentation have focused on automatically generating a physical prompt—i.e., point or box [ 15 , 20 ] or a prompt embedding [ 14 , 13 , 21 ] (see Fig. 1 ). However, these techniques still heavily rely on fully annotated segmentation masks , which are burdensome to obtain in the medical domain. In parallel to advanced prompt tuning techniques, weakly supervised learning offers a pragmatic approach to reduce the reliance on exhaustive manual annotations by incorporating more accessible forms of labeling. Weak labels can come in various shapes, such as scribbles [ 22 ] , image tags [ 23 ] , points [ 24 ] or bounding boxes [ 25 , 26 , 27 , 28 ] . Among these , bounding boxes are particularly appealing due to their simplicity and light storage—in practice, only two corner coordinates are needed to define a bounding box. Bounding boxes have been used as pseudo-labels to generate initial segmentation proposals [ 29 , 30 ] , which are refined iteratively until a more precise segmentation is obtained. However, such approaches are subject to error propagation if the initial segmentation is inaccurate, ultimately impairing model performance. To mitigate this issue , alternative methods have introduced attention mechanisms to improve gradient flow [ 31 , 32 ] or imposed constraints on the output probabilities during optimization [ 33 , 28 ] . Building on these advancements, our work integrates constraint-based optimization with bounding box annotations into the framework of visual prompt tuning. Our work introduces a novel framework that leverages the complementary strengths of foundation models and weakly supervised learning through prompt tuning . We train an auxiliary prompt module using only bounding box annotations— much easier to obtain than ground truth masks—to automatically generate informative prompt embeddings from input images . Relying on weak labels significantly reduces the annotation burden, making the adaptation of foundation models more scalable for medical imaging applications. Building on our preliminary work [ 34 ] , we address prior limitations by moving beyond using weak-label constraints that only exploited bounding box tightness. Our updated training strategy now employs a box-based multi-loss optimization framework that integrates predictions from the prompted foundation model with consistency-based regularization, leading to more robust performance. Furthermore, while our earlier work focused exclusively on MedSAM, we now demonstrate that SAM—despite being trained on out-of-domain data—can serve as a general backbone foundation model when coupled with a deeper prompt module and our refined optimization strategy. Figure 2: Examples of SAM predictions with varying noise levels in the box prompts. From left to right: no noise (tight box), 0-1.5%, 1.5-3%, and 3-5% pixel displacement from the original tight box. Ground-truth annotations and associated box prompts are shown in red and green, respectively, with predicted segmentation masks overlaid in blue. Prompt ambiguity and noise, combined with applications to out-of-domain data, can cause vision foundation