BALD-SAM: Disagreement-based Active Prompting in Interactive Segmentation
PRITHWIJIT CHOWDHURY 1, MOHIT PRABHUSHANKAR 1 , , AND GHASSAN ALREGIB. 1 · 2026
Read BALD-SAM: Disagreement-based Active Prompting in Interactive Segmentation by PRITHWIJIT CHOWDHURY 1, MOHIT PRABHUSHANKAR 1 , , AND GHASSAN ALREGIB. 1 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-10828.txt.
Introduction
Prithwijit Chowdhury, Mohit Prabhushankar, and Ghassan Alregib present BALD-SAM, a method for choosing where the next prompt should go in interactive segmentation with the Segment Anything Model. Real annotation is iterative: annotators see the mask and place prompts to clear up doubt, yet pipelines lean on visual judgment. The authors cast this as spatial active learning, using model scores to pick the most informative spot. BALD-SAM freezes the model and puts uncertainty modeling on a small learned head. Across sixteen datasets it ranked first or second on fourteen.
Catalog record
| CREATOR | PRITHWIJIT CHOWDHURY 1, MOHIT PRABHUSHANKAR 1 , , AND GHASSAN ALREGIB. 1 |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 67866 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.10828
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
BALD-SAM: Disagreement-based Active Prompting in Interactive Segmentation PRITHWIJIT CHOWDHURY 1, MOHIT PRABHUSHANKAR 1 , , AND GHASSAN ALREGIB. 1 Abstract The Segment Anything Model (SAM) has revolutionized interactive segmentation through spatial prompting. While existing work primarily focuses on automating prompts in various settings, real-world annotation workflows involve iterative refinement where annotators observe model outputs and strategically place prompts to resolve ambiguities. Current pipelines typically rely on the annotator’s visual assessment of the predicted mask quality. We postulate that a principled approach for automated interactive prompting is to use a model-derived criterion to identify the most informative region for the next prompt. In this work, we establish active prompting : a spatial active learning approach where locations within images constitute an unlabeled pool and prompts serve as queries to prioritize information-rich regions, increasing the utility of each interaction. We further present BALD-SAM : a principled framework adapting Bayesian Active Learning by Disagreement (BALD) to spatial prompt selection by quantifying model (epistemic) uncertainty. To do so, we freeze the entire model and apply Bayesian uncertainty modeling only to a small learned prediction head, making intractable uncertainty estimation practical for large multi-million parameter foundation models. Across 16 datasets spanning natural, medical, underwater, and seismic domains, BALD-SAM demonstrates strong cross-domain performance, ranking first or second on 14 of 16 benchmarks. We validate these gains through a comprehensive ablation suite covering 3 SAM backbones and 35 Laplace posterior configurations (5 subset sizes × \times 7 posterior sample counts), amounting to 38 distinct ablation settings. Beyond strong average performance, BALD-SAM surpasses human prompting and, in several categories, even oracle prompting, while consistently outperforming one-shot baselines such as Saliency, K-Medoids, Max Distance, and Shi-Tomasi in final segmentation quality, particularly on thin and structurally complex objects. Citation P. Chowdhury, M. Prabhushankar, and G. AlRegib, ”BALD-SAM: Disagreement-based Active Prompting in Interactive Segmentation”, submitted at IEEE Access. Review First submission: 02 March 2026 (Under Consideration) Code will be released upon acceptance Copyright © Creative Commons Attribution CC BY 4.0 Contact {pchowdhury6, alregib}@gatech.edu https://alregib.ece.gatech.edu/ Corresponding author [email protected] † † history: Date of publication xxxx 00, 0000, date of current version xxxx 00, 0000. † † doi: 10.1109/ACCESS.2017.DOI † † address: OLIVES at the Georgia Institute of Technology † † titlenote: This work is supported by the ML4Seismic Industry Partners at Georgia Tech † † corresponding: Corresponding author: Ghassan AlRegib (e-mail: [email protected]). I Introduction Interactive image segmentation enables users to delineate objects through iterative feedback, combining human semantic understanding with computational efficiency. This paradigm has proven essential across diverse applications: medical professionals annotate anatomical structures for diagnosis and treatment planning [ 61 ] , geoscientists identify subsurface formations in seismic surveys [ 60 , 45 ] , ecologists track species in underwater imagery [ 41 ] , and computer vision researchers create training datasets for recognition systems [ 32 ] . Traditional interactive segmentation methods require domain-specific models trained on labeled data from each target domain, limiting applicability and necessitating extensive retraining for new applications. The emergence of foundation models has transformed the landscape of interactive segmentation. The Segment Anything Model (SAM) [ 27 ] , trained on 11 million images and 1.1 billion masks, has demonstrated unprecedented zero-shot segmentation capabilities, enabling accurate mask generation on previously unseen images and domains without task-specific fine-tuning, through a unified promptable interface. SAM accepts spatial prompts in multiple modalities, including points, boxes, and masks, and produces high-quality segmentation outputs directly at inference time. This flexibility has catalyzed widespread adoption across medical imaging [ 66 ] , remote sensing [ 47 ] , robotics [ 64 ] , and content creation. Fig. 1 : Iterative prompt-based interactive segmentation using SAM. (a) In the interactive loop, SAM receives an input image and a set of user-provided point prompts (positive/inclusion and negative/exclusion) and returns a segmentation mask. A human expert compares the predicted mask against the desired target segmentation and provides additional corrective prompts, which are fed back to SAM in the next iteration. (b) Prompt accumulation and mask evolution across iterations: the left panels show the prompt set at iterations t=0,1,2 , and the right panels show the corresponding SAM outputs, demonstrating error correction and progressive convergence to the desired object mask. The success of promptable foundation models has naturally motivated investigation into optimal prompting strategies. Extensive research has explored automated prompt generation [ 33 ] , few-shot prompting techniques [ 19 ] , and reinforcement learning approaches for adaptive refinement [ 21 ] . However, these techniques focus on automating prompting through zero-shot (without task specific training example) or one-shot (with one task specific training example) strategies that minimize or eliminate human involvement. This emphasis, while promising for large-scale dataset creation, fundamentally mischaracterizes the way humans use interactive segmentation systems. Humans do not generate a fixed prompt set and passively evaluate results. They observe model outputs, identify failure modes, and strategically place additional prompts to resolve ambiguities [ 58 ] . Each prompt represents a response to the model’s current understanding, creating a feedback loop where the model segments, the human evaluates, the human prompts, and the model re-segments. This cycle continues until the segmentation meets the user’s quality threshold. Figure 1 illustrates an iterative SAM prompting sequence on a pigeon, where a human incrementally guides the model by adding prompts, inspecting the predicted mask, and correcting errors over successive rounds until the segmentation is satisfactory. Here, at t=0 a single inclusion (positive) point prompt produces an incomplete mask that captures only the pigeon’s tail; at t=1 the user adds another inclusion prompt to encourage the full bird, but the mask overshoots and incorrectly includes background (the black railing); at t=2 the user adds an exclusion (negative) prompt on the wrongly included region to mark it as background, and SAM then suppresses that area and outputs a clean segmentation of the pigeon. The PointPrompt dataset [ 43 ] is a large-scale benchmark of point-based visual prompting for interactive segmentation with SAM, created to fill the lack of publicly available datasets for systematically studying such human prompting strategies across diverse vision domains. In models like SAM, prompts are not just inputs but part of an iterative human model dialogue, and we still do not have principled ways to characterize prompt quality in terms of how much a prompt improves the mask, reduces uncertainty, or contributes useful information to subsequent interactions. In this paper, we formalize interactive iterative prompting in SAM as active prompting . Given a model and an unlabeled data pool, active learning asks: which examples should we query for labels to maximize model improvement under a limited annotation budget [ 54 ] ? Canonical approaches score unlabeled samples using uncertainty [ 18 ] , diversity [ 53 ] , or hybrid criteria [ 5 ] . Importantly, even in classical pool-based active learning, informativeness is not static: after each query, the labeled set \mathcal{D}_{t} changes, the model (or posterior) is updated, and acquisition scores must typically be recomputed for the remaining pool. We transpose this sequential selection perspective to interactive segmentation by treating candidate spatial locations within an image as the unlabeled pool [ 6 ] and user prompts as queries. The key insight is that not all prompts contribute equally to segmentation quality: some