BiPrompt-SAM: Enhancing Image Segmentation via Explicit Selection between Point and Text Prompts

Suzhe Xu, Jialin PengChengyuan ZhangHuaqiao University · 2025

Read BiPrompt-SAM: Enhancing Image Segmentation via Explicit Selection between Point and Text Prompts by Suzhe Xu, Jialin PengChengyuan ZhangHuaqiao University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-19769.txt.

Introduction

Suzhe Xu, Jialin Peng, and Chengyuan Zhang wrote a 2025 paper on a two-cue way to cut an object out of a picture, segmentation. One cue is a point: a known tool turns one tap into a set of mask candidates, sharp on location but blind to meaning. One cue is text: a twin tool reads the words and makes a meaning-rich mask. How to combine the two was the question.

Their framework lets the text mask pick the point mask it fits best, by how much the two masks overlap. No major rebuild of the tools is needed. On a medical scan set, one point per object reached top scores, with no domain-specific training. That cuts the annotation work a clinic must do. On word-cue sets the framework beat past methods by clear gaps.

Catalog record

CREATORSuzhe Xu, Jialin PengChengyuan ZhangHuaqiao University
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-19769

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

BiPrompt-SAM: Enhancing Image Segmentation via Explicit Selection between Point and Text Prompts Thanks: Citation : Xu, S., Peng, J., Zhang, C. BiPrompt-SAM: Enhancing Image Segmentation via Explicit Selection between Point and Text Prompts. Suzhe Xu, Jialin PengChengyuan ZhangHuaqiao University Abstract Segmentation is a fundamental task in computer vision, with prompt-driven methods gaining prominence due to their flexibility. The Segment Anything Model (SAM) excels at point-prompted segmentation, while text-based models, often leveraging powerful multimodal encoders like BEIT-3, provide rich semantic understanding. However, effectively combining these complementary modalities remains a challenge. This paper introduces BiPrompt-SAM, a novel dual-modal prompt segmentation framework employing an explicit selection mechanism. We leverage SAM’s ability to generate multiple mask candidates from a single point prompt and use a text-guided mask (generated via EVF-SAM with BEIT-3) to select the point-generated mask that best aligns spatially, measured by Intersection over Union (IoU). This approach, interpretable as a simplified Mixture of Experts (MoE), effectively fuses spatial precision and semantic context without complex model modifications. Notably, our method achieves strong zero-shot performance on the Endovis17 medical dataset (89.55% mDice, 81.46% mIoU) using only a single point prompt per instance. This significantly reduces annotation burden compared to bounding boxes and aligns better with practical clinical workflows, demonstrating the method’s effectiveness without domain-specific training. On the RefCOCO series, BiPrompt-SAM attained 87.1%, 86.5%, and 85.8% IoU, significantly outperforming existing approaches. Experiments show BiPrompt-SAM excels in scenarios requiring both spatial accuracy and semantic disambiguation, offering a simple, effective, and interpretable perspective on multi-modal prompt fusion. K eywords Image Segmentation \cdot Prompt Engineering \cdot Multi-modal Fusion \cdot Medical Imaging \cdot Computer Vision 1 Introduction Image segmentation is a fundamental task in computer vision, crucial for numerous applications including medical imaging, autonomous driving, and augmented reality. With the release of the Segment Anything Model (SAM) [ 1 ] by Meta AI Research, prompt-based segmentation methods have made significant strides. SAM demonstrated remarkable zero-shot segmentation capabilities through large-scale pretraining and prompt engineering, particularly excelling with spatial prompts such as points and bounding boxes. However, SAM primarily relies on spatial prompts (e.g., points, bounding boxes) and exhibits limited semantic understanding capabilities. This limitation becomes apparent in specialized domains like medical imaging or tasks requiring precise semantic comprehension. Conversely, text-based segmentation models such as EVF-SAM [ 2 ] provide rich semantic understanding but may lack the spatial precision offered by direct point prompts, potentially struggling with fine-grained localization. The current research challenge lies in effectively combining the complementary strengths of spatial prompts (points for precision) and semantic prompts (text for context). While methods exist, they often involve complex strategies like feature-level fusion (e.g., concatenating or element-wise operations on intermediate feature maps from different modality encoders) or joint training, which can be computationally demanding and intricate to implement. Feature-level fusion, while potentially powerful, can obscure the distinct contributions of each modality and may require significant architectural changes. Furthermore, these methods may not fully leverage the inherent ambiguity-handling mechanism of SAM (its multi-mask output) to explicitly resolve conflicts or enhance segmentation quality by harnessing the complementarity between spatial and semantic cues. Our approach focuses on leveraging this complementarity at the decision level rather than the feature level. To address these challenges, we propose BiPrompt-SAM, a dual-modal prompt segmentation framework based on an explicit selection mechanism. Our approach leverages the distinct strengths of each modality without complex fusion: SAM generates multiple spatially precise candidate masks from a point prompt, while EVF-SAM provides a single semantically relevant mask from a text prompt. The core idea is to use the text-guided mask to disambiguate and select the best among the point-generated candidates. We compute the spatial similarity, specifically Intersection over Union (IoU), between the text-generated mask ( M^{t} ) and each point-generated mask ( M^{p}_{i} ). IoU is chosen as it directly measures spatial overlap and alignment, which is crucial for comparing segmentation masks, unlike metrics like cosine similarity that primarily capture shape orientation rather than positional accuracy. The point-generated mask with the highest IoU score is selected as the final output. This explicit selection harnesses the complementarity of the prompts – text resolves semantic ambiguity inherent in point prompts (e.g., distinguishing between overlapping objects), while the point ensures spatial accuracy that text alone might lack. This principle of leveraging modality complementarity to resolve ambiguity is a central theme of our work. Figure 1: Overview of the BiPrompt-SAM framework. Our method leverages point prompts to generate multiple candidate masks, while simultaneously using text prompts to generate a semantically guided mask, then selects the best point-generated mask based on similarity as the final output. This explicit dual-selection approach can be viewed as a simplified Mixture of Experts (MoE) system. The point module (SAM) and text module (EVF-SAM) act as distinct "experts," and the IoU-based selection acts as a non-learnable "gating mechanism." This design is simple to implement, preserving the original architectures and capabilities (including the computational characteristics of the base SAM model) of SAM and EVF-SAM. While inference requires running both the point and text branches, the selection mechanism itself adds minimal computational overhead compared to complex fusion techniques or joint training strategies. Therefore, the efficiency largely depends on the underlying SAM and EVF-SAM models, with our contribution being a lightweight method for combining their outputs effectively. It utilizes the complementarity of the two modalities from a novel perspective. Our main contributions can be summarized as follows: • We propose BiPrompt-SAM, a novel dual-modal prompt segmentation framework that combines the advantages of point and text prompts through an explicit selection mechanism, without requiring complex feature fusion or model modifications. • We theoretically demonstrate the connection between our explicit dual-selection method and Mixture of Experts systems, providing a new theoretical perspective for multi-modal prompt fusion. • We conduct extensive evaluations on the Endovis17 medical dataset, where BiPrompt-SAM achieves 89.55% mDice and 81.46% mIoU, comparable to state-of-the-art specialized medical segmentation models. • On the RefCOCO series natural image datasets, our method attains 87.1%, 86.5%, and 85.8% IoU, significantly outperforming existing methods, demonstrating its broad applicability in natural image segmentation. • Through detailed ablation studies, we validate the impact of different text prompts and fusion strategies, analyzing the method’s applicability across various scenarios. Our research not only provides a simple yet effective implementation for multi-modal prompt fusion but also offers new insights into understanding and utilizing the complementarity of different modal prompts, potentially facilitating the development of more practical applications. 2 Related Work 2.1 Prompt-Driven Image Segmentation The field of image segmentation has witnessed a paradigm shift in recent years, moving from fully automatic segmentation to prompt-driven approaches. Prompt-driven segmentation allows users to guide the segmentation process through simple interactions, enhancing flexibility and adaptability. Point-Driven Segmentation. Points represent one of the most intuitive forms of interaction. Early works such as [ 3 ] utilized deep learning to process point prompts. The Segment Anything Model (SAM) [ 1 ] further advanced point-driven segmentation with its innovative ambiguity-aware design, capable of generating multiple reasonable segmentation candidates for a single point click and providing confidence scores for each candidate. This feature allows SAM to handle the inherent ambiguity of point locations, such as when a point falls on overlapping objects. However, SAM primarily relies on spatial prompts and may struggle to accurately

Read the complete text