Talk2SAM: Text-Guided Semantic Enhancement for Complex-Shaped Object Segmentation
Luka Vetoshkin, Dmitry Yudin · 2025
Read Talk2SAM: Text-Guided Semantic Enhancement for Complex-Shaped Object Segmentation by Luka Vetoshkin, Dmitry Yudin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-05396.txt.
Introduction
Outlining skinny, tangled objects defeats many image segmentation tools, and Luka Vetoshkin and Dmitry Yudin fix that with words. Their method, Talk2SAM, lets a user name the target in text. A vision-language model turns the phrase into a signal that highlights the matching region. That signal is then mapped into the feature space of a strong segmentation model and fed in as an extra prompt.
The text guidance helps the segmenter grip wires, bicycles, and grids whose thin parts usually vanish. It also lets a user pick one object among several inside the same box. On three benchmarks the method beat its high-quality baseline, gaining up to about six points of overlap and eight of boundary accuracy.
Catalog record
| CREATOR | Luka Vetoshkin, Dmitry Yudin |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-05396
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Talk2SAM: Text-Guided Semantic Enhancement for Complex-Shaped Object Segmentation Luka Vetoshkin, Dmitry Yudin Abstract Segmenting objects with complex shapes, such as wires, bicycles, or structural grids, remains a significant challenge for current segmentation models, including the Segment Anything Model (SAM) and its high-quality variant SAM-HQ. These models often struggle with thin structures and fine boundaries, leading to poor segmentation quality. We propose Talk2SAM , a novel approach that integrates textual guidance to improve segmentation of such challenging objects. The method uses CLIP-based embeddings derived from user-provided text prompts to identify relevant semantic regions, which are then projected into the DINO feature space. These features serve as additional prompts for SAM-HQ, enhancing its ability to focus on the target object. Beyond improving segmentation accuracy, Talk2SAM allows user-controllable segmentation, enabling disambiguation of objects within a single bounding box based on textual input. We evaluate our approach on three benchmarks: BIG, ThinObject5K, and DIS5K. Talk2SAM consistently outperforms SAM-HQ, achieving up to +5.9% IoU and +8.3% boundary IoU improvements. Our results demonstrate that incorporating natural language guidance provides a flexible and effective means for precise object segmentation, particularly in cases where traditional prompt-based methods fail. The source code is available on GitHub: github.com/richlukich/Talk2SAM . Figure 1: Graphical Abstract. The core idea of Talk2SAM is to guide open-world segmentation models using textual queries , enabling precise selection of target objects. This approach overcomes the limitations of prompt-based methods such as SAM or HQ-SAM, where spatial prompts (e.g., boxes) may be insufficient to disambiguate complex object boundaries or distinguish between visually similar objects in close proximity. 1 Introduction Segmenting objects in natural images is a fundamental task in computer vision with widespread applications in robotics, medical imaging, autonomous driving, and image editing. Recent advances in foundation models, such as the Segment Anything Model (SAM) [ 6 ] , have demonstrated remarkable generalization capabilities. However, these models often struggle with objects that possess complex boundaries or thin structures—such as wires, bicycles, and structural grids—where segmentation quality drops significantly. To address these limitations, we introduce Talk2SAM , a novel method that integrates natural language understanding into the segmentation process. As illustrated in Figure 1 , our approach leverages CLIP [ 11 ] embeddings derived from user-provided textual descriptions to guide segmentation toward semantically relevant regions. These embeddings are aligned with the DINO [ 2 ] feature space and injected as prompts into SAM-HQ [ 5 ] , enabling more accurate and controllable segmentations. This language-driven control allows Talk2SAM to better distinguish between overlapping or visually similar objects, overcoming the limitations of traditional prompt types such as points or boxes. Our key contributions are as follows: • We introduce Talk2SAM , a vision-language segmentation method that integrates CLIP, DINO, and SAM-HQ to segment complex-shaped objects by jointly leveraging semantic text prompts and high-resolution visual features. • Unlike traditional prompt-based approaches, Talk2SAM incorporates language-based guidance to localize and disambiguate overlapping or structurally similar objects within a single bounding box. • We conduct extensive experiments on three challenging datasets—BIG [ 4 ] , ThinObject5K [ 8 ] , and DIS5K [ 10 ] —demonstrating that Talk2SAM consistently outperforms SAM-HQ in both standard IoU and boundary IoU (bIoU) metrics. 2 Related Work SAM and its extensions. The Segment Anything Model (SAM) introduced a prompt-based segmentation framework capable of generalizing across a wide range of objects using simple inputs such as points, boxes, or masks. While SAM demonstrates strong generality, it struggles with fine structures and objects requiring precise boundary delineation. Several extensions have been proposed to address these limitations. SAM-HQ enhances boundary accuracy by refining high-frequency visual features, achieving consistently better mask quality than the original SAM. Other variants aim to improve semantic control or task adaptability: VRP-SAM [ 13 ] introduces visual reference prompts for contextual disambiguation; SAM-CLIP [ 14 ] incorporates CLIP features to enable category-aware segmentation; and CAT-SAM [ 15 ] leverages adapter tuning for efficient task-specific customization. These models illustrate the broader effort to enhance SAM’s spatial precision and semantic flexibility. Our approach, Talk2SAM , complements this line of work by introducing language-driven guidance for segmentation. In contrast to methods relying solely on visual prompting, Talk2SAM uses text prompts to inject semantic information, helping to distinguish structurally similar or overlapping objects—especially in cases involving thin or complex shapes. Importantly, Talk2SAM is designed as a modular and model-agnostic enhancement: it can be integrated with any SAM-based variant to improve its performance without architectural modification. In this work, we adopt SAM-HQ as our primary baseline, as it significantly outperforms SAM in boundary-level accuracy. We further show that incorporating text-driven guidance into SAM-HQ yields additional improvements in both IoU and bIoU metrics. Vision-language segmentation with CLIP and DINO. Recent developments in vision-language models have enabled semantically informed segmentation through the use of textual prompts. A common approach involves leveraging CLIP to extract global or localized semantic features, which are then used to guide mask prediction. CLIPSeg [ 9 ] was among the first to demonstrate this idea, conditioning a segmentation decoder on CLIP-derived embeddings from image and text inputs. While effective in zero-shot scenarios, CLIPSeg suffers from limited spatial precision, particularly in cluttered or fine-grained regions. To improve spatial grounding, DenseCLIP [ 12 ] introduces context-aware prompts and dense supervision, improving alignment between language and visual features. Similarly, LSeg [ 7 ] aligns a supervised segmentation backbone with CLIP’s latent space, enabling open-vocabulary prediction, though with limited control over instance-level segmentation. X-Decoder [ 16 ] extends this paradigm by unifying decoding across image and text modalities for a variety of dense prediction tasks, but prioritizes generality over precise structure-aware segmentation. A more recent and closely related method is Talk2DINO [ 1 ] , which bridges CLIP and DINO by projecting text-derived semantics into the high-resolution DINO feature space. This enables semantically controllable detection and forms the foundation for multimodal prompt-based reasoning at fine spatial granularity. We adopt Talk2DINO in our work as a semantic prior to compute dense similarity maps from text prompts. By injecting these maps into SAM-HQ, Talk2SAM extends semantic projection into the segmentation domain, enabling accurate and controllable segmentation of thin and structurally complex objects guided purely by natural language. 3 Method In this section, we present Talk2SAM , a vision-language segmentation method that enhances object segmentation by incorporating textual semantics into the high-quality SAM model (SAM-HQ). The method leverages language as an additional modality to provide semantic guidance for segmentation, particularly useful for fine-structured or ambiguous objects. Figure 2: Overview of the Talk2SAM framework. Given an input image and a user-defined text prompt, CLIP extracts a global text embedding and computes a relevance map with the image. This relevance is projected into DINO’s visual space to produce a dense semantic prompt, which is injected into SAM-HQ’s decoder for refined segmentation. Figure 2 illustrates the overall pipeline. Talk2SAM consists of three main stages: 1. Text-to-Semantic Mapping: The user provides a textual object category (e.g., "wire" or "bicycle" ), rather than a full natural language description. This category is encoded using CLIP’s text encoder to produce a semantic embedding. Following Talk2DINO, we apply a nonlinear learned mapping to project this embedding into the visual space of DINOv2 features. Specifically, the projection \psi(\cdot) is composed of two affine transformations with a