CLIP-Guided SAM: Parameter-Efficient Semantic Conditioning for Promptable Segmentation

Shayan Jalilian, Abdul Bais · 2026

Read CLIP-Guided SAM: Parameter-Efficient Semantic Conditioning for Promptable Segmentation by Shayan Jalilian, Abdul Bais (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-24807.txt.

Introduction

Shayan Jalilian and Abdul Bais build CLIP-Guided SAM, a segmentation framework with meaning built into the image encoder. Promptable models like the Segment Anything Model draw strong masks but stay semantically blind, needing external prompts for categories. Older vision-language methods make spatial prompts for SAM in a separate step. Instead, CLIP-derived text, vision, and similarity features pass through lightweight adapters, letting meaning guide mask prediction. The promptable interface stays intact, and tests show results hinge on train-test prompt consistency.

Catalog record

CREATORShayan Jalilian, Abdul Bais
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35502
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.24807

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CLIP-Guided SAM: Parameter-Efficient Semantic Conditioning for Promptable Segmentation Shayan Jalilian, Abdul Bais Abstract Promptable foundation models such as the Segment Anything Model (SAM) produce high-quality masks but remain semantically blind, relying on external prompts to specify categories. Existing vision–language approaches address this limitation by using external prompt coupling, in which a vision–language model generates spatial prompts for SAM as a separate stage. We propose CLIP-Guided SAM , a parameter-efficient segmentation framework built on internal semantic conditioning. Instead of using semantic signals only to generate prompts, we inject CLIP-derived text, vision, and similarity features directly into SAM’s image encoder via lightweight multi-modal semantic adapters. These adapters condition SAM’s internal feature representations, allowing semantic information to influence mask prediction while preserving SAM’s original promptable interface. Our framework is designed for low labelled-data settings and applies to both general-domain benchmarks and specialized downstream tasks. It supports two operating modes: Manual mode, for interactive segmentation with both text and spatial prompts, and Semi-Automatic (text-only) mode, for applications that require concept-specific segmentation using only textual input. We show that robustness depends on aligning training with the type of prompts used at inference, making train–test prompt consistency an important design principle. Through extensive experiments and ablations, we evaluate our method against SAM+PEFT baselines without semantic conditioning, vision–language + SAM pipelines, SAM 3, and strong semi-supervised segmentation methods that rely on large amounts of unlabelled data. Across these settings, CLIP-Guided SAM consistently achieves superior or competitive performance while remaining parameter-efficient in both training and deployment. 1 Introduction Prompt-based segmentation, popularized by the Segment Anything Model (SAM) [ 15 ] , has significantly expanded the flexibility of visual localization. With minimal spatial input—such as a point, box, or mask—SAM can produce high-quality object masks across diverse scenes. However, SAM is intentionally class-agnostic : it excels at grouping visual structures but does not encode semantic category information. As a result, identifying what to segment remains an external problem. Furthermore, SAM is sometimes adapted via Parameter-Efficient Fine-Tuning (PEFT) methods for specific tasks to enhance performance and address label scarcity or computational constraints. Vision–language models (VLMs) such as Contrastive Language–Image Pre-training (CLIP) [ 28 ] provide complementary strengths, offering semantic representations aligned with natural language but lacking the spatial precision required for dense segmentation. This complementarity has motivated VLM+SAM systems based on external prompt coupling , where a VLM generates spatial prompts—points, boxes, or masks—that are provided to a frozen or lightly adapted SAM. While effective in some settings, this stage-wise and shallow integration keeps semantic reasoning and spatial segmentation architecturally separate. Semantic information influences SAM only indirectly through sparse or noisy prompts, leaving its internal feature representations semantically uninformed. This limitation is even more pronounced in low-label regimes and text-only scenarios, where supervision is limited and spatial cues are suboptimal. Figure 1 illustrates the difference between existing approaches—such as SAM with PEFT or training-free VLM+SAM pipelines—and our proposed framework, which injects semantic signals directly into SAM’s internal representations. Figure 1 : CLIP-Guided SAM Overview. Comparison of (top) SAM with PEFT , (middle) typical VLM+SAM pipelines , and (bottom) our CLIP-Guided SAM . Existing approaches either rely on spatial prompting alone or couple a VLM to SAM externally via prompt generation, leaving SAM’s internal representations semantically uninformed. Our approach introduces internal semantic conditioning by injecting CLIP-derived features directly into SAM’s image encoder through multi-modal adapters, enabling joint SAM–CLIP co-adaptation. In this work, we propose CLIP-Guided SAM , a parameter-efficient framework that unifies internal semantic conditioning with external spatial prompting . Rather than relying solely on prompt-level guidance, we inject CLIP-derived text, vision, and similarity features directly into SAM’s image encoder via semantic adapters. This design allows semantic information to influence feature formation during segmentation, while spatial prompts remain complementary for boundary refinement and localization. A central component of our framework is the joint co-adaptation of SAM and CLIP. Through systematic ablations, we show that effective semantic conditioning requires optimizing CLIP not merely as a prompt generator, but as an internal semantic guide tailored to SAM’s feature space. Freezing CLIP or pre-fine-tuning it independently beforehand to improve spatial prompt quality leads to inferior performance, whereas end-to-end co-adapting both models enables CLIP to learn to produce semantic signals useful for segmentation and better aligned with SAM’s feature space. We further observe asymmetric sensitivity in tuning budgets: performance saturates quickly with additional SAM backbone tuning, whereas increasing the number of trainable parameters in CLIP produces larger gains. CLIP-Guided SAM is designed to support multiple deployment scenarios. In an interactive mode, the framework combines text prompts with user-provided point inputs, modelling user-in-the-loop segmentation or annotation workflows. In a text-only (semi-automatic) mode, segmentation is driven solely by class descriptions, without human spatial input. In this setting, spatial prompts serve as model-generated inputs rather than supervision. We therefore introduce a train-test-prompt-aligned training strategy that trains directly on noisy prompts sampled from CLIP similarity maps. This alignment mitigates the train–test mismatch introduced by idealized ground-truth clicks and yields substantially more robust inference under text-only prompting. We evaluate CLIP-Guided SAM on general semantic segmentation benchmarks (COCO, ADE20K, PASCAL VOC) and specialized downstream tasks such as camouflaged object detection. Under limited supervision, our method consistently outperforms SAM-based parameter-efficient baselines and achieves competitive performance relative to heavier vision–language systems. With only a few hundred labelled images, our modular co-adaptation framework approaches the zero-shot performance of large-scale models such as SAM 3, while using substantially fewer parameters and enabling efficient tuning on commodity GPUs. Contributions. In summary, we make the following contributions: • We introduce a framework for internal semantic conditioning of SAM, shifting VLM+SAM integration from external prompt coupling to joint feature-level co-adaptation. • We analyze co-adaptation dynamics between SAM and CLIP, showing that adapting the semantic guide is critical, whereas additional tuning of the SAM backbone yields diminishing returns. • We propose a train-test prompt alignment strategy for text-only segmentation that improves robustness by matching training and inference prompt distributions. • We present a parameter-efficient system applicable to both general and specialized segmentation tasks, supporting interactive (text + spatial prompts) and text-only deployment modes under low-label and modest-compute settings. • We demonstrate strong performance on different comparisons and experiments, along with comprehensive ablations, highlighting an efficient and strong alternative to SAM-PEFT baselines, VLM+SAM pipelines, large foundation models, and even strong semi-supervised methods in certain contexts. • We demonstrate strong empirical performance across diverse comparisons and ablations, providing an efficient and competitive alternative to SAM-PEFT baselines, VLM+SAM pipelines, large segmentation foundation models, and, in certain regimes, semi-supervised methods. 2 Related Works 2.1 SAM and Parameter-Efficient Fine-Tuning SAM is a powerful prompt-based segmentation foundation model, but often requires adaptation for specialized downstream tasks [ 5 ] . Because full fine-tuning is costly in low-label regimes, parameter-efficient fine-tuning (PEFT) methods such as adapters [ 11 ] , LoRA [ 13 ] , and prompt tuning [ 19 ] are widely used. Prior work adapts SAM via lightweight modules or modified heads, e.g., SAM-Adapter [ 4 ] , Conv-Meets-LoRA [ 42 ] , SU-SAM [ 32 ] , and TS-SAM [ 40 ] , while others provide light semantic conditioning through text embeddings, e.g., SAM-PTx [ 14 ] . In contrast, we keep SAM’s prompt encoder and mask decoder intact and inject CLIP-derived text,

Read the complete text