Plug-in Feedback Self-adaptive Attention in CLIP for Training-free Open-Vocabulary Segmentation

Zhixiang Chi, Yanan Wu, Li Gu, Huan Liu, Ziqiang Wang, Yang Zhang, Yang Wang, Konstantinos N Plataniotis, [0.1cm] University of Toronto · 2025

Read Plug-in Feedback Self-adaptive Attention in CLIP for Training-free Open-Vocabulary Segmentation by Zhixiang Chi, Yanan Wu, Li Gu, Huan Liu, Ziqiang Wang, Yang Zhang, Yang Wang, Konstantinos N Plataniotis, [0.1cm] University of Toronto (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-20265.txt.

Introduction

Zhixiang Chi and seven co-authors present a plug-in that sharpens how the CLIP model cuts images into labeled regions, in this 2025 paper. CLIP links images and text well, which lets it segment scenes for words it was never trained on. Its weakness is place: the model's sense of where things are fades before the final output.

The authors feed the model's own output back into its middle layers, since the output holds the richest mix of visual and textual meaning for each patch. Their modules isolate attention, prune low-confidence patches so the feedback stays sparse, and blend several rounds of adaptation. The plug-in needs no training and drops into four leading systems across three backbone sizes. On eight benchmarks it lifted the performance of every host method it joined.

Catalog record

CREATORZhixiang Chi, Yanan Wu, Li Gu, Huan Liu, Ziqiang Wang, Yang Zhang, Yang Wang, Konstantinos N Plataniotis, [0.1cm] University of Toronto
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-20265

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Plug-in Feedback Self-adaptive Attention in CLIP for Training-free Open-Vocabulary Segmentation Zhixiang Chi, Yanan Wu, Li Gu, Huan Liu, Ziqiang Wang, Yang Zhang, Yang Wang, Konstantinos N Plataniotis, [0.1cm] University of Toronto Abstract CLIP exhibits strong visual-textual alignment but struggle with open-vocabulary segmentation due to poor localization. Prior methods enhance spatial coherence by modifying intermediate attention. But, this coherence isn’t consistently propagated to the final output due to subsequent operations such as projections. Additionally, intermediate attention lacks direct interaction with text representations, such semantic discrepancy limits the full potential of CLIP. In this work, we propose a training-free, feedback-driven self-adaptive framework that adapts output-based patch-level correspondences back to the intermediate attention. The output predictions, being the culmination of the model’s processing, encapsulate the most comprehensive visual and textual semantics about each patch. Our approach enhances semantic consistency between internal representations and final predictions by leveraging the model’s outputs as a stronger spatial coherence prior. We design key modules, including attention isolation, confidence-based pruning for sparse adaptation, and adaptation ensemble, to effectively feedback the output coherence cues. Our method functions as a plug-in module, seamlessly integrating into four state-of-the-art approaches with three backbones (ViT-B, ViT-L, ViT-H). We further validate our framework across multiple attention types (Q-K, self-self, and Proxy augmented with MAE, SAM, and DINO). Our approach consistently improves their performance across eight benchmarks. 1 Introduction Open-vocabulary semantic segmentation seeks to localize segments and align visual features with novel class descriptions [ 56 , 6 , 51 ] . CLIP models [ 9 , 40 ] , trained on web-scale datasets [ 41 , 55 ] , have demonstrated remarkable zero-shot performance in open-vocabulary tasks due to strong visual-text alignment [ 61 , 28 , 40 ] . However, adapting CLIP for semantic segmentation remains challenging [ 67 ] since its pretraining primarily contrasts global image-text representations, limiting fine-grained localization and causing spatial misalignment in patch-level features [ 46 , 27 ] . Figure 1 : Comparison with the existing training-free methods. Top: Prior works refine intermediate attention but face limitations: (1) improvements may not propagate to final segmentation; (2) attention lacks direct class information. Bottom: We introduce a feedback-driven self-adaptive mechanism that reintegrates semantic outputs with visual and textual cues into CLIP. Our approach is orthogonal to existing methods and serves as a plug-in. Previous efforts to enhance visual localization have primarily focused on fine-tuning CLIP for segmentation tasks [ 38 , 7 , 54 , 47 , 62 ] . However, fine-tuning compromises CLIP’s robust generalization [ 49 , 67 , 13 , 12 ] . Consequently, recent studies have shifted toward a training-free paradigm, addressing the factors contributing to localization degradation in CLIP. ClearCLIP [ 26 ] identifies residual connections as a major source of noisy segmentation, while SCLIP [ 46 ] highlights the role of self-attention in disrupting spatial information arrangement. Since attention maps capture relationships among patches, patches with similar semantics should be strongly attended to reinforce spatial coherence. Building on this idea, recent works have modulated attention mechanisms, such as replacing Q-K attention with self-self [ 46 , 29 , 2 , 26 ] or Proxy attentions [ 27 ] , both demonstrating improved intermediate spatial consistency. (a) Atn of [ 27 ] (b) Seg of [ 27 ] (c) Atn of [ 27 ] +FSA (d) (e) Seg. of [ 27 ] +FSA Figure 2 : Effectiveness of our feedback self-adaptive mechanism on an image segmented into grass and cat . The attention maps correspond to a reference patch (red dot) on the cat face. In (a), the reference patch misaligns with grass in the attention map, causing incorrect segmentation in (b). Our proposed FSA adjusts (a) using output predictions to generate semantically aligned attention, focusing on cat in (c), and corrects segmentation in (d). However, the final segmentation relies on the output predictions rather than the intermediate attention. Enhanced intermediate coherence may deteriorate due to subsequent operations and hence, might not translate into improved segmentation results. More importantly, intermediate attention does not interface directly with class information. Therefore, neglecting the crucial interplay between intermediate attention and output predictions incorporating text representation is inherently sub-optimal. The output predictions intuitively represent the model’s most refined patch-level semantic understanding, encapsulating high-level visual and textual features. We posit that patches with similar predictions likely belong to the same or related class. Leveraging the semantic coherence extracted from the output prediction can provide valuable feedback for further spatial information rearrangement refinement. Building upon this idea, we propose a training-free F eedback-driven S elf-adaptive A ttention (FSA) framework that adapts output semantic cues back into CLIP through a feedback loop. Specifically, we compute pairwise patch similarities between their class predictions, which provides a stronger prior for guiding intermediate attention than using intermediate representations alone. Modulating the original attention maps with these output-based similarities improves consistency between internal representations and final predictions, enhancing information aggregation among semantically similar patches and improving segmentation accuracy. Fig. 1 shows a high-level comparison with the existing methods. The visual example in Fig. 2 demonstrates that our FSA successfully incorporates the semantic cues at the output to correct the wrongly segmented patches. Our FSA is an unsupervised adaptation method [ 31 , 5 ] that leverages the model’s self-outputs, sharing similarities with learning paradigms such as test-time adaptation (TTA [ 44 , 34 , 43 ] ) and knowledge distillation (KD [ 22 , 64 , 65 , 59 , 57 ] ), as shown in Table 1 . To focus on the modulation only on the initial intermediate attention maps, we introduce attention isolation to ensure that the initial output predictions reflect the influence of the original attention rather than interference from subsequent operations. Additionally, we propose confidence-based pruning to filter out irrelevant patches while enhancing the impact of semantically relevant regions. Furthermore, we design three adaptation strategies and integrate them into an ensemble to achieve consistent improvements across diverse attention configurations and backbones. TTA KD Our feedback mechanism Minimizes entropy using output logits. Treats output logits as soft labels. Computes patch semantic similarity from output logits. Table 1 : Similarity of our FSA with other learning paradigms. To demonstrate the versatility of our FSA framework, we integrate it into four SoTA methods: MaskCLIP [ 67 ] , SCLIP [ 46 ] , ClearCLIP [ 26 ] , and ProxyCLIP [ 27 ] . Additionally, we validate FSA across various attention types, including Q-K, self-self, and Proxy attention, augmented by MAE [ 21 ] , SAM [ 25 ] , and DINO [ 5 , 39 ] . Evaluated on eight open-vocabulary segmentation benchmarks, our FSA consistently enhances performance across these configurations. Our contributions are summarized as: • We introduce a feedback-driven self-adaptive mechanism that refines the attention process by integrating output-derived semantic coherence cues back into the model. • We develop key modules, including attention isolation, sparse attention via confidence-based pruning, and adaptation ensemble, to effectively propagate semantic coherence cues from the output. • Our FSA is training-free, and its adaptation process preserves the original model parameters. • Our FSA functions as a plug-in module that enhances existing methods and attention configurations, demonstrating effectiveness across eight benchmarks. 2 Related work Open-vocabulary segmentation. CLIP’s large-scale

Read the complete text