FastSeg: Efficient Training-Free Open-Vocabulary Segmentation via Hierarchical Attention Refinement Method Journal: Information fusion

Quang-Huy Che, Vinh-Tiep Nguyen · 2025

Read FastSeg: Efficient Training-Free Open-Vocabulary Segmentation via Hierarchical Attention Refinement Method Journal: Information fusion by Quang-Huy Che, Vinh-Tiep Nguyen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-23323.txt.

Introduction

Quang-Huy Che and Vinh-Tiep Nguyen present a 2025 method for labeling every pixel in an image from any text category a user names. No dense labeled training set is needed, since the method needs no training at all. Contrast based models can do this task but often lose fine detail at the pixel level. Diffusion models keep that detail through their attention maps, yet they can be slow or uneven in quality.

Their method runs in just two steps from a pretrained diffusion model and handles all classes in one pass. Three added parts sharpen the result: a dual prompt plan, a layered attention refinement, and a flip test for consistency. Across three standard benchmarks, the method reached the best scores among training free approaches. It also kept inference fast, the authors report.

Catalog record

CREATORQuang-Huy Che, Vinh-Tiep Nguyen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-23323

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FastSeg: Efficient Training-Free Open-Vocabulary Segmentation via Hierarchical Attention Refinement Method Journal: Information fusion Quang-Huy Che, Vinh-Tiep Nguyen Abstract Open-vocabulary semantic segmentation (OVSS) aims to segment objects from arbitrary text categories without requiring densely annotated datasets. Although contrastive learning based models enable zero-shot segmentation, they often lose fine spatial precision at pixel level, due to global representation bias. In contrast, diffusion-based models naturally encode fine-grained spatial features via attention mechanisms that capture both global context and local details. However, they often face challenges in balancing the number of iterations with the quality of the segmentation. In this work, we propose FastSeg, a novel and efficient training-free framework with only (1+1)-step of reverse process of a pretrained diffusion model (e.g., Stable Diffusion). Moreover, instead of running multiple times for different classes, FastSeg performs segmentation for all classes at once. To further enhance the segmentation quality, FastSeg introduces three key components: (i) a dual-prompt mechanism for discriminative, class-aware attention extraction, (ii) a Hierarchical Attention Refinement Method (HARD) that enhances fused cross-attention using scale-aligned self-attention maps, and (iii) a Test-Time Flipping (TTF) scheme designed to improve spatial consistency. Extensive experiments show that FastSeg achieves state-of-the-art training-free performance, obtaining 43.8% average mIoU across PASCAL VOC, PASCAL Context, and COCO Object benchmarks while maintaining superior inference efficiency. Our results demonstrate that FastSeg provides a strong foundation for extendability, bridging the gap between segmentation quality and inference efficiency. 1 Introduction Traditional semantic segmentation models rely heavily on fully supervised training using densely annotated datasets. While effective within a fixed set of predefined categories, these models struggle to scale to the vast diversity of visual concepts encountered in real-world settings due to labor-intensive annotated datasets. To address this limitation, Open-Vocabulary Semantic Segmentation (OVSS) has emerged as a compelling alternative, aiming to segment images based on arbitrary textual inputs without requiring manual annotations for every possible class. Figure 1 : Open-vocabulary semantic segmentation results using FastSeg. We visualize predictions on challenging scenarios, including contextual images, camouflaged objects, and internet-collected images. The highlighted results demonstrate that, in addition to accurate segmentation of both stuff and things, our method effectively handles camouflaged objects, distinguishes between different characters or dog breeds, and differentiates objects based on color variations. Existing OVSS methods primarily follow two dominant paradigms: vision-language models and diffusion models. Vision-language models such as CLIP [ 1 ] leverage contrastive learning to align visual and textual representations within a shared embedding space. Approaches based on CLIP can be broadly categorized into training-free methods [ 2 , 3 , 4 , 5 ] , which directly utilize pre-trained models, and training methods [ 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 , 14 ] , which improve multimodal alignment through additional supervision. However, these models often suffer from limited spatial precision and struggle to accurately capture object boundaries due to their reliance on global image-text embeddings. In contrast, diffusion models [ 15 , 16 , 17 , 18 , 19 , 20 , 21 ] have recently emerged as a powerful alternative, as their generative attention mechanisms inherently preserve spatial structures, enabling them to capture both object location and shape with high fidelity. Nevertheless, many diffusion-based methods require extensive supervised training and computationally expensive synthetic dataset generation, limiting their scalability and practical deployment. To address these challenges, recent research has explored training-free diffusion-based OVSS approaches that eliminate the need for supervision or retraining. While promising, existing methods often trade inference efficiency for segmentation quality: some require repeated inference for each candidate class, while others involve time-consuming optimization or prompt tuning steps. This creates a critical gap between segmentation accuracy and computational efficiency, particularly for real-world open-vocabulary scenarios. These limitations reveal a critical open question: How can training-free diffusion-based OVSS methods achieve high segmentation quality while remaining computationally efficient enough for practical deployment? In this paper, we propose FastSeg, an efficient zero-shot framework for open-vocabulary semantic segmentation (OVSS) that leverages text-to-image diffusion models, as shown in Figure 1 . FastSeg balances high segmentation quality with computational efficiency through key innovations. It performs image reconstruction using DDIM Inversion, enabling the simultaneous extraction of both self-attention and cross-attention maps across all candidate classes. A dual-prompt mechanism—a text prompt for reconstruction and a class prompt for generating class-specific cross-attention maps—enables multi-class segmentation without repeated inference. Additionally, FastSeg introduces the H ierarchical A ttention R efinement Metho d ( HARD ), which aggregates attention maps from multiple scales, combining fine details from higher resolutions and broader context from lower resolutions to improve localization, shape accuracy, and boundary delineation. Finally, the Test-Time Flipping (TTF) strategy enhances segmentation accuracy by averaging predictions from both the original and flipped images, improving spatial consistency and robustness. We summarize our key contributions as follows: 1. We introduce FastSeg, a novel training-free open-vocabulary segmentation method that leverages diffusion models with fast (1+1)-step DDIM inversion and a unified class prompt, enabling efficient attention map generation for all candidate classes in a single inference run. 2. To enhance segmentation accuracy, we propose three key techniques: (i) a dual-prompt mechanism that separately guides semantic reconstruction and class-specific attention extraction, (ii) HARD proposed that improves fused cross-attention maps using transformed self-attention maps, and (iii) TTF that improves the spatial consistency of the segmentation mask. 3. We summarize our key contributions as follows: We conduct comprehensive experiments on three benchmark datasets—PASCAL VOC, PASCAL Context, and COCO Object—demonstrating that FastSeg achieves state-of-the-art performance among training-free methods while maintaining superior computational efficiency. 2 Related work 2.1 Contrastive Vision-Language Models for OVSS While self-supervised learning methods such as MoCo [ 22 ] and DINO [ 23 ] have shown success in learning visual features, they struggle with flexible semantic understanding for unseen categories. Large-scale vision–language models like CLIP [ 1 ] address this limitation by aligning visual and textual features in a shared embedding space, making them suitable for open-vocabulary semantic segmentation (OVSS). Several recent training-free methods directly exploit CLIP’s inherent localization capability by extracting or refining attention maps for segmentation. MaskCLIP [ 24 ] utilizes value embeddings from CLIP’s final self-attention layer for label-free segmentation. SCLIP [ 4 ] and CLIPSurgery [ 25 ] modify the query-key attention mechanism via self-self attention to better focus on semantic regions, while ClearCLIP [ 5 ] simplifies CLIP’s final transformer layer to improve fine-grained localization. Other works, such as ReCo [ 2 ] , leverage CLIP’s retrieval capability and the alignment between visual features and category semantics. Additionally, CLIP-DIY [ 3 ] combines unsupervised object localization with CLIP to generate foreground-background guidance maps. In contrast, some unsupervised training approaches learn segmentation models from image–text pairs without relying on pixel-level labels. GroupViT [ 26 ] , SegCLIP [ 11 ] , and OVSegmentor [ 12 ] introduce group tokens into the model backbone to learn semantic clusters aligned with textual concepts. Specifically, GroupViT treats group tokens as class proxies from text supervision, SegCLIP combines CLIP features with learnable tokens, while OVSegmentor introduces masked entity completion and cross-image mask consistency to improve segmentation. ZeroSeg [ 9 ] distills CLIP’s visual concepts into learnable segmentation tokens for localizing semantic regions, and ViewCo [ 10 ] enforces multi-view consistency across augmented views. TCL [ 7 ] leverages masked regions to

Read the complete text