N erve : N eighbourhood & E ntropy-guided R andom-walk for training free open- V ocabulary s E gmentation

Kunal Mahatha Jose DolzChristian Desrosiers[1em] LIVIA, ÉTS Montréal, CanadaInternational Laboratory on Learning Systems (ILLS),McGILL - ETS - MILA - CNRS - Université Paris-Saclay - CentraleSupélec, Canada🖂 [email protected][0.5em] · 2025

Read N erve : N eighbourhood & E ntropy-guided R andom-walk for training free open- V ocabulary s E gmentation by Kunal Mahatha Jose DolzChristian Desrosiers[1em] LIVIA, ÉTS Montréal, CanadaInternational Laboratory on Learning Systems (ILLS),McGILL - ETS - MILA - CNRS - Université Paris-Saclay - CentraleSupélec, Canada🖂 [email protected][0.5em] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-08248.txt.

Introduction

Kunal Mahatha, Jose Dolz, and Christian Desrosiers present NERVE, a training free method for open vocabulary semantic segmentation, labeling image regions for categories named in text. Existing training free methods often need costly affinity refinement or smooth attention maps with fixed kernels that ignore object shape. NERVE instead uses the neighbourhood structure inside a stable diffusion model self attention layer.

A stochastic random walk refines affinities, spreading labels across connected, related areas of any shape. Entropy based uncertainty selects which attention maps from different heads and layers deserve weight, instead of treating all equally. The method needs no standard post processing and reaches top zero shot results on seven benchmarks.

Catalog record

CREATORKunal Mahatha Jose DolzChristian Desrosiers[1em] LIVIA, ÉTS Montréal, CanadaInternational Laboratory on Learning Systems (ILLS),McGILL - ETS - MILA - CNRS - Université Paris-Saclay - CentraleSupélec, Canada🖂 [email protected][0.5em]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-08248

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

N erve : N eighbourhood & E ntropy-guided R andom-walk for training free open- V ocabulary s E gmentation Kunal Mahatha Jose DolzChristian Desrosiers[1em] LIVIA, ÉTS Montréal, CanadaInternational Laboratory on Learning Systems (ILLS),McGILL - ETS - MILA - CNRS - Université Paris-Saclay - CentraleSupélec, Canada🖂 [email protected][0.5em] Abstract Despite recent advances in Open-Vocabulary Semantic Segmentation (OVSS), existing training-free methods face several limitations: use of computationally expensive affinity refinement strategies, ineffective fusion of transformer attention maps due to equal weighting or reliance on fixed-size Gaussian kernels to reinforce local spatial smoothness, enforcing isotropic neighborhoods. We propose a strong baseline for training-free OVSS termed as NERVE (Neighbourhood & Entropy-guided Random-walk for open-Vocabulary sEgmentation), which uniquely integrates global and fine-grained local information, exploiting the neighbourhood structure from the self-attention layer of a stable diffusion model. We also introduce a stochastic random walk for refining the affinity rather than relying on fixed-size Gaussian kernels for local context. This spatial diffusion process encourages propagation across connected and semantically related areas, enabling it to effectively delineate objects with arbitrary shapes. Whereas most existing approaches treat self-attention maps from different transformer heads or layers equally, our method uses entropy-based uncertainty to select the most relevant maps. Notably, our method does not require any conventional post-processing techniques like Conditional Random Fields (CRF) or Pixel-Adaptive Mask Refinement (PAMR). Experiments are performed on 7 popular semantic segmentation benchmarks, yielding an overall state-of-the-art zero-shot segmentation performance, providing an effective approach to open-vocabulary semantic segmentation. 1 Introduction Deep learning models have made significant strides in dense image prediction tasks, particularly in semantic segmentation [ 29 ] . However, their reliance on a fixed set of predefined classes limits their applicability in numerous real-world scenarios where the categories of interest are often unknown or variable. To address this limitation, open-vocabulary semantic segmentation (OVSS) has recently emerged as a promising alternative [ 4 , 58 , 60 ] . OVSS enables models to recognize and segment novel categories that were not encountered during training, offering greater flexibility and generalization. 1 1 1 Our code is publicly available at: https://github.com/kunal-mahatha/nerve/ Most OVSS methods are built on vision-language models, particularly Contrastive Language-Image Pre-training (CLIP) [ 33 ] , which has demonstrated strong zero-shot performance in visual recognition. These methods generally fall into two broad categories: training-based [ 9 , 54 , 56 , 1 , 13 , 22 , 32 , 51 , 55 ] and training-free [ 19 , 44 , 60 ] approaches. Methods in the first category typically include a fully-supervised (or weakly-supervised) training step, where pixel-wise (or image-wise) annotations from a limited set of categories are used to transfer language–vision relationships from image level to the pixel level. However, these methods still depend on extensive annotations for related categories, and their performance can be influenced by the choice of training dataset used for adaptation [ 14 ] . In contrast, training-free methods [ 2 , 3 , 10 , 16 , 21 , 28 , 38 , 44 , 47 , 60 , 19 ] do not require access to additional data for adaptation, making them suitable for realistic scenarios where large labeled datasets are scarce and novel classes cannot be anticipated. Several methods leverage the visual representations learned by image-level supervised vision transformers (ViTs) such as CLIP. Some approaches modify the self-attention mechanism—for instance, by incorporating query-query or key-key interactions (also known as self-self attention) [ 3 , 21 , 60 , 44 , 19 ] . Others directly use the value vectors from CLIP as dense features for pixel-level classification [ 60 ] . An alternative line of work employs additional vision encoders that have been pre-trained using supervised [ 31 ] , self-supervised [ 15 ] , or unsupervised [ 40 ] learning objectives. More recently, state-of-the-art performance in OVSS has been achieved by harnessing visual features from generative models such as Stable Diffusion [ 2 , 10 , 16 , 51 , 45 , 42 ] . In this paper, we consider the more challenging scenario of training-free OVSS. Despite the recent advancements in OVSS, current training-free methods for this problem still face notable limitations. First, most of these methods obtain segmentation predictions by combining attention maps from the last ViT layer [ 42 , 14 , 11 , 19 , 44 , 20 ] or from multiple layers with upsampling [ 18 , 41 ] , without taking into account their relative usefulness in the final prediction (e.g., simply averaging the maps). Moreover, methods that rely directly on CLIP’s coarse attention maps typically require an additional segmentation refinement step to address their low spatial resolution [ 44 , 19 , 14 ] . This limitation can be partially mitigated by leveraging the finer-grained attention maps produced by diffusion models [ 42 , 45 ] . Nevertheless, this comes at a substantial computational cost, as it involves operations over a large, dense similarity matrix that models interactions between all pairs of spatial positions. Additionally, since attention maps do not explicitly capture local spatial relationships, existing approaches may struggle to preserve the contextual integrity of individual objects. To address this, [ 14 ] introduced a simple yet effective solution: augmenting attention maps with 2D isotropic Gaussian kernels centered on each patch to increase the attention weight of neighboring regions. While this improves OVSS performance, it assumes that local object context is circular and fixed in size, an assumption that often does not hold in practice. As a result, a separate refinement step is required in post-processing to improve the segmentation. To address these limitations, we introduce N erve , a training-free and computationally efficient open-vocabulary semantic segmentation (OVSS) method that exploits neighborhood structure and entropy-guided random walks. N erve formulates segmentation as a linear stochastic process, where image patches are treated as graph nodes and edge weights determine the transition probabilities between nodes during a random walk. Rather than relying on fixed-size Gaussian kernels to model local object context [ 14 ] , our method promotes locality within the random walk by assigning higher transition probabilities to neighboring regions. This spatial diffusion process encourages propagation across connected and semantically related areas, enabling the random walk to effectively delineate objects with arbitrary shapes. N erve also captures pairwise relationships across the entire image efficiently by leveraging the low-rank structure of the self-attention matrix. In contrast to current approaches that explicitly reconstruct the full attention matrix—resulting in quadratic scaling with the number of patches [ 42 ] —our method scales linearly, significantly accelerating the segmentation refinement process. Finally, whereas most existing approaches treat self-attention maps from different transformer heads or layers equally [ 42 , 14 , 11 , 19 , 44 , 20 , 18 , 41 ] , our method uses entropy-based uncertainty to select the most relevant maps for the random walk. Our main contributions can be summarized as follows: • We propose N erve , a novel training-free method for open-vocabulary semantic segmentation (OVSS) based on neighborhood- and entropy-guided random walks. A key innovation of N erve is its ability to efficiently capture both local and global relationships within an image, enabling faster and more accurate segmentation. By accounting for the uncertainty across self-attention maps during their combination,

Read the complete text