Similarity-Aware Selective State-Space Modeling for Semantic Correspondence
Seungwook Kim Minsu Cho · 2025
Read Similarity-Aware Selective State-Space Modeling for Semantic Correspondence by Seungwook Kim Minsu Cho (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-24318.txt.
Introduction
Matching the same object parts across two photos is core to vision, yet correlation methods for it can be costly. Seungwook Kim and Minsu Cho built MambaMatcher, which refines the four-dimensional correlation map between images with selective state-space models. Feature methods can miss tangled relations between parts, while full correlation processing is heavy.
Their similarity-aware scan adapts the linear-cost trick of Mamba models to this task. It keeps feature resolution and reach intact while cutting cost. On standard semantic correspondence benchmarks, MambaMatcher reached the top performance. The result shows state-space models can handle high-dimensional visual matching well.
Catalog record
| CREATOR | Seungwook Kim Minsu Cho |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-24318
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Similarity-Aware Selective State-Space Modeling for Semantic Correspondence Seungwook Kim Minsu Cho Abstract Establishing semantic correspondences between images is a fundamental yet challenging task in computer vision. Traditional feature-metric methods enhance visual features but may miss complex inter-correlation relationships, while recent correlation-metric approaches are hindered by high computational costs due to processing 4D correlation maps. We introduce MambaMatcher, a novel method that overcomes these limitations by efficiently modeling high-dimensional correlations using selective state-space models (SSMs). By implementing a similarity-aware selective scan mechanism adapted from Mamba’s linear-complexity algorithm, MambaMatcher refines the 4D correlation map effectively without compromising feature map resolution or receptive field. Experiments on standard semantic correspondence benchmarks demonstrate that MambaMatcher achieves state-of-the-art performance. 1 Introduction Establishing semantic correspondences between images is a fundamental problem in computer vision, with wide-ranging applications in augmented and virtual reality (AR/VR) [ 21 ] , virtual try-on [ 19 ] , edit propagation [ 9 , 45 ] , and instance swapping [ 57 , 44 ] . The task involves identifying semantically corresponding regions between pairs of images depicting different instances of the same class [ 4 , 39 , 52 , 41 ] . Despite significant advancements in deep learning, reliably establishing semantic correspondences remains challenging due to substantial intra-class variations, including differences in pose, scale, and appearance. Current state-of-the-art methods predominantly adopt a feature-metric approach , enhancing the quality of dense features extracted from the images. This enhancement is achieved by either (i) employing powerful feature extractors [ 51 , 57 ] or (ii) improving feature representations through additional convolutional or attention layers [ 49 , 25 , 36 ] . While feature-metric methods can robustly identify semantic similarities across local pixels, they may struggle to capture complex inter-correlation relationships. An alternative is the correlation-metric approach , where the methods aim to model and refine inter-correlation relationships by processing the high-order correlation map constructed using the extracted image features [ 46 , 38 , 5 , 20 ] . Although this approach can alleviate ambiguous or noisy correspondences by considering global correlation patterns, processing the 4D correlation map incurs up to quartic complexity with respect to the feature map dimensions. This severely limits the feature map resolution and necessitates compromises on the receptive field or network expressivity - which are critical factors for accurate and robust correspondences. Consequently, despite their potential, correlation-metric methods are often outperformed by feature-metric methods that utilize stronger backbones and higher-resolution images [ 36 , 15 , 30 , 51 , 57 ] . In this paper, we propose MambaMatcher, a novel approach that overcomes the limitations of both feature-metric and correlation-metric methods by efficiently modeling high-dimensional correlation maps using selective state-space models (SSMs). To the best of our knowledge, MambaMatcher is the first method to treat multi-level correlation scores at each position in the correlation map as a state in a state-space model, enabling effective and efficient modeling of inter-image correlations. At the core of MambaMatcher is a similarity-aware selective scan mechanism, which adapts Mamba’s linear selective scan algorithm to refine the 4D correlation map. This mechanism allows us to robustly and scalably process the correlation map without compromising feature map resolution or receptive field, thereby capturing rich inter-image relationships while maintaining computational efficiency. The key contributions of our work are as follows: • We introduce MambaMatcher, the first method to model high-order correlation maps using selective state-space models by treating multi-level correlation scores at each position as states. • We propose a novel similarity-aware selective scan mechanism, enabling efficient and accurate mining of inter-correlation relationships at high resolutions. • MambaMatcher seamlessly integrates feature-metric and correlation-metric approaches into a unified pipeline, leveraging the strengths of both methods without compromising feature resolution or receptive field. • Extensive experiments demonstrate that MambaMatcher achieves state-of-the-art performance on standard semantic correspondence benchmarks. 2 Related Work Feature-metric semantic correspondence prioritizes producing high-quality features to establish robust correspondences. Traditional methods [ 32 , 2 , 4 , 13 ] typically use handcrafted descriptors [ 35 , 7 , 1 ] which are simple yet effective. Recent methods demonstrate that using local features extracted from deep neural networks leads to significant performance improvements [ 40 , 51 , 36 , 30 ] . While ResNets [ 14 ] were the conventional choice for the visual feature extractor, more recent works propose employing stronger feature extractors such as DINOv2 [ 43 ] or Stable Diffusion [ 47 ] . In the presence of supervision, there are attempts to yield richer features by refining the extracted features, e.g. , by using additional convolutional or attention layers [ 49 , 25 , 18 ] . In our method, we leverage DINOv2 and refine its features using 2D convolutional layers tailored to enhance correspondence accuracy. However, MambaMatcher takes a step further by harmoniously integrating the correlation-metric approach through our proposed similarity-aware selective scan. Correlation-metric semantic correspondence aims to refine ambiguities in the correlation map to establish more robust and accurate correspondences. NCNet [ 46 ] established this idea via a 4D convolutional network to consider neighborhood consensus, which motivated follow-up work to formulate more effective or efficient ways [ 29 , 26 , 22 ] . However, high-dimensional convolutional kernels are constrained by their local receptive field and static transformations. To address this, CATs [ 5 , 6 ] and TransforMatcher [ 20 ] apply self-attention to the correlation maps to consider inter-correlation relations in a dynamic global fashion. However, applying self-attention to the correlation map incurs up to quartic computational complexity with respect to the feature map dimensions. This necessitates a compromise on either the feature map dimensions, the receptive field of inter-correlation relationship mining, or the expressivity of the algorithm used, leading to sub-optimal results. In our work, we introduce a novel approach that models the correlation space using selective state-space models, applying a similarity-aware selective scan to the correlation maps. Building on the efficiency and scalability of Mamba [ 10 ] , this method effectively overcomes previous limitations by avoiding compromises on feature map dimensions, receptive field, or network expressivity. State-space models for computer vision gained rapid interest with the advent of Mamba [ 10 ] , which showed promising results compared to transformer-based architectures for sequence modeling in natural language processing. Notably, Mamba exhibits linear computational complexity at inference, in contrast to attention-based methods that typically have quadratic complexity. VMamba [ 34 ] , Vision Mamba [ 59 ] , and PlainMamba [ 55 ] concurrently propose adapting the selective scan algorithm to the 2D image domain by varying the scan directions to accommodate spatial dimensions. These endeavors show competitive or superior performance compared to existing methods in the computer vision domain, motivating the application of Mamba to various downstream vision tasks [ 28 , 3 , 56 , 48 , 33 ] . In our work, we extend Mamba’s selective scan algorithm by introducing a similarity-aware selective scan specifically designed to refine 4D correlation maps. This adaptation enables us to effectively model the correlation space using selective state-space models, allowing for seamless handling of high-dimensional data in semantic correspondence tasks. Figure 1 : Overview of MambaMatcher . The multi-level correlation map is processed using our proposed similarity-aware selective scan mechanism to yield a