Benchmarking Feature Upsampling Methods for Vision Foundation Models using Interactive Segmentation

Volodymyr Havrylov 1 Haiwen Huang 1,3 Dan Zhang 2 Andreas Geiger 1,3 1 University of Tübingen 2 Bosch Center for Artificial Intelligence 3 Tübingen AI Center · 2025

Read Benchmarking Feature Upsampling Methods for Vision Foundation Models using Interactive Segmentation by Volodymyr Havrylov 1 Haiwen Huang 1,3 Dan Zhang 2 Andreas Geiger 1,3 1 University of Tübingen 2 Bosch Center for Artificial Intelligence 3 Tübingen AI Center (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-02075.txt.

Introduction

Volodymyr Havrylov, Haiwen Huang, Dan Zhang, and Andreas Geiger test ways to sharpen vision model features in a 2025 paper. Giant vision models are trained once and reused for many tasks. But they output coarse feature maps, too rough for tasks that need fine masks. A bolt-on module can raise that resolution without retraining the giant.

The team needed a fair test for such modules. They chose interactive segmentation, where a user clicks on an image and the model draws a mask. The task mixes image and click inputs and demands a fine output, so it stresses scene grasp hard. Trials show the choice of sharpening method shifts feature quality a lot. Their test code is shared in public.

Catalog record

CREATORVolodymyr Havrylov 1 Haiwen Huang 1,3 Dan Zhang 2 Andreas Geiger 1,3 1 University of Tübingen 2 Bosch Center for Artificial Intelligence 3 Tübingen AI Center
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-02075

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Benchmarking Feature Upsampling Methods for Vision Foundation Models using Interactive Segmentation Volodymyr Havrylov 1 Haiwen Huang 1,3 Dan Zhang 2 Andreas Geiger 1,3 1 University of Tübingen 2 Bosch Center for Artificial Intelligence 3 Tübingen AI Center Abstract Vision Foundation Models (VFMs) are large-scale, pre-trained models that serve as general-purpose backbones for various computer vision tasks. As VFMs’ popularity grows, there is an increasing interest in understanding their effectiveness for dense prediction tasks. However, VFMs typically produce low-resolution features, limiting their direct applicability in this context. One way to tackle this limitation is by employing a task-agnostic feature upsampling module that refines VFM features resolution. To assess the effectiveness of this approach, we investigate Interactive Segmentation (IS) as a novel benchmark for evaluating feature upsampling methods on VFMs. Due to its inherent multimodal input, consisting of an image and a set of user-defined clicks, as well as its dense mask output, IS creates a challenging environment that demands comprehensive visual scene understanding. Our benchmarking experiments show that selecting appropriate upsampling strategies significantly improves VFM features quality. The code is released at https://github.com/havrylovv/iSegProbe . Figure 1 : General architecture of a typical IS setup (a) and the proposed benchmark (b, c). The benchmark considers two options for injecting click features. 1 Introduction High-quality representations from pre-trained Vision Foundation Models (VFMs) have become a crucial component of modern computer vision architectures, ensuring robustness and generalizability across various tasks Kirillov et al. 2023 ; Radford et al. 2021 ; Tschannen et al. 2025 ; Yu et al. 2023 ; Ravi et al. 2024 ; Huang et al. 2024 ; Yuan et al. 2024 ; Waldmann et al. 2025 ; Cui et al. 2024 . However, due to the patchification or extensive pooling operations, VFM features typically have a spatial resolution that is 16 or more times smaller than the input image. This limits their effectiveness for dense prediction tasks, where high-resolution reasoning and fine-grained detail preservation are essential. The issue of insufficient feature resolution is commonly addressed by training a task-specific decoder, a multi-layer architecture designed for upsampling Kirillov et al. 2023 ; Li et al. 2022 ; Liu et al. 2023 ; Yu et al. 2023 . Nevertheless, this strategy has several drawbacks, including high computational costs, the need to retrain the decoder for each new task, and potential data scarcity for training a high-quality decoder. A recent and promising alternative is to use a feature upsampler module Wang et al. 2019 ; Kirillov et al. 2020 ; Lu et al. 2022c ; Lu et al. 2022b ; Hu et al. 2022b ; Hu et al. 2022b ; Suri et al. 2024 ; Fu et al. 2024 that optionally conditions on the input image and reconstructs high-resolution features, restoring lost spatial information. As a new trend, these modules are being increasingly designed to generalize across various model configurations and downstream applications Lu et al. 2022b ; Suri et al. 2024 ; Fu et al. 2024 . Integrating feature upsampler modules into existing pipelines has been shown to improve performance and explainability Fu et al. 2024 . Moreover, combining them with a lightweight, task-specific decoder provides an alternative to using a conventional, heavy decoder when computational resources are limited. To properly evaluate feature upsamplers on VFMs, we explore Interactive Segmentation (IS) as a novel benchmark task. IS generates object masks based on user input, where positive and negative clicks define which regions should be included or excluded from the final prediction. We specifically focus on IS due to three key characteristics: its demand for fine-grained semantic understanding, its integration of an additional input modality, and a wide range of practical applications. As a representative of 2D dense prediction tasks Xie et al. 2021 ; Godard et al. 2017 ; Wang et al. 2015 ; Yu et al. 2023 , IS poses inherent challenges for pixel-level scene understanding. Furthermore, an additional input modality in the form of sparse geometric clues (user clicks) imposes specific requirements on the architecture and allows VFMs to be tested in a multimodal regime. This setup is reminiscent of the open-vocabulary segmentation Yu et al. 2023 ; Huang et al. 2024 ; Yuan et al. 2024 , where a textual input, instead of a spatial one, guides the predictions. Finally, IS plays a significant role in Human-Computer Interaction Kohli et al. 2012 ; Amrehn et al. 2019 ; Ramkumar et al. 2017 , with applications including accelerated pixel-level annotation Benenson et al. 2019 ; Marinov et al. 2024 ; Pavoni et al. 2022 , controllable image generation Zhang et al. 2023 ; Rombach et al. 2022 ; Li et al. 2024 , and image editing Brooks et al. 2023 ; Couairon et al. 2023 ; Brack et al. 2024 . Moreover, with the increasing popularity of Segment Anything Model Kirillov et al. 2023 , the interest in IS applications continues to grow. In this work, we systematically explore the design space for a feature upsampling evaluation pipeline on VFMs within the IS framework and propose a concrete architecture for this task. Specifically, we retain the VFM as a backbone, followed by an upsampler and a lightweight segmentation head. Additionally, we define modules for encoding user clicks (hereafter referred to as click encoders) and determine an appropriate strategy for integrating their features with the VFM output. Depending on the complexity of the click encoder, its features are injected into the pipeline either before or after the VFM. Compared to the typical IS model architecture shown in Fig. 1 (a), our approach eliminates multiscale feature extraction via Feature Pyramid Networks (FPN) and maintains a simplified segmentation head. During the experiments, we freeze the VFM and the feature upsampler, optimizing only the click encoder and segmentation head. This configuration significantly reduces both training time and computational requirements. Moreover, it more accurately reflects the effectiveness of feature upsampling. Using this architecture, we benchmark various feature upsampling techniques to identify the most effective methods for enhancing VFM features in this context. The experiments on four standard IS datasets demonstrate that the LoftUp Huang et al. 2025 upsampler yields up to a 50% performance improvement over conventional bilinear interpolation. Furthermore, as shown in Fig. 2 , the upsampled features present improved sharpness, preserving fine-grained details from the input image. Finally, by establishing IS as a benchmark, we emphasize its potential to serve as a structured and effective framework for evaluating models in real-world interactive scenarios. 2 Related Work Interactive Segmentation. IS refers to the task of extracting object masks based on a limited number of user clicks, which serve as a guidance. Before the advent of deep learning, IS is primarily addressed using low-level image features (e.g., intensity) in combination with graph-based optimization techniques Boykov and Jolly 2001 ; Boykov and Kolmogorov 2001 ; Grady 2006 ; Gulshan et al. 2010 ; Rother et al. 2004 . With the rise of deep learning, DIOS Xu et al. 2016 becomes the first method to apply neural networks to IS. DIOS also introduces a sampling strategy for generating subsequent clicks based on the current mask and formalizes training and evaluation protocols that have since become the de facto standard for click-based IS. FCA-Net Lin et al. 2020 highlights

Read the complete text