LoftUp: Learning a Coordinate-Based Feature Upsampler for Vision Foundation Models
Haiwen Huang, Anpei Chen, Volodymyr Havrylov, Andreas Geiger, Dan Zhang · 2025
Read LoftUp: Learning a Coordinate-Based Feature Upsampler for Vision Foundation Models by Haiwen Huang, Anpei Chen, Volodymyr Havrylov, Andreas Geiger, Dan Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-14032.txt.
Introduction
Haiwen Huang and co-authors published this study in 2025. It sharpens the blurry feature maps that vision foundation models produce. Models like DINOv2 and CLIP see images in coarse patches, which limits tasks needing pixel-level detail. Upsampling those features is the fix, and the team names two keys: the upsampler's design and its training target. Their upsampler, LoftUp, is a transformer that reads image coordinates alongside the coarse features.
For training targets, they build high-grade reference features from class-free masks and self-teaching. The result is sharp features that adapt to varied input sizes. Across downstream tasks, LoftUp beat existing upsamplers by clear margins. Code is public. The work matters for any tool that needs a foundation model to point at exact spots, from segmentation to robot grasping.
Catalog record
| CREATOR | Haiwen Huang, Anpei Chen, Volodymyr Havrylov, Andreas Geiger, Dan Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-14032
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LoftUp: Learning a Coordinate-Based Feature Upsampler for Vision Foundation Models Haiwen Huang, Anpei Chen, Volodymyr Havrylov, Andreas Geiger, Dan Zhang Abstract Vision foundation models (VFMs) such as DINOv2 and CLIP have achieved impressive results on various downstream tasks, but their limited feature resolution hampers performance in applications requiring pixel-level understanding. Feature upsampling offers a promising direction to address this challenge. In this work, we identify two critical factors for enhancing feature upsampling: the upsampler architecture and the training objective. For the upsampler architecture, we introduce a coordinate-based cross-attention transformer that integrates the high-resolution images with coordinates and low-resolution VFM features to generate sharp, high-quality features. For the training objective, we propose constructing high-resolution pseudo-groundtruth features by leveraging class-agnostic masks and self-distillation. Our approach effectively captures fine-grained details and adapts flexibly to various input and feature resolutions. Through experiments, we demonstrate that our approach significantly outperforms existing feature upsampling techniques across various downstream tasks. Our code is released at https://github.com/andrehuang/loftup . 1 Introduction High-quality pretrained representations from Vision Foundation Models (VFMs) have become standard for a wide range of computer vision tasks [ 10 , 41 , 62 , 16 , 37 , 56 , 43 , 21 , 45 , 63 , 54 ] . However, because of the patrification or aggressive pooling operations in VFMs, the output features are typically 16 or even more times smaller in spatial resolution than the input images, limiting their utility for tasks that require fine-grained, pixel-level understanding. To address the challenge of limited feature resolution in VFMs, one straightforward approach is to use larger image inputs to obtain higher-resolution features. However, processing high-resolution inputs incurs a quadratic increase in computational cost and can introduce severe artifacts if the VFMs are not trained for such resolutions. Moreover, training or fine-tuning VFMs on high-resolution images demands substantial computational resources and meticulous tuning [ 37 , 43 , 45 , 55 ] . An alternative strategy is to train task-specific decoders that leverage multi-layer intermediate features to upsample VFM outputs [ 27 , 3 , 62 , 16 , 24 ] . Yet, this approach often comes with significant training costs and necessitates retraining the decoder for each new application. In addition, many downstream tasks lack sufficient data needed to fine-tune a high-quality decoder. More recently, FeatUp [ 12 ] and LiFT [ 52 ] have independently introduced task-agnostic feature upsamplers trained with general reconstruction losses, demonstrating that such methods can substantially enhance VFM performance across a variety of tasks. By upsampling VFM features and pairing them with a lightweight task-specific decoder, these approaches provide a promising alternative that avoids heavy, task-specific training while achieving a more generalizable solution. Nonetheless, as shown in Fig. 1 , current feature upsamplers still fall short of reaching optimal performance. Semantic Seg. Depth Estimation Normal Estimation Video Object Seg. (Object Tracking) Open-Voc Seg. Interactive Seg. 61.11 91.35 72.61 60.25 27.82 78.49 53.76 88.71 69.83 44.30 26.61 65.89 LoftUp (ours) Max(FeatUp, LiFT) Backbone only Figure 1 : LoftUp improves significantly across various tasks over the VFM backbone (DINOv2-S [ 37 ] ) and current SoTA feature upsampling performance (FeatUp [ 12 ] and LiFT [ 52 ] ). See experiment details in Sec. 5 . (a) Original image (b) Low res (c) Bilinear (d) Resize-conv (e) LIIF [ 2 ] (f) LiFT [ 52 ] (g) FeatUp [ 12 ] (h) LoftUp (Ours) Figure 2 : Comparison of features from upsamplers. Backbone is DINOv2-S/14 [ 37 ] . In this work, we systematically explore the design space of feature upsamplers and identify two critical components: the upsampler architecture and the training objective. The architecture determines the capacity of the upsampler to learn effectively, while the training objective defines the upper performance limit. By optimizing both elements, our approach achieves substantially stronger results than previous state-of-the-art methods. Regarding the upsampler architecture , to capture high-resolution details while avoiding the cumulative artifacts from multiple layers of interpolation or deconvolution, we propose a simple coordinate-based transformer that directly predicts high-resolution features for each pixel. Specifically, our model takes image coordinates and RGB values as inputs and performs cross-attention with the low-resolution VFM feature map. This facilitates a fine-grained, content-aware mapping from coordinates to high-resolution features, effectively bypassing the constraints imposed by fixed local kernels and standard upsampling layers. Moreover, unlike the implicit approach in FeatUp [ 12 ] , which requires test-time optimization, our method learns feature upsampling directly from the training dataset and generalizes to diverse scenes without additional test-time adjustments. For the training objective , the primary challenge is the absence of groundtruth high-resolution feature annotations. Although downstream task labels such as depth or masks can be used, this approach risks compromising the task-agnostic nature of the upsampler and hinders its ability to generalize to unseen tasks. Due to this challenge, previous task-agnostic feature upsampling works, FeatUp [ 12 ] and LiFT [ 52 ] , both compute the training loss at a low resolution. In this work, we address these limitations by constructing pseudo-groundtruth (pseudo-GT) features directly at the input image resolution. Specifically, we leverage class-agnostic masks generated by off-the-shelf segmentation foundation models [ 21 , 45 ] to ensure that the pseudo-GT accurately reflects the underlying geometry and delineates object boundaries. We further refine the pseudo-GT using a self-distillation strategy that reduces noise and artifacts. This high-resolution pseudo-GT enables loss computation at a high resolution, empowering the upsampler to learn fine-grained details. In Fig. 2 , we qualitatively show that LoftUp yields markedly sharper and more detailed features than alternative upsampling methods. To further demonstrate the versatility and effectiveness of our approach, we evaluate it on a range of downstream tasks such as semantic segmentation, depth estimation, and video objectschannen2025siglip2. As shown in Fig. 1 , our approach leads to performance gains of 10–20% over previous SoTA upsamplers for most tasks, and an impressive nearly 50% improvement on video object segmentation [ 39 ] . Furthermore, thanks to its coordinate-based design, our upsampler adapts seamlessly to various input and feature resolutions, catering to the diverse requirements of downstream applications. Overall, with less than a 20% increase in parameters compared to the original foundation models, our feature upsampler offers a task-agnostic, lightweight, and plug-and-play enhancement that significantly boosts VFM backbones across multiple tasks. 2 Related Work Feature upsampling refers to increasing the spatial resolution of a feature map. In this work, our goal is to increase the feature resolution to the original image resolution, that is, full resolution. Traditional non-learnable methods include various ways of interpolation [ 33 , 7 ] and image-adaptive filtering such as joint bilateral filtering (JBU) [ 22 ] and guided image filtering [ 15 ] . In modern deep learning, previous work has proposed various architecture- and downstream-task-specific feature upsamplers. For example, Index Networks [ 29 ] and A2U [ 5 ] are effective on image matting, but fall short in other tasks. PointRend [ 20 ] proposes a point-rendering method specifically for upsampling segmentation output. And CARAFE [ 57 ] , SAPA [ 31 ] , and FADE [ 30 ] are proposed specifically for encoder-decoder architectures. More recently, with the success of vision foundation models such as DINOv2 [ 37 ] and CLIP [