JAFAR: Jack up Any Feature at Any Resolution
Louis Serrano, Jean-Emmanuel Haugeard, Matthieu Cord, Nicolas Thome · 2025
Read JAFAR: Jack up Any Feature at Any Resolution by Louis Serrano, Jean-Emmanuel Haugeard, Matthieu Cord, Nicolas Thome (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-11136.txt.
Introduction
Couairon and five co-authors present JAFAR, a lightweight upsampler that raises vision features to any resolution, in a paper from twenty twenty-five. Foundation vision encoders output coarse spatial features by design. Dense tasks like segmentation need fine detail that coarse maps lack. JAFAR uses attention to align high-resolution queries with rich low-resolution keys.
Queries derive from low-level image features that retain spatial detail. A spatial transform modulation step conditions the alignment on semantics. Training uses only low upsampling ratios, yet generalizes to far higher scales. No high-resolution supervision is required at any point. Extensive tests beat prior upsamplers across diverse dense tasks.
Catalog record
| CREATOR | Louis Serrano, Jean-Emmanuel Haugeard, Matthieu Cord, Nicolas Thome |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-11136
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
JAFAR: Jack up Any Feature at Any Resolution Louis Serrano, Jean-Emmanuel Haugeard, Matthieu Cord, Nicolas Thome Abstract Foundation Vision Encoders have become essential for a wide range of dense vision tasks. However, their low-resolution spatial feature outputs necessitate feature upsampling to produce the high-resolution modalities required for downstream tasks. In this work, we introduce JAFAR—a lightweight and flexible feature upsampler that enhances the spatial resolution of visual features from any Foundation Vision Encoder to an arbitrary target resolution. JAFAR employs an attention-based module designed to promote semantic alignment between high-resolution queries—derived from low-level image features—and semantically enriched low-resolution keys, using Spatial Feature Transform (SFT) modulation. Notably, despite the absence of high-resolution supervision, we demonstrate that learning at low upsampling ratios and resolutions generalizes remarkably well to significantly higher output scales. Extensive experiments show that JAFAR effectively recovers fine-grained spatial details and consistently outperforms existing feature upsampling methods across a diverse set of downstream tasks. Project page : https://jafar-upsampler.github.io Figure 1 : JAFAR upsamples features from any foundation vision encoder to any image resolution, using the input image as high-resolution guidance. It generates sharp, boundary-aligned feature maps and serves as a versatile drop-in module for a variety of downstream tasks—including semantic segmentation, open-vocabulary segmentation, depth estimation, CAM evaluation, and bird’s-eye-view segmentation—consistently enhancing performance. 1 Introduction Foundation vision encoders—whether trained with language supervision [ 1 , 2 , 3 , 4 , 5 ] or purely on visual data [ 6 , 7 , 8 ] —have become core components of modern computer vision pipelines. Vision-language models excel at tasks requiring generalization, such as zero-shot classification and open-vocabulary segmentation [ 9 , 10 ] . In contrast, image-only models, which focus on visual structure, often outperform in dense prediction tasks that demand fine-grained spatial reasoning, including semantic segmentation, depth estimation, object discovery, and point tracking [ 11 , 12 , 13 ] . To handle high-resolution inputs and large-scale training, foundation vision encoders typically downsample spatial information aggressively—by a factor of 14\times to 16\times —yielding semantically rich but spatially coarse feature maps. This compression introduces a bottleneck for downstream tasks that require pixel-level accuracy. As a result, downstream pipelines [ 14 , 15 , 11 , 16 , 17 ] often rely on interpolation or dedicated modules [ 18 , 19 ] designed to produce high-resolution outputs. Several strategies have been explored to overcome this bottleneck, but each comes with trade-offs in efficiency and output quality. A straightforward solution is to apply training-free interpolation methods, such as bilinear upsampling. While computationally efficient, these direct interpolations—relying solely on low-resolution feature maps—fail to leverage information from the original high-resolution image, often resulting in blurry outputs. Alternatively, one can upsample the input image prior to encoding to increase feature resolution. However, this approach significantly increases computational cost due to the quadratic complexity of self-attention—common in foundation models—and may introduce artifacts in the feature maps, ultimately degrading performance [ 20 , 21 ] . Focusing specifically on a target downstream task, [ 22 , 23 , 24 , 25 , 26 ] learn feature upsamplers using high-resolution supervision from task-specific labels. While generally lightweight, these upsamplers depend on labeled data tied to the end application, which limits their generalization and may bias the learned features toward optimizing task-specific losses. To address this, recent methods such as LiFT [ 27 ] and FeatUp [ 28 ] adopt task-agnostic training objectives. LiFT is trained to perform 2\times upsampling by regressing feature maps extracted from images at twice the input resolution. However, its convolution-based architecture is limited to fixed 2\times scaling, restricting its flexibility for arbitrary output resolutions. FeatUp, in contrast, uses augmented views and self-reconstruction to support higher upsampling ratios. Yet, its Joint Bilateral Upsampling (JBU) variant suffers from over-smoothed outputs, while its implicit variant requires training the upsampler for each image, making it impractical in real-world scenarios. In this paper, we introduce a feature upsampler designed to satisfy the following criteria: (i) a task-agnostic training objective, (ii) support for arbitrary output resolutions, (iii) compatibility with any vision encoder, and (iv) minimal computational overhead at inference time. To enable upsampling to arbitrary target resolutions, we formulate our approach as a global interpolation mechanism using a cross-attention block. The success of this attention-based method depends critically on achieving strong semantic alignment between the queries and keys. In JAFAR, we construct these representations asymmetrically (see Fig. 2 ): the queries retain high-resolution, low-level details such as color and texture, while the keys are hybrid features that combine high-level semantics with spatial cues. We find that enriching the keys with low-level information significantly improves query-key alignment and enhances generalization to unseen output resolutions. Additionally, we propose a simple training objective similar to [ 27 ] , but without being constrained to a fixed upsampling factor. Notably, we find that training on low upsampling factors at low resolutions (e.g., 8\times 8\rightarrow 32\times 32 ) is sufficient to generalize effectively to much larger scales (e.g., 32\times 32\rightarrow 448\times 448 ) while keeping memory requirements low during training, unlike training directly at higher resolutions and factors. Our contributions can be summarized as follows: • We introduce JAFAR, a novel lightweight attention-based feature upsampler that naturally supports upsampling to arbitrary resolutions. It explicitly promotes spatial alignment between high-resolution queries extracted from low-level image features and semantically enriched low-resolution keys. • We enforce this alignment by computing both queries and keys from the same input features, and injecting semantic information from the encoder’s deep features via spatial feature modulation. This design enables precise fusion of spatial detail and semantic context without reliance on external supervision. • We propose a highly efficient, task-agnostic training objective that requires no high-resolution supervision signal. Remarkably, we show that training at low resolutions and low upsampling ratios generalizes robustly to significantly higher output scales. • We demonstrate that the combination of our architecture and training objective yields substantial performance gains across a variety of downstream tasks. When used as a drop-in module, JAFAR consistently outperforms existing upsampling methods by a wide margin. 2 Related Work Feature Upsampling Feature upsampling aims to increase the spatial resolution of intermediate feature maps within deep networks—analogous to image upsampling, but performed in a latent space. This process is essential for dense prediction tasks such as segmentation and depth estimation, where fine spatial detail is critical. Traditional interpolation techniques, such as bilinear, spline, or Lanczos [ 29 , 30 , 31 , 32 ] , provide simple and efficient baselines but do not adapt to the underlying content. Recent neural methods improve on static approaches by learning to reconstruct high-resolution features from data. These methods fall into two categories: task-dependent, trained with downstream labels supervision, and task-agnostic, trained independently of the end task. For example, CARAFE [ 22 ] and DySample [ 24 ] predict content-aware kernels or dynamic sampling positions. SAPA [ 23 ] and ReSFU [ 25 ] exploit a similarity based approach to refine spatial semantics. However, task-specific reliance on labels limits generalization. Recent task-agnostic methods like LiFT [ 27 ] and FeatUp [ 28 ] remove this dependency. LiFT introduces a CNN module trained with a simple fixed scale training objective, while FeatUp relies on a complex multi-loss objective which makes training