REN: Fast and Efficient Region Encodings from Patch-Based Image Encoders
Savya Khosla, Sethuraman TV, Barnett Lee, Alexander Schwing, Derek Hoiem · 2025
Read REN: Fast and Efficient Region Encodings from Patch-Based Image Encoders by Savya Khosla, Sethuraman TV, Barnett Lee, Alexander Schwing, Derek Hoiem (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-18153.txt.
Introduction
Savya Khosla and co-authors present REN, a network that builds region descriptions from patch based image encoders in 2025. Recent pipelines pair a segmenter with an encoder to sum up image regions. The segment step is slow and memory hungry. REN skips it, using a light module that turns point prompts into region tokens directly.
Attention blocks read the point as a query over patch features. Tokens match the prompted object. Training ran with three known encoders, and new ones can join without fresh training. Token making runs sixty times faster with far less memory. On sorting and search tasks, REN beat the base encoders and matched segment based rivals. Code and models are public.
Catalog record
| CREATOR | Savya Khosla, Sethuraman TV, Barnett Lee, Alexander Schwing, Derek Hoiem |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-18153
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
REN: Fast and Efficient Region Encodings from Patch-Based Image Encoders Savya Khosla, Sethuraman TV, Barnett Lee, Alexander Schwing, Derek Hoiem Abstract We introduce the Region Encoder Network (REN) , a fast and effective model for generating region-based image representations using point prompts. Recent methods combine class-agnostic segmenters (e.g., SAM) with patch-based image encoders (e.g., DINO) to produce compact and effective region representations, but they suffer from high computational cost due to the segmentation step. REN bypasses this bottleneck using a lightweight module that directly generates region tokens, enabling 60 × \times faster token generation with 35 × \times less memory , while also improving token quality . It uses a few cross-attention blocks that take point prompts as queries and features from a patch-based image encoder as keys and values to produce region tokens that correspond to the prompted objects. We train REN with three popular encoders—DINO, DINOv2, and OpenCLIP—and show that it can be extended to other encoders without dedicated training. We evaluate REN on semantic segmentation and retrieval tasks, where it consistently outperforms the original encoders in both performance and compactness, and matches or exceeds SAM-based region methods while being significantly faster. Notably, REN achieves state-of-the-art results on the challenging Ego4D VQ2D benchmark and outperforms proprietary LMMs on Visual Haystacks’ single-needle challenge. The code and pretrained models are available at https://github.com/savya08/ren . 1 Introduction Do we really need hundreds of patch-based tokens to understand every image? Existing ViT encoders divide images into a fixed grid of patch tokens. This approach is inefficient for two main reasons: (1) images with few objects are represented using hundreds of tokens, leading to unnecessarily high memory and compute costs for applications like video understanding; and (2) patch tokens do not align with object boundaries, making tasks like retrieval and segmentation more challenging. A more scalable and effective alternative is a region-based representation, which divides images into object-level tokens and represents simpler images using far fewer tokens. Recent region-based representations average-pool patch features over regions generated by class-agnostic segmentation methods, such as SAM. These approaches improve semantic segmentation and retrieval accuracy, but generating accurate segmentation masks for all objects in an image remains computationally expensive, with high-quality segmentation models requiring up to 30 \times more time and memory than patch-based image encoders. This significantly limits the practical usability of region-based representations. Furthermore, existing SAM-based methods for region token generation rely on simple linear aggregation of patch features within object masks, which can remove fine-grained details and discard valuable context surrounding objects. Current segmentation models also tend to produce overlapping masks or miss parts of an image, resulting in over-representation of some regions and no representation for others. To address these limitations, we propose the Region Encoder Network (REN), a point promptable model that transforms patch-based features into region-based representations without requiring explicit segmentation . The key idea is that image features already contain sufficient information to segment objects. So instead of relying on external segmentation masks, we train REN to implicitly segment and pool information through attention. Specifically, as shown in Figure 1 , REN uses a lightweight cross-attention module trained with two learning objectives—contrastive learning and feature similarity—to extract high-quality region tokens directly from patch features. Given a point prompt (query), REN applies cross-attention over patch features (keys/values) to produce a token representing the object corresponding to the queried point. To generate region tokens for all objects, REN can be prompted with a grid of points followed by a simple token aggregation step to merge tokens that correspond to the same object. We explore the design choices that make REN effective and apply it to multiple patch-based image encoders—DINO [ 5 ] , DINOv2 [ 32 ] , and OpenCLIP [ 17 ] . Our experiments show that REN consistently outperforms the patch-based feature backbone on semantic segmentation and object retrieval. Further, it performs comparably to SAM-based approaches, while being more than 60 \times faster and while using 35 \times less memory . Our main contributions can be summarized as follows: • We introduce REN, a point promptable model that efficiently transforms patch-based feature maps into semantically meaningful region tokens using a lightweight cross-attention module, eliminating the need for costly segmentation models. We show that while REN can be trained for any patch-based encoder, we can also extend it to other encoders without dedicated training. • We study the design choices that affect the effectiveness and efficiency of transforming patch-based features into region tokens. For instance, we investigate the impact of prompting strategies, how to guide attention mechanisms effectively, how to preserve the information encoded in backbone features, and how to move beyond simple pooling operations for feature aggregation—such as average or max pooling—used in prior works. • We train REN with multiple image encoders and will release the code and trained models to facilitate future research and applications. Figure 1 : Overview of REN. Point prompts interact with patch-based features through cross-attention blocks to produce region tokens. The training objective combines two components: (1) a contrastive loss that aligns region tokens with those generated from an augmented view of the same image, and (2) a feature similarity loss that aligns a linear projection of these tokens with average-pooled patch features obtained using SAM masks. REN eliminates the need for explicit segmentation at inference time while producing efficient and semantically rich region representations. We also show thresholded attention maps for three query points inside the cross-attention block, which show that the model learns to aggregate features primarily from the regions marked by the corresponding point prompts. 2 Related Works Patch-Based Image Encoders. The Vision Transformer [ 9 ] introduced the paradigm of representing images as sequences of fixed-size patch tokens processed by Transformers [ 43 ] . Building on this, numerous patch-based image encoders have been developed to learn rich visual semantics [ 5 , 32 , 14 , 54 , 2 ] or align vision with other modalities like text [ 33 , 17 , 50 , 41 , 53 ] . These backbones power a wide range of applications—for example, SAM [ 22 ] uses MAE [ 14 ] , and LLaVA [ 25 ] builds on CLIP [ 33 ] . However, they represent each image with hundreds of tokens, regardless of content, leading to high computational overhead. Techniques like token pruning [ 34 , 26 , 49 , 31 ] and merging [ 28 , 23 , 47 , 37 , 30 , 27 ] aim to reduce token count, but they often trade off performance and still rely on grid-based, semantically weak tokens. This motivates exploring beyond patch tokens. Learning Objectness. The notion of objectness—that certain regions in an image correspond to coherent, meaningful entities—has inspired many works. Self-supervised methods like LOST [ 40 ] and TokenCut [ 45 ] aim to segment objects by clustering image patches based on feature affinity. In parallel, supervised models like DETR [ 4 ] and MaskFormer [ 8 , 7 ] learn to recognize objects using cross-attention between learnable queries and patch features under dense supervision. More recently, SAM and its variants [ 22 , 35 , 55 ] have demonstrated that sparse prompts (e.g., points) can effectively guide segmentation,