Region-R1: Reinforcing Query-Side Region Cropping for Multi-Modal Re-Ranking
Chan-Wei Hu, Zhengzhong Tu · 2026
Read Region-R1: Reinforcing Query-Side Region Cropping for Multi-Modal Re-Ranking by Chan-Wei Hu, Zhengzhong Tu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-05268.txt.
Introduction
Hu and Tu propose Region-R1, a cropping framework that treats region selection as a decision problem during ranking. It learns to keep the full image or focus on a relevant region before scoring candidates. Standard re-rankers process the full image as a global embedding, vulnerable to distractors such as background clutter skewing similarity scores. Region-R1 learns a policy with group relative policy optimization to dynamically crop a region. On E-VQA and InfoSeek, Region-R1 gains consistently, achieving top performance by increasing Recall@1 by up to 20 percent.
Catalog record
| CREATOR | Chan-Wei Hu, Zhengzhong Tu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34675 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.05268
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Region-R1: Reinforcing Query-Side Region Cropping for Multi-Modal Re-Ranking Chan-Wei Hu, Zhengzhong Tu Abstract Multi-modal retrieval-augmented generation (MM-RAG) relies heavily on re-rankers to surface the most relevant evidence for image-question queries. However, standard re-rankers typically process the full query image as a global embedding, making them susceptible to visual distractors (e.g., background clutter) that skew similarity scores. We propose Region-R1 , a query-side region cropping framework that formulates region selection as a decision-making problem during re-ranking, allowing the system to learn to retain the full image or focus only on a question-relevant region before scoring the retrieved candidates. Region-R1 learns a policy with a novel region-aware group relative policy optimization (r-GRPO) to dynamically crop a discriminative region. Across two challenging benchmarks, E-VQA and InfoSeek, Region-R1 delivers consistent gains, achieving state-of-the-art performances by increasing conditional Recall@1 by up to 20%. These results show the great promise of query-side adaptation as a simple but effective way to strengthen MM-RAG re-ranking. Figure 1: Overview of query-side region cropping approach. Conventional re-rankers treat the query as fixed and only re-order candidates. Our method instead automatically adapt the query by selecting an informative region or keeping the original query before scoring candidates. 1 Introduction Multi-modal retrieval-augmented generation (MM-RAG) has become an increasingly important paradigm for grounding vision language models (VLMs) with both textual and visual information 14 ; 17 ; 35 ; 6 . In MM-RAG systems, queries may include images in addition to natural language, and retrieved evidence may consist of heterogeneous modalities such as images, captions, diagrams, or structured visual content. As a result, the effectiveness of MM-RAG depends not only on language understanding, but also on how visual information is represented. In practical MM-RAG pipelines, an upstream retriever first produces a candidate set, after which a re-ranking model refines their ordering with respect to a given query. Recent work has primarily focused on improving retrievers 18 ; 17 ; 38 or designing more expressive multi-modal re-rankers 36 ; 33 ; 34 . By contrast, comparatively little attention has been paid to how the multi-modal query representation itself is constructed and controlled during re-ranking. In particular, re-rankers typically rely on a single global embedding of the query image, implicitly assuming that all regions of the image are relevant to the user’s question. This assumption is often violated in realistic MM-RAG scenarios. Query images frequently contain distractive objects, background regions, or contextual elements that are irrelevant to the question being asked. When such irrelevant regions dominate the global visual representation, they can distort visual similarity estimates and degrade re-ranking performance, even when the candidate set is fixed. This issue is specific to multi-modal settings and does not arise in purely textual RAG. A natural response is to perform region cropping on the query image in a question-conditioned manner. Modern vision-language models exhibit strong localization capabilities, suggesting that they can identify regions relevant to a given question. Indeed, our preliminary analysis shows that, under a fixed candidate pool and a scoring model, replacing the full query image with an appropriately selected region can substantially improve multi-modal re-ranking performance. However, naive or heuristic cropping strategies can also remove useful visual context and lead to worse rankings. This raises a key challenge for MM-RAG systems: how can query-side visual information be selectively modified in a way that reliably improves multi-modal re-ranking performance? In this work, we formulate query-side region cropping as a decision problem optimized directly for multi-modal re-ranking objectives, named Region-R1 . We focus on the re-ranking stage because retrieval is a coarse-grained, high-recall step that must run efficiently at corpus scale, whereas re-ranking performs fine-grained discrimination over a small top- K set. Applying region cropping during retrieval would require policy execution during large-scale negative mining and may even necessitate re-indexing, substantially increasing training and system cost. In addition, it also risks discarding information too early and hurting recall. Accordingly, we apply region cropping only at re-ranking, where computation is bounded and the selected region helps disambiguate among already plausible candidates. Given a query image and user question, a policy determines whether to retain the full image or to select a region, with the explicit goal of improving the ranking of a fixed multi-modal candidate set. Our model learns the region cropping policy using reinforcement learning, with rewards defined as improvements over baseline measured by standard information retrieval metrics such as Mean Reciprocal Rank 11 and Normalized Discounted Cumulative Gain 29 . We evaluate the proposed approach on two challenging datasets, E-VQA 21 and InfoSeek 7 . Experimental results show that the learned policy selectively applies region cropping when beneficial. Most notably, Region-R1 improves top-1 recall over prior re-rankers in our setting. Ours CondRecall@1 improves 20% on E-VQA and 8% on InfoSeek. These gains translate into more reliable downstream grounding because MM-RAG systems often condition generation on the top-ranked evidence. In summary, this paper makes the following contributions: • We introduce query-side region cropping for multi-modal re-ranking, and formulate it as a decision problem to improve ranking. • We propose Region-R1 , a candidate-agnostic reinforcement learning framework that learns a region cropping policy optimized directly for multi-modal re-ranking. • Our experiment shows Region-R1 consistently improves performance on two challenging datasets, boosting CondRecall@1 by 20% on E-VQA and 8% on InfoSeek. We further provide diagnostic analyses of when region cropping is applied and ablations isolating the effect of each reward component. 2 Related Works Knowledge-Based VQA. Knowledge-based Visual Question Answering (KB-VQA) requires models to answer questions by retrieving external knowledge beyond the query input 20 ; 25 . Early methods leveraged structured knowledge graphs 23 ; 40 ; 12 ; 15 ; 32 , but often struggled with the open-ended and diverse nature of real-world questions. Recent work has largely shifted to Retrieval-Augmented Generation (RAG), retrieving evidence from unstructured corpora (e.g., Wikipedia). Benchmarks such as InfoSeek 7 and Encyclopedic-VQA (E-VQA) 21 formalize this paradigm and emphasize fine-grained entity linking and multi-hop reasoning. State-of-the-art systems commonly follow a “Retriever-Reranker-Generator” pipeline. Wiki-LLaVA 5 applies hierarchical retrieval, while EchoSight 33 and OMGM 34 use Q-Former-based re-ranking after initial retrieval. More recently, studies optimize the interaction between VLMs and retrieval. ReflectiVA 9 and MMKB-RAG 19 add self-reflective signals and consistency checks to decide when retrieval is needed, and ReAG 10 and Iterative-RAG 8 investigate iterative retrieval to refine evidence chains. A persistent challenge is the semantic gap between visual queries and textual documents. To mitigate it, several works adapt query expansion from text retrieval 22 , using generated captions or entity-aware rewriting to better guide search 1 ; 39 . However, most still treat the visual input as a fixed global representation, underutilizing discriminative visual query reformulation for filtering distractors. Multi-Modal Re-Ranking. Re-ranking is a critical refinement stage in MM-RAG pipeline, designed to re-order candidate evidence to maximize the rank of true positives. Recent approaches fall into two primary categories. (1) Interaction-Centric Methods. These models rely on deep cross-modal alignment or generative reasoning to score candidates. EchoSight 33 , OMGM 34 , and ReflectiVA 9 orchestrate fine-grained evidence fusion and self-reflective tokens to weigh relevance dynamically. RAMQA 3 and Mario 24 employ Large Multimodal Models as generative listwise re-rankers, synthesizing text and visual history to predict the optimal permutation of documents. (2)