Learning to Focus and Precise Cropping: A Reinforcement Learning Framework with Information Gaps and Grounding Loss for MLLMs
Xuanpu Zhao, Zhentao Tan, Dianmo Sheng, Tianxiang Chen, Yao Liu, Tao Gong, Nenghai Yu · 2026
Read Learning to Focus and Precise Cropping: A Reinforcement Learning Framework with Information Gaps and Grounding Loss for MLLMs by Xuanpu Zhao, Zhentao Tan, Dianmo Sheng, Tianxiang Chen, Yao Liu, Tao Gong, Nenghai Yu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-27494.txt.
Introduction
Zhao and co-authors train vision models to crop images with real care. Many models glance at the whole scene and skim the crop. Their two-stage reinforcement method attacks that habit. Stage one blurs the global view to create an information gap. The model must then mine the crop for answers. Stage two adds a grounding loss for exact crop boxes. Only a few box labels are needed. Tests show clear gains in fine visual question answering.
Catalog record
| CREATOR | Xuanpu Zhao, Zhentao Tan, Dianmo Sheng, Tianxiang Chen, Yao Liu, Tao Gong, Nenghai Yu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47746 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.27494
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Learning to Focus and Precise Cropping: A Reinforcement Learning Framework with Information Gaps and Grounding Loss for MLLMs Xuanpu Zhao, Zhentao Tan, Dianmo Sheng, Tianxiang Chen, Yao Liu, Tao Gong, Nenghai Yu Abstract To enhance the perception and reasoning capabilities of multimodal large language models in complex visual scenes, recent research has introduced agent-based workflows. In these works, MLLMs autonomously utilize image cropping tool to analyze regions of interest for question answering. While existing training strategies, such as those employing supervised fine-tuning and reinforcement learning, have made significant progress, our empirical analysis reveals a key limitation. We demonstrate the model’s strong reliance on global input and its weak dependence on the details within the cropped region. To address this issue, we propose a novel two-stage reinforcement learning framework that does not require trajectory supervision. In the first stage, we introduce the “Information Gap” mechanism by adjusting the granularity of the global image. This mechanism trains the model to answer questions by focusing on cropped key regions, driven by the information gain these regions provide. The second stage further enhances cropping precision by incorporating a grounding loss, using a small number of bounding box annotations. Experiments show that our method significantly enhances the model’s attention to cropped regions, enabling it to achieve state-of-the-art performance on high-resolution visual question-answering benchmarks. Our method provides a more efficient approach for perceiving and reasoning fine-grained details in MLLMs. Code is available at: https://github.com/XuanPu-Z/LFPC . 1 Introduction Figure 1 : Reasoning example for RL-based methods. In example (a), the model crops the region correctly but still fails to answer the color of the logo. While in example (b), the model confuses the sign with a telephone booth even after zooming in. Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities in understanding and reasoning about the visual world, achieving impressive performance across a wide range of vision-language tasks 10 ; 35 ; 23 ; 7 . However, their ability to perform fine-grained perception and reasoning in complex visual scenes remains a significant challenge. Especially when the object of reasoning is small or obscured by a complex background 41 ; 45 , the model struggles to accurately locate the target and attend to detailed cues in a single inference. To address these limitations, a promising direction in recent research has been the introduction of agent-based workflows 39 ; 51 ; 13 ; 54 . In this paradigm, the MLLM is not merely a passive observer but an active agent that can use external tools to dynamically explore visual input. A particularly effective approach involves empowering the MLLM with a “cropping tool”, allowing it to autonomously identify and zoom in on specific regions of interest to gather detailed information necessary for answering complex questions. These agentic MLLMs have achieved substantial progress through two primary training strategies: a hybrid of Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) 39 ; 51 ; 13 , or a pure Reinforcement Learning approach 54 . Despite their success, existing training methodologies have some inherent limitations. The SFT+RL hybrid approach heavily relies on generating vast quantities of reasoning trajectories from a powerful, proprietary teacher MLLM. This process is not only computationally expensive and time-consuming but also creates a performance ceiling, as the student model’s capabilities are inherently upper-bounded by those of its teacher. While pure RL methods circumvent the need for a teacher model, they introduce a subtler yet critical issue that we observed, which is also present to some extent in hybrid approaches. Taking DeepEyes 54 , a typical RL-based model, as a sample, we discovered a recurring pattern in its reasoning process: the model often predicts the final answer before executing a crop, and subsequently uses the cropped region merely to confirm its pre-existing conclusion, as shown in Figure 1 . We hypothesize that this “answer first, crop later” behavior indicates that the model is often performing a perfunctory tool call for cropping, rather than genuinely leveraging the information within the cropped region to aid its reasoning and question-answering process. To validate this hypothesis, we constructed a dedicated evaluation benchmark, which confirmed that the model’s performance is not significantly dependent on the content of the cropped region. To tackle this fundamental challenge of ensuring authentic reliance on tool-based observations, we propose a novel two-stage pure reinforcement learning framework. Our core objective is to compel the model to actively seek out and utilize information from cropped regions. In the first stage, we introduce an “Information Gap Mechanism”. In contrast to previous work that directly feeds original high-resolution images into the model, we strategically downsample the initial input images. The degree of downsampling is determined by the model’s own uncertainty; we select an appropriately low resolution such that the model yield answers inconsistent with those obtained at higher resolutions. However, when the agent decides to use its cropping tool, the region is extracted from the original, full-resolution image. This creates a crucial “information gap” between the low-detail global view and the high-detail local view, making the information within the crop indispensable for a correct answer and thus enhancing the model’s focus on the cropped region. In the second stage, to further refine the agent’s behavior, we annotate a small number of bounding boxes and introduce a grounding reward signal. This encourages the model to not only use the cropping tool but also to place the cropped region at a more precise location. Our experiments demonstrate the effectiveness of our method. It significantly improves the model’s attention to and reliance on cropped regions, while mitigating the formalistic cropping tool invocation issue observed in previous work. Remarkably, our method achieves state-of-the-art performance under both visual token budgets of 16,384 and 1,024 for the input image. Furthermore, when the visual token budget is limited to 1,024, our method still outperforms other approaches on some benchmarks even when they are allowed 16,384 visual tokens, highlighting both its effective utilization of fine-grained details within cropped regions and its computational efficiency. 2 Related Work Multi-modal Large Language Models. Multimodal large language models (MLLMs) has progressed from early systems that aligned a vision encoder with a large language model 6 ; 38 to more tightly integrated structures enabled by joint training. BLIP-2 19 and LLaVA 22 ; 21 pioneered the alignment of visual features with LLMs using lightweight Q-Former or projectors, while approaches such as LLaVA-OneVision 16 enhanced visual fidelity by accommodating flexible image resolutions. These advancements have fostered a rich ecosystem of powerful open-source models, including the Qwen-VL 1 ; 40 ; 2 , and InternVL 5 ; 4 ; 3 ; 55 series. With the maturation of these models, attention has increasingly turned to their reasoning abilities, particularly through Multimodal Chain-of-Thought (MCoT). Research in MCoT reasoning can be broadly categorized into two main paradigms. One paradigm depends on manually designed workflows to structure reasoning 25 ; 27 , typically targeting problems like localization 8 ; 43 or knowledge retrieval 36 ; 17 . The other explores RL-based approaches 29 ; 32 to adapt text-centric reasoning for multimodal challenges like spatial reasoning 28 ; 20 ; 44 and reasoning segmentation 24 . High-Resolution Visual Question Answering. Multimodal Large Language Models (MLLMs) performs poorly on High-Resolution Visual Question Answering 45 ;