Poivre: Self-Refining Visual Pointing with Reinforcement Learning

Wenjie Yang, Zengfeng Huang · 2025

Read Poivre: Self-Refining Visual Pointing with Reinforcement Learning by Wenjie Yang, Zengfeng Huang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-23746.txt.

Introduction

Vision-language models point at objects far worse than people do, partly because they must answer in one shot. Wenjie Yang and Zengfeng Huang propose Poivre, a self-refining loop named for its steps: point, visualize, then refine. The model marks its guessed point on the image, looks again, and adjusts if needed, like a person checking a fingertip. Reinforcement learning trains that habit, guided by a process reward with tidy theoretical properties.

Their trained model, Poivre-7B, set a new best score on Point-Bench. It beat closed models such as Gemini-2.5-Pro and large open ones such as Molmo-72B by over three percent. Code, data, and the model are released.

Catalog record

CREATORWenjie Yang, Zengfeng Huang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-23746

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Poivre: Self-Refining Visual Pointing with Reinforcement Learning Wenjie Yang, Zengfeng Huang Abstract Visual pointing, which aims to localize a target by predicting its coordinates on an image, has emerged as an important problem in the realm of vision–language models (VLMs). Despite its broad applicability, recent benchmarks show that current VLMs still fall far behind human performance on this task. A key limitation is that VLMs are typically required to complete the pointing task in a single step, akin to asking humans to point at an object without seeing their own fingers. To address this issue, we propose a simple yet effective self-refining procedure: Poi nt, V isualize, then Re fine (Poivre 1 1 1 Poivre means “pepper” in French. ). This procedure enables a VLM to first mark its estimated point, then iteratively refine the coordinates if necessary. Inspired by advances of reasoning models in the natural language domain, we employ reinforcement learning (RL) to incentivize this self-refining ability. For the RL training, we design a neat process reward that is not only empirically effective but also grounded in appealing properties. Our trained model, Poivre-7B , sets a new state of the art on Point-Bench, outperforming both proprietary models such as Gemini-2.5-Pro and large open-source models such as Molmo-72B by over 3%. To support future research, we release our training and inference code, dataset, and the Poivre-7B checkpoint. Figure 1 : Performance of VLMs on Point-Bench ( Cheng et al. 2025 ) . Circle radius indicates model size. Dotted lines represent models whose average inference time is not available. 1 Introduction Pointing is one of the most fundamental and intuitive mechanisms for grounding language in visual contexts. From early childhood communication to robotics and human–computer interaction, pointing provides a precise yet low-bandwidth way to bridge abstract intent and concrete spatial reference ( Cheng et al. 2025 ) . In the realm of VLM, visual pointing is typically formulated as predicting the coordinates of a target object in an image, given a natural language instruction. This capability is critical for a wide range of applications, including assistive technologies for the visually impaired ( Yuan et al. 2024b ) , interactive educational tools ( Hu et al. 2024 ) , and robotic manipulation systems that require fine-grained spatial reasoning ( Yuan et al. 2024a ) . Despite its importance, recent benchmarks such as Point-Bench ( Cheng et al. 2025 ) reveal that even the most advanced VLMs still fall far short of human-level performance. While humans naturally refine their gestures by observing and adjusting, current VLMs are usually constrained to produce a pointing result in a single step. This mismatch leads to inaccurate predictions and limits the robustness of VLMs in realistic settings. Analogous to asking humans to point without seeing their own fingers, the one-shot paradigm overlooks the natural process of iterative refinement that underlies human pointing behavior. To address this gap, we propose Poi nt, V isualize, then Re fine (Poivre), a simple yet effective procedure that enables self-refining for visual pointing. Under this procedure, a model first generates an initial estimation of the target location, then visualizes this prediction by marking it on the image, and refines its estimate in subsequent rounds if necessary. This iterative interaction not only makes the task more natural and improve robustness, but also opens a path for VLMs to generalize beyond their training setup by extrapolating to more refinement steps at inference time. Inspired by advances in reasoning models within the natural language domain, we employ reinforcement learning to incentivize this self-refining ability. In particular, we introduce a process reward inspired by potential-based reward shaping (PBRS), which encourages the model to improve across refinement steps rather than optimizing only for the final outcome. Our RL-trained model, Poivre-7B, achieves new state-of-the-art results on Point-Bench ( Cheng et al. 2025 ) , surpassing both proprietary models (e.g., Gemini-2.5-Pro) and large open-source models (e.g., Molmo-72B). Moreover, Poivre-7B demonstrates strong generalization on robotics benchmarks such as where2place from RoboPoint ( Yuan et al. 2024a ) , despite not being specifically trained on robotics datasets. Our contributions are summarized as follows: • We propose the Point, Visualize, then Refine procedure, which allows VLMs to iteratively refine predictions by observing their own outputs, making the pointing task more natural and improve robustness compared to the one-shot setting. • We design a PBRS-inspired process reward that encourages consistent improvement across refinement steps, extending beyond the conventional outcome reward. This leads to more effective RL training for visual pointing. Our RL-trained model, Poivre-7B, achieves a new state of the art, outperforming both proprietary models (e.g., Gemini-2.5-Pro) and large open-source models (e.g., Molmo-72B) by a significant margin. • Beyond the results on Point-Bench, we further conduct experiments to demonstrate the effectiveness of our process reward, the extrapolation capability of the trained model, and its generalization to the robotics domain. Our findings indicate several promising future directions for research in visual test-time scaling and chain-of-thought prompting. To facilitate future research, we release our training and inference code, dataset, and the Poivre-7B checkpoint. 2 Related Work Visual pointing. Visual pointing has recently emerged as an important research problem in the VLM community. Molmo and Pixmo ( Deitke et al. 2025 ) pioneered this direction by introducing open-source VLMs and datasets explicitly targeting the pointing task. In particular, the Pixmo-Points dataset contains 223k images paired with 2.3M question–point annotations, providing large-scale supervision for training visual pointing models. Following its release, several open-source models, including Qwen2.5-VL ( Bai et al. 2025 ) , incorporated pointing supervision to enhance their performance. Beyond general-purpose VLMs, visual pointing has also been studied in robotics. For instance, RoboPoint ( Yuan et al. 2024a ) links natural language instructions with keypoints relevant for manipulation, bridging pointing with real-world control. However, in contrast to the natural language domain, the application of reinforcement learning to visual pointing remains surprisingly limited, which motivates us to examine its potential for this task. Reinforcement learning for (V)LMs. Reinforcement learning has been widely explored as a mechanism to align both language models and vision–language models with desired behaviors. In the language domain, RL has demonstrated remarkable success in reasoning-intensive tasks, such as mathematical problem solving ( Shao et al. 2024 ; Luo et al. 2025 ; Xu et al. 2025 ) . In multimodal contexts, most RL efforts have concentrated on visual question answering (VQA), often under the paradigm of “thinking with images” ( Zheng et al. 2025b ; Liu et al. 2025 ; Zhu et al. 2025 ) . To the best of our knowledge, the most relevant work is VisionReasoner ( Liu et al. 2025 ) , which employs an L1 reward based on grounding points. While VisionReasoner achieves impressive results, it relies solely on single-turn RL, which limits its ability to incentivize self-refinement, a key focus of our work. Another notable study is Point-RFT ( Ni et al. 2025 ) , which leverages the pointing capabilities of VLMs to improve question-answering performance. However, Point-RFT addresses a fundamentally different task from ours. We also wish to highlight several excellent studies that explore a related but distinct task: graphical user interface grounding ( Tang et al. 2025 ; Yuan et al. 2025 ; Zhou et al.

Read the complete text