PointVG-R: Internalizing Geometric Reasoning in MLLMs for Precise Pointing Localization via Visual Chain of Thought
Ling Li, Bowen Liu, Zinuo Zhan, Jianhui Zhong, Ziyu Zhu, Bingcai Wei, Kenglun Chang, Zhidong Deng · 2026
Read PointVG-R: Internalizing Geometric Reasoning in MLLMs for Precise Pointing Localization via Visual Chain of Thought by Ling Li, Bowen Liu, Zinuo Zhan, Jianhui Zhong, Ziyu Zhu, Bingcai Wei, Kenglun Chang, Zhidong Deng (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-24539.txt.
Introduction
Li and colleagues teach a multimodal model to reason geometrically about pointing, before naming what is pointed at. Pointing-based visual grounding asks a system to locate the object a gesture indicates. It decodes spatial relations between scene and hand. Traditional methods encode the image once into static features and reason mostly in language. Perceptual cues and explicit geometry the image plainly contains are discarded.
PointVG-R is a reasoning guided multimodal large language model, built against that cognitive vulnerability in interpreting gestural space. A visual chain of thought has the model work through the geometry stepwise. Spatial reasoning is internalised, rather than bolted on at the output. The framework shares a companion pointing study's conviction that gesture understanding is geometry first. Li's 2026 system aims for precise pointing localisation in scenes where language only reasoning guesses, forcing the model to look, orient, and only then answer.
Catalog record
| CREATOR | Ling Li, Bowen Liu, Zinuo Zhan, Jianhui Zhong, Ziyu Zhu, Bingcai Wei, Kenglun Chang, Zhidong Deng |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 67 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.24539
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PointVG-R: Internalizing Geometric Reasoning in MLLMs for Precise Pointing Localization via Visual Chain of Thought DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 4701 Thanks: Code: https://github.com/lingli1724/PointVG-R CCS: Computing methodologies Artificial intelligence CCS: Computing methodologies Object detection CCS: Computing methodologies Spatial and physical reasoning CCS: Computing methodologies Computer vision Ling Li, Bowen Liu, Zinuo Zhan, Jianhui Zhong, Ziyu Zhu, Bingcai Wei, Kenglun Chang, Zhidong Deng Abstract . Pointing-based visual grounding requires models to precisely locate target objects by deciphering complex spatial relationships between the visual scene and pointing gestures. Traditional methods typically encode input images into static feature representations and perform reasoning primarily within the linguistic domain, often overlooking the rich perceptual cues and explicit spatial geometry inherent in images. In this study, we aim to mitigate the cognitive vulnerability of models in interpreting gestural spatial relations by proposing PointVG-R, a reasoning-guided Multi-modal Large Language Model (MLLM). PointVG-R introduces geometric-aware reasoning for pointing-based grounding, enabling the model to think with images through the strategic integration of Reinforcement Learning (RL) and cold-start data. Specifically, we design a novel geometric reasoning pipeline that simulates the iterative cognitive process humans employ when interpreting pointing gestures. Furthermore, we construct EgoPoint-CoT, a high-quality visual Chain-of-Thought (CoT) dataset featuring detailed reasoning trajectories to guide the model via Supervised Fine-Tuning (SFT) and RL. To address the varying quality of learning signals encountered during training, we further propose an Adaptive Importance Weighting strategy based on Group Variance, which dynamically adjusts reward signals to optimize the learning process. Experimental results demonstrate that PointVG-R achieves SOTA performance, outperforming the baseline by 15.86 points in mIoU. Extensive ablation studies further validate the efficacy of our proposed modules. Our code and models will be made publicly available. Figure 1. Comparative analysis between PointVG-R and mainstream Multimodal Large Language Models (MLLM) in first-person pointing localization tasks. Above (baseline models): Traditional LVLMs are prone to geometric localization failure, saliency bias, and semantic misinterpretation due to a lack of logical reasoning when processing complex spatial relationships. Below (our method): In stark contrast, PointVG-R achieves a highly interpretable self-explanation process through the Visual Chain-of-Thought (V-CoT). The model not only outputs the final bounding box but also demonstrates a complete logical trajectory from hand localization and geometric ray estimation to target semantic identification. This "thinking with images" paradigm enables it to overcome the black-box limitations of traditional models and achieve more robust spatial understanding. 1. Introduction Visual grounding ( 108 ; 20 ; 98 ; 117 ; 97 ; 3 ) serves as a fundamental bridge between computer vision and natural language processing ( 29 ; 96 ) , aiming to precisely localize target objects in images based on specific referential expressions ( 85 ; 108 ; 98 ; 54 ) . While traditional research has primarily focused on semantic descriptions (e.g., "the red cup on the left") ( 93 ; 105 ; 60 ; 62 ) , gestural pointing provides a more intuitive means of referencing objects within a shared physical environment ( 7 ; 23 ; 24 ; 12 ; 31 ) . This shift toward gesture-based visual grounding ( 44 ; 16 ; 59 ; 28 ) necessitates that models effectively exploit the physical and geometric cues embedded in visual information to achieve accurate localization ( 116 ; 45 ) . We argue that simply treating pointing gestures as another "visual prompt" within existing Multimodal Large Language Model (MLLM) ( 103 ; 10 ; 107 ) architectures is insufficient ( 64 ; 68 ) . Standard MLLMs typically encode input images into holistic feature representations and reason primarily through linguistic associations ( 53 ; 69 ) . Consequently, they fail to explicitly account for the spatial-geometric constraints inherent in pointing behaviors. These models often exhibit "cognitive fragility," leading to hallucinated localization biases in complex interactive scenarios ( 73 ; 47 ) . To address this challenging task, as illustrated in Figure 1 , we propose PointVG-R, a reasoning-guided MLLM that internalizes geometric reasoning ( 52 ; 29 ; 41 ) into the visual grounding process. At the core of our approach is the "Thinking with Images" paradigm ( 75 ; 88 ; 91 ) , which advocates for bridging the gap between gesture perception ( 35 ; 22 ) and object localization via a Visual Chain-of-Thought (Visual CoT) ( 70 ; 58 ; 112 ; 63 ) . Rather than performing one-step direct predictions, PointVG-R is trained to execute a human-like, iterative cognitive process: identifying pointing cues, projecting implicit geometric trajectories, and anchoring the target through spatial reasoning. To facilitate this, we constructed a pioneering dataset, EgoPoint-CoT ( 44 ) , featuring fine-grained spatial reasoning trajectories. We employ a two-stage training protocol combining Supervised Fine-Tuning (SFT) ( 6 ; 32 ; 39 ) and Reinforcement Learning (RL) ( 114 ; 29 ; 89 ; 61 ) to ensure the model’s internal reasoning aligns with objective spatial logic. During the reinforcement learning phase, we define a reward function with multi-dimensional verifiable rewards to provide precise geometric feedback tailored to gesture-based grounding. In this process, we identify a critical bottleneck in applying RL to localization: the heterogeneity in learning signal quality ( 66 ; 36 ) . Due to the stochastic nature of sampling in RL, different sample groups offer varying levels of optimization value, where noisy samples can severely destabilize the training of precise bounding box (BBox) coordinates ( 79 ; 19 ) . To resolve this, we introduce a Group-Variance-based Adaptive Importance Weighting strategy. By dynamically recalibrating the contribution of reward signals based on intra-group consistency, we significantly enhance both training stability and localization precision. Experimental evaluations on egocentric benchmarks demonstrate that PointVG-R establishes a new state-of-the-art (SOTA) performance, surpassing competing baselines by 15.86\% . Our results underscore the necessity of explicit geometric reasoning for robust gesture-based grounding. Our contributions include: (1) PointVG-R: We propose PointVG-R, a reasoning-guided MLLM that enhances gesture-based grounding by internalizing explicit spatio-geometric reasoning. (2) Paradigm and Dataset: We introduce the "Thinking with Images" paradigm and the EgoPoint-CoT dataset, utilizing exhaustive visual reasoning chains to guide the model’s iterative cognitive process. (3) Algorithmic Optimization: We design multi-dimensional geometric rewards and a group-variance-based adaptive weighting strategy to address the challenges of signal quality variance and training instability in RL-based coordinate regression. (4) SOTA Performance: PointVG-R achieves superior results across relevant tasks. We will release our code, dataset, and model weights to the public. Figure 2. Visual Chain-of-Thought (V-CoT) reasoning trajectory of PointVG-R for pointing localization. The process comprises four sequential steps: (1) hand detection, (2) fingertip keypoint extraction, (3) pointing geometry (ray drawing and endpoint prediction), and (4) target bounding box alignment. Each step is accompanied by its corresponding attention heatmap, visualizing where the model focuses during inference. 2. Method 2.1. Problem Formulation Task Definition and Notation. Given an egocentric image I\in\mathbb{R}^{H\times W\times 3} containing a human pointing gesture G , the objective of this study is to learn a parameterized model f_{\theta} that accurately localizes the target object referenced by the gesture. We define the output as a spatial bounding box b=[x_{1},y_{1},x_{2},y_{2}]\in[0,1]^{4} , where (x_{1},y_{1}) and (x_{2},y_{2}) denote the normalized coordinates of the top-left and