Do You See What I Am Pointing At? Gesture-Based Egocentric Video Question Answering

Yura Choi, Roy Miles, Rolandos Alexandros Potamias, Ismail Elezi, Jiankang Deng, Stefanos Zafeiriou, Imperial College London · 2026

Read Do You See What I Am Pointing At? Gesture-Based Egocentric Video Question Answering by Yura Choi, Roy Miles, Rolandos Alexandros Potamias, Ismail Elezi, Jiankang Deng, Stefanos Zafeiriou, Imperial College London (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-12533.txt.

Introduction

Choi and co-authors tackle question answering about pointing gestures in egocentric video. Current multimodal models struggle here, lacking gesture-rich data and fine-grained pointing intent. The team builds EgoPointVQA, a benchmark of 4000 synthetic and 400 real videos across deictic reasoning tasks. Their HINT method encodes tokens from 3D hand keypoints and interleaves them with the model input. These Hand Intent Tokens give explicit spatial and temporal context for each gesture. HINT-14B reaches 68.1% average accuracy across six tasks, ahead of strong baselines.

Catalog record

CREATORYura Choi, Roy Miles, Rolandos Alexandros Potamias, Ismail Elezi, Jiankang Deng, Stefanos Zafeiriou, Imperial College London
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56973
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.12533

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Do You See What I Am Pointing At? Gesture-Based Egocentric Video Question Answering Yura Choi, Roy Miles, Rolandos Alexandros Potamias, Ismail Elezi, Jiankang Deng, Stefanos Zafeiriou, Imperial College London Abstract Understanding and answering questions based on a user’s pointing gesture is essential for next-generation egocentric AI assistants. However, current Multimodal Large Language Models (MLLMs) struggle with such tasks due to the lack of gesture-rich data and their limited ability to infer fine-grained pointing intent from egocentric video. To address this, we introduce EgoPointVQA , a dataset and benchmark for gesture-grounded egocentric question answering, comprising 4,000 synthetic and 400 real-world videos across multiple deictic reasoning tasks. Built upon it, we further propose H and In tent T okens (HINT), which encodes tokens derived from 3D hand keypoints using an off-the-shelf reconstruction model and interleaves them with the model input to provide explicit spatial and temporal context for interpreting pointing intent. We show that our model outperforms others in different backbones and model sizes. In particular, HINT-14B achieves 68.1% accuracy, on average over 6 tasks, surpassing the state-of-the-art, InternVL3-14B, by 5.4%. To further facilitate the open research, we will release the code, model, and dataset. Figure 1 : Illustration of EgoPointVQA . Left : EgoPointVQA includes questions with deictic pronouns requiring gesture understanding, either identifying single pointed objects (top) or tracking multiple references across frames (bottom). Right : State-of-the-art models, including GPT-4o 20 and Qwen3-VL-32B 37 , fail to resolve the question with pointing gestures, incorrectly stating the two pots have different colors despite both being black. Zoomed circles highlight the pointed objects. 1 Introduction As AI assistants become deeply integrated into daily wearable devices, from augmented and virtual reality (AR/VR) platforms like the Apple Vision Pro and Meta Orion to smart glasses like Meta Ray-Ban 39 ; 29 with built-in cameras, their ability to understand a user’s attention within their environment becomes essential 24 ; 45 ; 32 ; 35 . In particular, it is important to resolve spatial references through pointing gestures and deictic expressions (e.g., “Should I use this one?”), which are natural in human communication 43 ; 40 ; 8 . Thus, an AI assistant must understand not just what it sees, but also where the user is directing their attention. Typically, this requires: (1) identifying the deictic expression within the question that depends on gesture for meaning, (2) interpreting the hand’s movement and pose to understand the user’s referential intent, and (3) grounding this intent to identify the referred object and generate an appropriate response. Despite the rapid progress of Multimodal Large Language Models (MLLMs), such gesture-aware and region-specific question answering remains largely unexplored. From examples in Fig. Do You See What I Am Pointing At? Gesture-Based Egocentric Video Question Answering , we observe that even state-of-the-art MLLMs fail to address questions grounded in the user’s gesture. This can be primarily attributed to two issues. First, the datasets these MLLMs are trained on contain limited gesture-rich video data, especially those capturing natural pointing behavior and egocentric interactions between users and nearby objects. As a result, models rarely encounter examples where gestures and deictic language co-occur in realistic contexts. Second, current architectures are not designed to explicitly encode or reason about gesture information. Most MLLMs integrate visual and textual inputs globally, without mechanisms to interpret hand positions or pointing directions. Consequently, they struggle to connect deictic expressions like “these” to the correct objects being referenced. To address this, we introduce EgoPointVQA , a dataset composed of 4,000 synthetic and 400 real egocentric videos. These videos feature multiple objects and are specifically designed to support questions such as “What is the object I am pointing to?”, “How many objects of this type are there?”, or “What is the order of the objects I am pointing at?”. To enable systematic evaluation, we benchmark several models across our deictic task categories. Our evaluation benchmark contains 672 question–answer pairs over 300 real-world egocentric videos. Unlike prior egocentric video question answering (VQA) datasets that emphasize general scene understanding, EgoPointVQA focuses on fine-grained region-level reasoning, requiring both temporal and spatial comprehension to resolve gesture-based references accurately. While training and benchmarking on this dataset helps mitigate the problem, the model must also explicitly attend to gestures. To achieve this, we introduce H and In tent T okens (HINT), which are gesture-aware tokens, offering a compact representation and allowing the model to better interpret the user’s intent. We obtain 3D hand keypoints for each frame of the video using an off-the-shelf hand reconstruction model, and pass them through a lightweight adapter to produce frame-aligned hand-intent tokens. We then insert these tokens into the MLLM input sequence alongside the corresponding visual tokens, so the model receives an explicit gesture stream in addition to video and text. This design allows the MLLM to associate hand motion and pointing direction with the question and answer, leading to more precise grounding of deictic expressions (e.g., “this one”). In the experimental section, we demonstrate that training with gesture tokens on deictic questions improves performance by 6.5% over standard fine-tuning, and surpasses the baseline by 8.6%. In summary, our contributions are the following: • We introduce EgoPointVQA , the first dataset specifically designed for deictic question answering in egocentric videos, where user gestures and pointing behaviors are central to interpretation. • We propose HINT , a simple yet effective approach that encodes gesture tokens derived from off-the-shelf 3D hand keypoints and interleaves them with the model input, providing explicit spatial and temporal context for interpreting pointing intent. • We demonstrate that HINT achieves state-of-the-art performance on gesture-grounded question answering tasks, outperforming existing MLLMs and establishing a strong foundation for future research. Figure 2 : Task taxonomy and examples from EgoPointVQA . EgoPointVQA includes six subsets of questions regarding the properties of a pointed object. Each example shows egocentric video frames and a question using deictic references. Tasks include reference (object identification), counting (number of same objects), spatial (location and relative depth), temporal (order of multiple gestures), attribute (object properties), and feedback (object function). All questions require resolving deictic references through visual grounding of pointing gestures. The pointed objects are highlighted with red circles for visualization purposes. 2 Related Work Egocentric video question answering. The field of egocentric video question answering (VQA) has emerged as a critical challenge, established by foundational large-scale egocentric video datasets 9 ; 6 ; 33 ; 17 . Building upon these rich video corpora, dedicated VQA benchmarks were developed to specifically test complex reasoning about the wearer’s actions and environment 1 ; 31 ; 28 ; 22 ; 36 ; 7 ; 52 ; 13 . Benchmarks such as EgoThink 5 and VidEgoThink 4 focus on questions written in first-person perspective. Recently, EgoGPT 47 attempted to fine-tune MLLM 25 on wearer-perspective data, achieving improved performance on egocentric QA benchmarks 47 . Similarly, Ego-R1 38 introduced a chain-of-tool-thought agent to reason over ultra-long first-person videos by decomposing queries and using external tools. These works mostly focus on long-term memory retrieval, habit analysis, or high-level queries. However, none address region-specific ambiguities: current MLLMs fail to resolve deictic questions (e.g., “what is this?”) when the referent is only indicated via a pointing gesture. Region-specific visual question answering. Region-level question answering has

Read the complete text