Leveraging Gaze and Set-of-Mark in VLLMs for Human-Object Interaction Anticipation from Egocentric Videos
Daniele Materia, Francesco Ragusa, Giovanni Maria Farinella · 2026
Read Leveraging Gaze and Set-of-Mark in VLLMs for Human-Object Interaction Anticipation from Egocentric Videos by Daniele Materia, Francesco Ragusa, Giovanni Maria Farinella (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-03667.txt.
Introduction
Daniele Materia and colleagues worked on guessing which object a person wearing a camera will handle next. Their method marks candidate objects in the view and follows the wearer's recent eye movements to read intent, sampling frames more densely just before contact. On the HD-EPIC egocentric collection, the approach beat previous methods and worked across different vision-language models.
Catalog record
| CREATOR | Daniele Materia, Francesco Ragusa, Giovanni Maria Farinella |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29937 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.03667
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Leveraging Gaze and Set-of-Mark in VLLMs for Human-Object Interaction Anticipation from Egocentric Videos Daniele Materia, Francesco Ragusa, Giovanni Maria Farinella Abstract The ability to anticipate human-object interactions is highly desirable in an intelligent assistive system in order to guide users during daily life activities and understand their short and long-term goals. Creating systems with such capabilities requires to approach several complex challenges. This work addresses the problem of human-object interaction anticipation in Egocentric Vision using Vision Large Language Models (VLLMs). We tackle key limitations in existing approaches by improving visual grounding capabilities through Set-of-Mark prompting and understanding user intent via the trajectory formed by the user’s most recent gaze fixations. To effectively capture the temporal dynamics immediately preceding the interaction, we further introduce a novel inverse exponential sampling strategy for input video frames. Experiments conducted on the egocentric dataset HD-EPIC demonstrate that our method surpasses state-of-the-art approaches for the considered task, showing its model-agnostic nature. We release the code at the following GitHub repository . 1 Introduction Understanding human behavior from an egocentric point of view enables the development of systems that can support individuals in their daily activities across various domains [ 27 ] , including kitchens [ 1 , 17 ] , sports and hobbies [ 9 ] , as well as industrial workplaces [ 32 , 30 ] . The ability to accurately anticipate human-object interactions is particularly valuable for such systems, as it enables them to assist users and predict their intentions [ 4 , 5 ] . For example, in the kitchen domain, the system should alert the user if they are about to touch a hot pot without using pot holders [ 1 , 29 , 25 ] . Similarly, in industrial workplaces, forecasting potentially hazardous interactions can help prevent accidents by alerting workers and thereby improving safety [ 9 , 10 , 32 ] . Toward this direction, previous works have investigated different aspects of anticipation, focusing on predicting hand trajectories [ 13 , 19 ] , actions [ 5 , 30 , 35 , 34 ] , user localization [ 23 , 20 ] , and objects and their interactions [ 4 , 25 , 37 , 29 , 9 ] . With the recent advent of Vision Large Language Models (VLLMs), the anticipation task has been increasingly investigated using these models due to their ability to address a wide range of challenges. In particular, recent studies have focused on predicting future actions [ 14 , 39 ] , motion tracking [ 11 ] , and human-object interactions [ 25 ] . Figure 1 : Conceptual scheme of the VQA Interaction Anticipation task. In this paper, we address the problem of human-object interaction anticipation from egocentric videos leveraging the capabilities of VLLMs. We formulate this task as Visual Question Answering (VQA), adopting the protocol established by the recent HD-EPIC Gaze Interaction Anticipation benchmark [ 25 ] . Specifically, the model is queried to distinguish the interaction’s target object from a set of candidates, included in the input prompt, based on the visual context (see Figure 1 ). To this end, we propose a novel strategy to enhance the visual grounding capabilities of VLLMs by incorporating spatial scene information through Set-of-Mark (SoM) prompting [ 41 ] and modeling human intent via user gaze trajectories. Furthermore, we introduce an effective video sampling strategy for the visual input, aimed at optimizing the information required to anticipate human-object interactions while discarding irrelevant data. The proposed approach is model-agnostic, meaning it can be applied to any VLLM. Experiments on the HD-EPIC dataset [ 25 ] show that the proposed method outperforms state-of-the-art approaches, highlighting that the adopted models significantly benefit from our proposed strategy to anticipate human-object interactions. Additionally, we performed an ablation study to assess the impact of each component of the proposed approach on the overall performance. The contributions of this work are as follows: 1) we propose a novel approach that incorporates spatial scene information and models user intention through gaze trajectories; 2) through extensive experiments on the HD-EPIC dataset, our method outperforms state-of-the-art approaches in anticipating human-object interactions; 3) we release the source code of our approach at the following GitHub repository to encourage future research in this area. 2 Related Work 2.1 Egocentric Datasets Research in Egocentric Vision has been significantly accelerated in the last years by the release of several datasets. Initial efforts were made by the authors of the ADL (Activities of Daily Living) dataset [ 26 ] , which featured 20 videos of subjects performing unscripted daily routine activities like making coffee or washing dishes in their own homes, providing a rich source of complex human-object interactions. The authors of [ 1 ] released the first large-scale egocentric dataset: EPIC-KITCHENS. With 55 hours of video and 32 participants involved, it includes dense annotations for 39.6K action segments (spanning 125 verbs and 352 noun classes) and 454K object bounding boxes. Focusing exclusively on unscripted kitchen activities, this pioneering work was a turning point for the field, despite considering solely visual data. The dataset was later expanded to 100 hours of recordings in [ 3 ] , further consolidating its status as a standard benchmark for the community. EGTEA Gaze+ [ 17 ] was among the first datasets to incorporate user gaze information, providing it alongside egocentric video to support action recognition. The MECCANO dataset [ 31 ] focused on the industrial-like domain, specifically capturing recordings of subjects assembling a toy motorbike model. It comprises 20 videos from 20 subjects and includes 8,857 temporal annotations, involving 61 action classes, 12 verbs and 20 object classes. Its extension, MECCANO Multimodal [ 30 ] is one of the earliest examples of multimodal egocentric dataset, providing synchronized recordings of RGB, depth, and gaze data. Additionally, the authors emphasize the utility of the gaze signal in the proposed Action Recognition and Active Object Detection benchmarks, where they explicitly compare RGB-only models against RGB-Gaze models, demonstrating that integrating gaze helps the model focus on relevant visual features. The ENIGMA-51 dataset [ 32 ] was designed to bridge the gap to real-world industrial environments, focusing on the fine-grained understanding of human-object interactions during the repair of electrical boards. It comprises 51 videos featuring dense, fine-grained annotations including 14,036 interactions and 275K object bounding boxes, making it a challenging benchmark for human-object interaction detection in industrial environments. Later, the scale of egocentric research was redefined by Ego4D [ 9 ] , a massive collaborative effort comprising 3,670 hours of video, collected by 923 participants across 9 countries; covering a vast range of scenarios beyond just the kitchen or industry, including household chores, workplace activities, and leisure. The authors introduced a benchmark composed of five tasks: Episodic Memory , Hands and Objects , Audio-Visual Diarization , Social Interactions , and Forecasting . This effort was subsequently expanded by Ego-Exo4D [ 10 ] , which focuses on skilled human activities, such as sports, music, and dance, captured simultaneously from time-synchronized egocentric and exocentric perspectives. It provides rich annotations including expert commentary and 3D body pose to support benchmarks like proficiency estimation. A significant step forward was marked by the release of HD-EPIC [ 25 ] , a highly detailed dataset