EgoTraj: Real-World Egocentric Human Trajectory Dataset for Multimodal Prediction
Ahmad Yehia, Abduallah Mohamed, Tianyi Wang, Jiseop Byeon, Kun Qian, [3pt] Junfeng Jiao, Christian Claudel, [6pt] Department of Civil, Architectural, Environmental Engineering, The University of Texas at Austin, Austin, TX 78712, USA · 2026
Read EgoTraj: Real-World Egocentric Human Trajectory Dataset for Multimodal Prediction by Ahmad Yehia, Abduallah Mohamed, Tianyi Wang, Jiseop Byeon, Kun Qian, [3pt] Junfeng Jiao, Christian Claudel, [6pt] Department of Civil, Architectural, Environmental Engineering, The University of Texas at Austin, Austin, TX 78712, USA (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-19004.txt.
Introduction
Ahmad Yehia and six colleagues release EgoTraj, an egocentric dataset of human walking shot on Meta Quest Pro headsets in city streets. Forecasting walking paths from a first-person view matters for humanoid robots, wearables and assistive guidance, yet egocentric path data have been scarce. EgoTraj holds 75 sequences from wearers, each pairing synced video with head poses, three-dimensional eye gaze vectors and scene labels. Unlike typical sets, it captures long, self-directed routes with varied wearers, and benchmarks show gaze, scene and motion cues each helping forecast quality.
Catalog record
| CREATOR | Ahmad Yehia, Abduallah Mohamed, Tianyi Wang, Jiseop Byeon, Kun Qian, [3pt] Junfeng Jiao, Christian Claudel, [6pt] Department of Civil, Architectural, Environmental Engineering, The University of Texas at Austin, Austin, TX 78712, USA |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59231 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.19004
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
EgoTraj: Real-World Egocentric Human Trajectory Dataset for Multimodal Prediction Ahmad Yehia, Abduallah Mohamed, Tianyi Wang, Jiseop Byeon, Kun Qian, [3pt] Junfeng Jiao, Christian Claudel, [6pt] Department of Civil, Architectural, Environmental Engineering, The University of Texas at Austin, Austin, TX 78712, USA Abstract Accurately forecasting human trajectories from an egocentric perspective plays a central role in applications such as humanoid robotics, wearable sensing systems, and assistive navigation. However, progress in this direction remains limited due to the scarcity of egocentric trajectory datasets collected in real-world environments. Addressing this need, we introduce EgoTraj, an egocentric multimodal open dataset recorded using Meta Quest Pro (MQPro). EgoTraj contains 75 sequences of human navigation collected from multiple MQPro wearers in real-world urban environments. Each recording provides synchronized RGB video along with ground-truth data, including continuous time-synchronized 6-degree-of-freedom head poses, per-frame 3D eye gaze vectors, scene annotations. To the best of our knowledge, EgoTraj differs from typical egocentric trajectory datasets by capturing long-horizon, self-directed navigation across diverse urban routes with broad participant diversity. To demonstrate the potential of the dataset, we benchmark several state-of-the-art methods for egocentric trajectory prediction and conduct ablation studies to analyze the contributions of gaze, scene, and motion cues. The results highlight the utility of EgoTraj for AR-based perception, navigation, and assistive systems. The EgoTraj dataset, code, and EgoViz Dashboard are publicly available at https://github.com/yehiahmad/EgoTraj . K eywords Egocentric trajectory prediction \cdot Multimodal dataset \cdot Augmented reality \cdot Pedestrian navigation \cdot Gaze tracking Figure 1: Overview of EgoTraj. (a) Protocol design: the Meta Quest Pro headset records synchronized RGB video, 6DoF head pose, and gaze signals during in-the-wild navigation. (b) Multimodal EgoTraj capture: representative egocentric frames from a crosswalk navigation scenario with multiple social interactions. (c) Analysis and applications: the dataset supports downstream tasks such as assistive navigation for blind and visually impaired users. 1 Introduction Egocentric perception forms the foundation of human navigation behavior [ 44 ] . Humans naturally perceive and interpret their surroundings from an egocentric viewpoint to guide decisions about subsequent motions [ 4 ] . Replicating this effortless human capability in machines, however, remains a long-standing challenge in embodied AI and computer vision [ 21 ] . Learning such perceptual cues plays a pivotal role in forecasting the trajectory of humanoid robots [ 29 , 16 , 25 ] , AR-VR systems [ 5 ] , and autonomous vehicles [ 45 , 11 , 12 ] . At the same time, recent advances in egocentric vision and wearable sensing technologies have facilitated the development of assistive wayfinding systems, including blind navigation [ 37 ] , object echo-location [ 36 ] , and personalized object recognition [ 14 ] . As augmented reality (AR) technologies evolve, assistive systems increasingly leverage egocentric observations to interpret the user’s surroundings and provide context-aware guidance for subsequent actions. For instance, by modeling navigation behaviors learned from sighted individuals, forecasting the short-term trajectory can help blind or visually impaired users navigate more independently and safely in social spaces [ 30 ] . Moreover, using synchronized multimodal sensors, predicted trajectories can further be conveyed to the headset users through visual overlays [ 44 ] , haptic feedback [ 38 ] , or auditory instructions [ 7 ] . Beyond assistive applications, insights from human navigation behavior provide valuable guidance for social-aware robotic navigation. Humans naturally adhere to implicit social conventions when navigating crowded environments, such as maintaining interpersonal distance, yielding in constrained spaces, and adjusting motion based on gaze and attention cues. By modeling these behaviors, mobile robots [ 40 ] and humanoid robots [ 30 ] can learn navigation strategies that are not only collision-free but also socially compliant, enabling safer operation in human-centered environments. Existing human trajectory prediction methods have made remarkable progress; however, this progress largely relies on datasets captured from bird’s-eye viewpoints (BEV) or static cameras [ 1 , 23 ] . Such datasets capture only externally observable motion and fail to model how humans perceive, plan, and initiate their motion from a first-person perspective. Consequently, most trajectory forecasting methods [ 1 , 34 ] rely primarily on motion history and scene context rather than intention-related cues. In contrast, egocentric vision provides direct access to how humans perceive their surroundings and navigate through them. Over the last two decades, egocentric vision research has produced large-scale datasets spanning diverse activities and environments [ 8 , 26 , 19 , 9 ] . However, these datasets are primarily designed for action recognition and video understanding, with limited emphasis on forecasting the wearer’s trajectory for navigation. As a result, several works have introduced egocentric datasets for trajectory prediction [ 25 , 31 ] . Nevertheless, many of these egocentric trajectory datasets remain limited in scale, gathered from a single participant, restricted to indoor environments, or lacking synchronized gaze data necessary for intention-aware trajectory forecasting. Critically, only a few leverage AR-style sensing devices, which introduce realistic challenges such as rapid ego-motion, viewpoint changes, unstable visual observations, and low-power sensing hardware. Motivated by the lack of real-world egocentric trajectory datasets, we introduce EgoTraj, a multimodal dataset designed to advance research in egocentric trajectory forecasting and assistive AR navigation. To the best of our knowledge, EgoTraj is the first large-scale egocentric trajectory dataset to jointly provide synchronized 6DoF head pose, per-frame 3D gaze vectors, egocentric RGB video, and scene annotations, collected from various participants navigating self-chosen routes across pedestrian crossings, crowded sidewalks, and busy streets. The main contributions of this work are summarized as follows: • EgoTraj Dataset. We introduce a large-scale multimodal dataset for egocentric trajectory forecasting collected by Meta Quest Pro (MQPro) wearers navigating in real-world traffic environments, containing synchronized 6DoF head pose, gaze tracking data, egocentric RGB video, and scene annotations. • Benchmark and Evaluation Protocol. We establish a standardized benchmark for egocentric trajectory forecasting by evaluating several state-of-the-art prediction models for both qualitative and quantitative analyses. • Multimodal Analysis. Through ablation studies, we analyze the impact of gaze, social interactions, and scene context on trajectory prediction under realistic egocentric conditions. 2 Related Work 2.1 Non-Egocentric Human Trajectory Datasets Most human trajectory prediction studies rely on a third-person perspective, where scenes are captured either from a BEV or stationary cameras. In particular, the ETH [ 28 ] and UCY [ 18 ] datasets have become the dominant benchmarks for trajectory prediction, with a total of 1,536 tracked pedestrians covering trajectory patterns such as people walking together, crossing paths, and forming or separating into groups. However, these benchmarks remain limited in scene diversity and restricted to pedestrian-only trajectories. To address these limitations, the Stanford Drone Dataset [ 33 ] and inD [ 2 ] expanded dataset scale using drone footage to capture the movements of multiple road users, including pedestrians, bicyclists, skateboarders, and vehicles. In parallel, the autonomous driving domain introduced large-scale ego-vehicle datasets such as nuScenes [ 3 ] , Waymo Open [ 6 ] , and JRDB [ 22 ] . Despite their scale and diversity, these datasets remain fundamentally third-person in nature. While they accurately capture where pedestrians move, they do not model how individuals perceive the scene or how gaze and visual attention shape future actions. This limitation motivates the development of egocentric trajectory datasets that integrate motion with perceptual