EgoLive: A Large-Scale Egocentric Dataset from Real-World Human Tasks
2026
Read EgoLive: A Large-Scale Egocentric Dataset from Real-World Human Tasks (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-23570.txt.
Introduction
Li and colleagues present EgoLive, a large video dataset that teaches robots from real human routines rather than lab teleoperation. Filmed with a custom head-mounted device in everyday settings such as homes and shops, it is the largest openly available annotated set focused on task-based work. The team argues its scale, annotation quality, and diversity will speed progress toward robots that generalize.
Catalog record
| YEAR | 2026 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33386 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.23570
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
EgoLive: A Large-Scale Egocentric Dataset from Real-World Human Tasks Abstract The advancement of robot learning is currently hindered by the scarcity of large-scale, high-quality datasets. While established data collection methods such as teleoperation and universal manipulation interfaces dominate current datasets, they suffer from inherent limitations in scalability and real-world deployability. Human egocentric video collection, by contrast, has emerged as a promising approach to enable scalable, natural and in-the-wild data collection. As such, we present EgoLive , a large-scale, high-quality egocentric dataset designed explicitly for robot manipulation learning. EgoLive establishes three distinctive technical advantages over existing egocentric datasets: first, it represents the largest open-source annotated egocentric dataset focused on real-world task-oriented human routines to date; second, it delivers leading data quality via a customized head-mounted capture device and comprehensive high-precision multi-modal annotations; third, all data is collected exclusively in unconstrained real-world scenarios and encompasses vertical field human working data, including home service, retail, and other practical work scenarios, providing superior diversity and ecological validity. With the introduction of EgoLive , we aim to provide the research community with a scalable, high-quality dataset that accelerates breakthroughs in generalizable robotic models and facilitates the real-world deployment of robot systems. \checkdata [Dataset] https://robotdata-market.jdcloud.com/console/market 1 Introduction The lack of large-scale datasets has emerged as a critical bottleneck in advancing the generalization capabilities of contemporary robot manipulation learning models. While recent progress in pre-training Vision-Language-Action (VLA) models [ 30 , 27 ] has demonstrated remarkable progress in facilitating real-world robot manipulation, the pursuit of enhanced generalization cannot be achieved solely through scaling existing robot datasets. Existing datasets are mainly constrained by two key limitations: one is restricted environmental diversity that fails to capture real-world complexity; the other is poor extensibility that makes large-scale data collection implausible. These constraints collectively hinder the development of truly generalizable embodied systems capable of operating in open-world scenarios. Egocentric data [ 18 , 7 , 36 ] present a promising solution to address current limitations in acquiring data at scale for robot learning. By capturing first-person visual perspectives using head-mounted wearable devices, this approach embodies human-like visual perception in a natural manner and offers several key advantages. Human Learning Inspiration: The egocentric approach advocates human-centric visual perception and learning processes, enabling curiosity-driven exploration similar to infant developments [ 46 , 40 ] and supporting incremental skill acquisition through cumulative first-person experience. In-the-wild Collection: Unlike fixed camera systems, this method allows for unrestricted movement in real-world environments, capturing natural human behaviors and interactions without spatial constraints or significant physical interference on the wearer. Scalability Potential: The relatively simple hardware setup (e.g., single head-mounted camera) enables large-scale data collection across diverse environments and users with low equipment costs. Figure 1 : EgoLive is a large-scale high-quality and diverse egocentric dataset. It records human daily manipulation tasks with stereo vision. We introduce EgoLive, a large-scale egocentric dataset capturing real-world human task demonstrations. It comprises 1,680 hours of high-fidelity stereo video recordings at 60 frames per second, with 65,866 episodes spanning 346 different real-world tasks. Notably, EgoLive exhibits several key characteristics that distinguish it from existing egocentric datasets: • To our knowledge, EgoLive is the largest open-source annotated egocentric dataset for real-world human tasks to date. • The dataset achieves superior quality in two dimensions. Collection specifications : Our custom-designed head-mounted capture device achieves human-like stereo vision with a 130^{\circ} horizontal × 130^{\circ} vertical field of view (FOV), capturing video at 60 frames per second with 2160 × 2160 resolution per camera. Annotation richness and accuracy : It provides comprehensive multi-modal annotations, including 6 -DoF motion tracking, fine-grained semantic segmentation and 3D scene reconstruction, all with high accuracy. • In contrast to existing egocentric datasets, EgoLive is collected entirely in unconstrained real-world environments, offering significantly improved scene diversity and ecological validity for robot learning. 2 Related Work 2.1 Manipulation Data Collection Manipulation data collection falls broadly into three paradigms: real-robot teleoperation , universal manipulation interfaces (UMI), and human egocentric video collection . Real-robot teleoperation systems primarily employ three approaches: isomorphic master-follower control, VR-based systems, and motion capture. Notable implementations include ALOHA [ 49 ] (low-cost synchronized teleoperation via tethered robotic arms), AgiBot [ 1 ] (VR/motion-capture humanoid control), and Tesla Optimus [ 41 ] (wearable motion capture suits). While these methods generate high-fidelity trajectories with strong physical priors for policy learning, they face scalability limitations due to specialized hardware requirements, intensive human involvement, and prohibitive costs for large-scale data collection. Universal Manipulation Interfaces (UMI) [ 10 ] employ instrumented grippers with GoPro cameras to capture human demonstrations, aligning human-robot perceptual views. FastUMI [ 50 ] enhances tracking precision via optimized wrist pose estimation. Recent works like Gen series [ 14 , 15 ] leverage UMI data for scalable manipulation policy training. Although these handheld interfaces enable natural interaction and reduce visual domain gaps, they are typically tailored to specific robot embodiments and thus offer limited cross-platform compatibility. In addition, UMI systems remain difficult to adapt to dexterous hands, further limiting their application scope. Human egocentric video collection offers a scalable, minimal-hardware paradigm for capturing natural interactions through wearable cameras. EgoScale [ 51 ] extracts high-DoF dexterous hand motions from 20 k + hours of in-the-wild videos. EgoDex [ 18 ] enables large-scale bare-hand tracking via head-mounted devices; EgoMimic [ 19 ] implements portable multi-modal data collection through smart glasses. Compared to real-robot teleoperation, this vision-only approach eliminates hardware constraints and spatial limitations, significantly accelerating real-world data acquisition. Unlike physical gripper-based interfaces like UMI, it preserves natural human hand morphology and thereby supports direct collection of high-DoF dexterous manipulation data. 2.2 Human Egocentric Dataset Dataset Scene Scale Resolution FPS Multi-view Motion-tracking Language Annot. Depth Generalist EPIC-KITCHENS-100 [ 11 ] Real-world 100 h 1920*1080 50 ✗ ✗ ✓ ✗ Ego4D [ 16 ] Real-world 3680 h Inconsistent 30 ✗ ✗ ✓ ✗ Manipulation-Centric EgoMimic [ 19 ] Laboratory 55 h 1408*1408 30 ✗ ✓ ✗ ✗ HOI4D [ 24 ] Laboratory 22 h 1920*1080 15 ✗ ✓ ✗ ✓ HOT3D [ 4 ] Laboratory 14 h 1408*1408 30 ✓ ✓ ✗ ✗ EgoDex [ 18 ] Laboratory 829 h 1920*1080 30 ✗ ✓ ✓ ✗ Deployment-Scale Egocentric-10K [ 7 ] Real-world 10k h 1920*1080 30 ✗ ✗ ✗ ✗ Xperience-10M [ 36 ] Real-world 1059 h 512*512 20 ✓ ✓ ✓ ✓ EgoLive (Ours) Real-world 1680 h 2160*2160 60 ✓ ✓ ✓ ✓ Table 1 : Comparison of representative human egocentric datasets relevant to embodied manipulation and human-to-robot transfer. A modality is marked as present only if provided in the main public release of the dataset. EgoLive targets real-world scenes with the second longest collection duration and achieves superior spatiotemporal resolution and annotation comprehensiveness. Existing human egocentric datasets can be categorized into three major trends: Generalist , Manipulation-centric , and Deployment-scale . They reflect different design priorities for first-person data collection, namely semantic breadth, interaction fidelity, and real-world operational coverage. Generalist related works such as EPIC-KITCHENS-100 [ 11 ] and Ego4D [ 16 ] have played a foundational role in egocentric visual understanding. They emphasize large-scale video coverage, rich language supervision, and wide diversity of daily activities and environments. As a result, they have been highly influential for action recognition and general-purpose first-person representation learning. However, these