VISTA: A Generative Egocentric Video Framework for Daily Assistance
Yu-Hsiang Liu, Yu-Chien Tang, An-Zi Yen · 2026
Read VISTA: A Generative Egocentric Video Framework for Daily Assistance by Yu-Hsiang Liu, Yu-Chien Tang, An-Zi Yen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-10579.txt.
Introduction
Yu-Hsiang Liu and two colleagues generate egocentric video to train assistant AI agents. Training agents to help humans daily, from household to safety cases, requires large visual data. Capturing such scenes in real life is often difficult, costly, or unsafe. Simulators lack the visual fidelity to transfer learned behaviors to real life. Their 2026 paper introduces VISTA.
VISTA is a video system producing vivid egocentric videos for training AI agents. It uses a script pipeline with reverse reasoning to create diverse intervention modes. Scenarios span two autonomy levels: reactive and proactive. In reactive modes, the user requests help from the agent. In proactive modes, the agent steps in unasked. Explicit types cover users who know help is needed but do not address the agent. Implicit types cover agents intervening before the user realizes help is needed. VISTA lets users customize video benchmarks for daily tasks.
Catalog record
| CREATOR | Yu-Hsiang Liu, Yu-Chien Tang, An-Zi Yen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.10579
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
VISTA: A Generative Egocentric Video Framework for Daily Assistance Yu-Hsiang Liu, Yu-Chien Tang, An-Zi Yen Abstract Training AI agents to proactively assist humans in daily activities, from routine household tasks to urgent safety situations, requires large-scale visual data. However, capturing such scenarios in the real world is often difficult, costly, or unsafe, and physics-based simulators lack the visual fidelity needed to transfer learned behaviors to real settings. Therefore, we introduce VISTA, a video synthesis system that produces high-fidelity egocentric videos as training and evaluation data for AI agents. VISTA employs a 5-step script generation pipeline with causal reverse reasoning to create diverse, logically grounded intervention modes. These scenarios span two levels of agent autonomy: reactive and proactive. In reactive modes, the user explicitly asks the agent for help. In proactive modes, the agent offers help without receiving a direct request. We further divide proactive modes into explicit and implicit types. In explicit proactive scenarios, the user is aware of needing help but does not directly address the agent. In implicit proactive scenarios, the agent intervenes before the user even realizes that help is needed. VISTA allows users to customize and refine scenarios to generate video benchmarks for daily tasks, offering a scalable and controllable alternative to real-world data collection for training and evaluating AI agents in realistic environments. 1 Introduction Recent breakthroughs in Video Foundation Models (VFMs) have demonstrated an unprecedented ability to synthesize photorealistic and temporally coherent video content ( 11 ) . While these advancements have primarily been showcased in creative industries, they offer a transformative potential for AI agent evaluation as Generative Simulators . Traditional benchmarks in Embodied AI often rely on physics-based simulators such as AI2-THOR or Habitat to test agent interactions ( 7 ; 14 ) . However, these platforms frequently suffer from a significant sim-to-real gap due to their limited visual assets and simplified environmental dynamics, which fail to capture the visual complexity and unpredictability of the real world. Recent work has also shown that video world simulators can evaluate robot policies under nominal, OOD, and safety stress tests, further motivating generative benchmarks that retain visual realism ( 16 ) . In parallel, recent video-generation and video-reasoning evaluation suites ( 6 ; 15 ; 4 ; 10 ) have highlighted the need for rigorous, multi-dimensional assessment beyond purely visual quality. Figure 1: Reactive and Proactive Assistance Modes in VISTA. Left: reactive mode, where the user makes a request and the agent responds. Right: proactive mode, where the agent intervenes without a specific request, covering both explicit-need and implicit-need cases. Figure 2: Examples of reactive and proactive assistance modes in VISTA. Each row shows a short temporal slice of the same hazard scenario. From top to bottom, the rows illustrate reactive, explicit proactive, and implicit proactive assistance. Reactive assistance responds to a direct user request, while proactive assistance is provided without one and can be either explicit or implicit depending on the user’s awareness of the need for help. Conversely, acquiring naturalistic egocentric (first-person) data through real-world recording is prohibitively expensive and difficult to scale. Standard procedures require human participants to wear specialized recording devices, such as GoPro cameras or smart glasses, to perform daily tasks over extended periods, as seen in large-scale egocentric datasets like Ego-Exo4D ( 2 ) . Recent efforts have expanded egocentric supervision with procedural ego-exo alignment, action scene graphs, and large-scale egocentric generation data, but they remain costly and limited for rare-risk event ( 5 ; 12 ; 17 ; 8 ) . Moreover, evaluating agents in high-risk or emergency scenarios such as household accidents (e.g., fires, glass breakage) or medical emergencies (e.g., sudden falls, seizures) is practically impossible in reality due to severe ethical concerns and physical safety risks. These limitations leave a critical gap in testing how AI agents should proactively intervene in diverse and hazardous situations. To address these challenges, we present VISTA ( Vi deo S ynthesis for T raining A gents), an end-to-end system for generating and evaluating proactive agent behaviors. VISTA shifts the paradigm from physical recording to generative synthesis , enabling researchers to create a wide array of interaction scenarios through a structured Script-to-Video workflow. By combining open-world reasoning from Large Language Models (LLMs) with high-fidelity rendering from VFMs, VISTA produces realistic first-person videos that include rare, long-tail, and high-stakes events without physical risk ( 11 ) . The core contribution of VISTA lies in its systematic taxonomy of assistance modes, organized in two levels: • Reactive Mode: Situations involving direct, natural language instructions or clear calls for assistance (e.g., “Help me find my keys”). • Proactive Mode: (1) Explicit Proactive Scenarios: Situations where the user is aware of needing assistance but does not directly request it from the agent, for example by thinking aloud or speaking to themselves. (2) Implicit Proactive Scenarios: Situations where the user is not aware of needing assistance, and the agent intervenes before the need is recognized (e.g., noticing a user struggling to open a jar). Figure 2 presents the examples of these assistance modes. In the reactive scenarios, the user notices the spilled drink near the power strip and directly asks what to do. In the explicit proactive scenarios, the user knocks over the drink and realizes the mistake, but does not directly request help. In the implicit proactive scenarios, the user carelessly places the drink near the table edge above the power strip and continues reading, unaware of the potential danger. VISTA is designed to generate such videos for training and evaluating agent behavior across different assistance modes. In sum, our contributions are as follows: (1) We introduce a collaborative interface that bridges the gap between script generation and video synthesis, enabling researchers to iteratively define and render precise safety-critical scenarios and human-environment interactions. 1 1 1 VISTA: https://huggingface.co/spaces/NLPLAB-VISTA-research/VISTA-Demo 2 2 2 https://www.youtube.com/watch?v=Xeqv85FR44g (2) We propose a comprehensive, multi-dimensional assessment pipeline that leverages Large Multimodal Models to evaluate an agent’s contextual reasoning and situational awareness in real-time settings. (3) Our results suggest that VISTA provides a scalable, safe, and customizable simulation sandbox for developing and evaluating AI assistants. 2 Related Work This section situates VISTA within the broader landscape of video generation and embodied AI, focusing on generative world models, egocentric vision, and causal reasoning in data generation. 2.1 Generative World Models Frontier video generation models are increasingly framed as world simulators for embodied agents ( 11 ) , enabling safety stress tests ( 16 ) and policy evaluation ( 9 ) . Concurrently, video generation and reasoning benchmarks have improved metric coverage for visual fidelity and instruction following ( 6 ; 15 ; 4 ; 10 ) . However, these models primarily target open-ended generation rather than the precise control required for evaluating proactive assistance. VISTA addresses this by imposing a strict, structured script-generation pipeline that constrains video outputs to be logically consistent and physically plausible. 2.2 Egocentric Vision and Synthesis Understanding human activity from a first-person perspective is essential for proactive assistants. This research is supported by large-scale datasets capturing skilled activities ( 2 ; 5 ) , action scene graphs ( 12 ) , 4D reconstructions ( 8 ) , and fine-grained reasoning benchmarks ( 13 ) . Because collecting dangerous or rare