EgoExo-Gen: Ego-centric Video Prediction by Watching Exo-centric Videos

Jilan Xu, Yifei Huang, Baoqi Pei, Junlin Hou, Qingqiu Li, Guo Chen, Yuejie Zhang, Rui Feng, Weidi Xie · 2025

Read EgoExo-Gen: Ego-centric Video Prediction by Watching Exo-centric Videos by Jilan Xu, Yifei Huang, Baoqi Pei, Junlin Hou, Qingqiu Li, Guo Chen, Yuejie Zhang, Rui Feng, Weidi Xie (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-11732.txt.

Introduction

Jilan Xu and co-authors tackle predicting first-person video from a third-person view, in this 2025 paper. Given an outside video, a first frame from the wearer's view, and text instructions, the goal is to grow the wearer's future frames. Hand-object contact carries most of the actor's intent.

Their system first predicts masks of future hand-object contact, learned from cross-view matches over space and time. A video diffusion model then grows the frames, guided by those masks. Training masks were auto-built with vision foundation models. Tests on two benchmarks beat prior video predictors, with masks clearly sharpening hands and handled objects.

Catalog record

CREATORJilan Xu, Yifei Huang, Baoqi Pei, Junlin Hou, Qingqiu Li, Guo Chen, Yuejie Zhang, Rui Feng, Weidi Xie
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-11732

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

EgoExo-Gen: Ego-centric Video Prediction by Watching Exo-centric Videos Jilan Xu, Yifei Huang, Baoqi Pei, Junlin Hou, Qingqiu Li, Guo Chen, Yuejie Zhang, Rui Feng, Weidi Xie Abstract Generating videos in the first-person perspective has broad application prospects in the field of augmented reality and embodied intelligence. In this work, we explore the cross-view video prediction task, where given an exo-centric video, the first frame of the corresponding ego-centric video, and textual instructions, the goal is to generate future frames of the ego-centric video. Inspired by the notion that hand-object interactions (HOI) in ego-centric videos represent the primary intentions and actions of the current actor, we present EgoExo-Gen that explicitly models the hand-object dynamics for cross-view video prediction. EgoExo-Gen consists of two stages. First, we design a cross-view HOI mask prediction model that anticipates the HOI masks in future ego-frames by modeling the spatio-temporal ego-exo correspondence. Next, we employ a video diffusion model to predict future ego-frames using the first ego-frame and textual instructions, while incorporating the HOI masks as structural guidance to enhance prediction quality. To facilitate training, we develop an automated pipeline to generate pseudo HOI masks for both ego- and exo-videos by exploiting vision foundation models. Extensive experiments demonstrate that our proposed EgoExo-Gen achieves better prediction performance compared to previous video prediction models on the Ego-Exo4D and H2O benchmark datasets, with the HOI masks significantly improving the generation of hands and interactive objects in the ego-centric videos. 1 1 footnotetext: ∗ Corresponding author. Figure 1: The cross-view video prediction task aims to predict future RGB frames of the ego-centric video, given the first ego-centric frame, a text instruction, and a synchronised exo-centric video. 1 Introduction This paper considers the task of animating an ego-centric frame based on a third-person (exo-centric) video captured simultaneously in the same environment. As illustrated in Fig. 1 , given the first frame of an ego-centric video, a textual instruction, and a synchronised exo-centric video ( Grauman et al., 2024 ; Kwon et al., 2021 ; Jia et al., 2020 ; Sener et al., 2022 ) , the goal is to predict the subsequent frames in ego-view. Exo-centric views typically provide broader environmental context and body kinetics ( Carreira & Zisserman, 2017 ; Miech et al., 2019 ) , but are less focused on fine-grained actions. In contrast, ego-centric views center on the hands of the camera wearer and the interacting objects ( Grauman et al., 2022 ; Damen et al., 2018 ) , which are critical for tasks like manipulation and navigation ( Nair et al., 2022 ; Bharadhwaj et al., 2023 ; Rhinehart & Kitani, 2017 ; Shah et al., 2022 ) . The challenge in cross-view video prediction arises from the significant perspective shift between these views, as the future frames must align with both the contextual environment of the first frame and the actor’s motion as indicated by the textual instruction and the exo-centric video. Bridging these two perspectives offers two key benefits: first , it allows agents to build a robust ego-centric world model from third-person demonstrations, enabling them to translate broader scene information into a first-person perspective ( Huang et al., 2024a ; Luo et al., 2024b ) ; second , it enables the agents to think from the humans’ perspective by aligning their viewpoint with human users, improving their ability to anticipate future actions and make more informed decisions. This capability is particularly valuable in real-time applications such as augmented reality (AR) ( Huang et al., 2018 ; Huang et al., 2024b ) and robotics ( Wang et al., 2023a ) , where agents must synchronize their actions with humans or the environment. By learning to translate exo-centric video into accurate ego-centric frames, agents can become more adaptive when engaging in dynamic, real-world environments. Recent researches in video prediction models have shown tremendous progress ( Xing et al., 2023 ; Chen et al., 2023 ; Ren et al., 2024 ; Guo et al., 2023a ) . These models mainly rely on the first frame and textual instructions as inputs for diffusion models ( Rombach et al., 2022b ; Song et al., 2020 ) , with a primary focus on generating videos from an exo-centric perspective. Regarding ego-centric video prediction, ( Gu et al., 2023 ; Xing et al., 2024 ) have explored the decomposition of text instructions, but they lack customised design for the ego-centric view. In the context of cross-view video generation, specialised models have been developed ( Luo et al., 2024a ; Luo et al., 2024b ) . For example, ( Luo et al., 2024a ) leveraged head trajectory data from ego-centric videos to generate optical flow and occlusion maps in exo-centric videos, while ( Luo et al., 2024b ) estimated hand poses in ego-centric views to guide conditional ego-video generation. However, these approaches rely on 3D scene reconstruction ( Schonberger & Frahm, 2016 ; Tschernezki et al., 2024 ) or precise human annotations ( Kwon et al., 2021 ) , limiting their scalability and generalization ability across diverse scenarios. In this paper, we propose EgoExo-Gen , a cross-view video prediction model that generates future ego-centric video frames by explicitly modeling hand-object dynamics. EgoExo-Gen employs a two-stage approach: first , it predicts semantic hand-object interaction (HOI) masks in ego-centric view, and second , it uses these masks as condition, driving a diffusion model to produce the corresponding RGB frames. For the cross-view HOI mask prediction model, it takes an exo-centric video and the first frame of an ego-centric video as inputs, and predicts HOI masks for future ego-centric frames. To handle the drastic change in viewpoints, we design an ego-exo memory attention module that captures spatio-temporal correspondences between the two views, enabling the model to infer HOI masks even when the current ego-centric frame is not visible. The predicted HOI masks are then integrated into the diffusion model, guiding the generation of accurate future ego-centric frames. To ensure scalable training and minimise reliance on manual annotations, we also develop a fully automated HOI mask annotation pipeline for both ego-centric and exo-centric videos by leveraging powerful vision foundation models ( Khirodkar et al., 2024 ; Kirillov et al., 2023 ; Ravi et al., 2024 ) . This combination of methods allows EgoExo-Gen to generate high-quality ego-centric videos while significantly improving scalability and generalisation across diverse environments. We conduct extensive experiments on the cross-view video benchmark datasets, i.e. , Ego-Exo4D ( Grauman et al., 2024 ) and H2O ( Kwon et al., 2021 ) that include rich and diverse hand-object interactions and shooting environment. Experimental results show that EgoExo-Gen significantly outperforms prior video prediction models ( Chen et al., 2023 ; Ren et al., 2024 ; Gu et al., 2023 ) and improves the quality of predicted videos by leveraging hand and object dynamics. Also, EgoExo-Gen demonstrates strong zero-shot transfer ability on unseen actions and environments. 2 Methodology Figure 2: An overview of EgoExo-Gen. Given an exo-centric video, a text instruction, and the first frame of an ego-centric video, (1) a cross-view mask prediction model first anticipates the hand-object masks of the unobserved future frames; (2) an HOI-aware video diffusion model then predicts future frames of an ego-centric video by incorporating the predicted hand-object masks. 2.1 Problem

Read the complete text