EgoReAct: Ego centric Video-Driven 3D Human React ion Generation

Libo Zhang, Zekun Li, Tianyu Li, Zeyu Cao, Xiaoxiao Long, Wenjia Wang, Jingbo Wang, Yuan Liu, Wenping Wang, Daquan Zhou, Taku Komura, Zhiyang Dou, [0.6em] THU · 2025

Read EgoReAct: Ego centric Video-Driven 3D Human React ion Generation by Libo Zhang, Zekun Li, Tianyu Li, Zeyu Cao, Xiaoxiao Long, Wenjia Wang, Jingbo Wang, Yuan Liu, Wenping Wang, Daquan Zhou, Taku Komura, Zhiyang Dou, [0.6em] THU (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-22808.txt.

Introduction

Libo Zhang and a twelve-author team address a gap in motion generation: producing a person's physical reaction to what they themselves are seeing. Their starting point is data. Existing datasets pair reaction motion with video that does not line up in space. Moving reactions were matched to fixed-camera footage. So the team built the Human Reaction Dataset with properly aligned egocentric video and reaction motion.

On that data they train EgoReAct, described as the first autoregressive system that generates three-dimensionally aligned human reactions from a first-person video stream in real time. Motion is first compressed by a vector-quantized autoencoder, then a generative transformer predicts reactions while using depth estimates and head movement to stay grounded in the scene. Experiments show stronger realism, spatial consistency, and speed than earlier methods, while each frame is generated strictly from past input.

Catalog record

CREATORLibo Zhang, Zekun Li, Tianyu Li, Zeyu Cao, Xiaoxiao Long, Wenjia Wang, Jingbo Wang, Yuan Liu, Wenping Wang, Daquan Zhou, Taku Komura, Zhiyang Dou, [0.6em] THU
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-22808

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

EgoReAct: Ego centric Video-Driven 3D Human React ion Generation Libo Zhang, Zekun Li, Tianyu Li, Zeyu Cao, Xiaoxiao Long, Wenjia Wang, Jingbo Wang, Yuan Liu, Wenping Wang, Daquan Zhou, Taku Komura, Zhiyang Dou, [0.6em] THU Abstract Humans exhibit adaptive, context-sensitive responses to egocentric visual input. However, faithfully modeling such reactions from egocentric video remains challenging due to the dual requirements of strictly causal generation and precise 3D spatial alignment. To tackle this problem, we first construct the Human Reaction Dataset ( HRD ) to address data scarcity and misalignment by building a spatially aligned egocentric video–reaction dataset, as existing datasets (e.g., ViMo) suffer from significant spatial inconsistency between the egocentric video and reaction motion, e.g., dynamically moving motions are always paired with fixed-camera videos. Leveraging HRD, we present EgoReAct , the first autoregressive framework that generates 3D-aligned human reaction motions from egocentric video streams in real-time. We first compress the reaction motion into a compact yet expressive latent space via a Vector Quantised-Variational AutoEncoder and then train a Generative Pre-trained Transformer for reaction generation from the visual input. EgoReAct incorporates 3D dynamic features, i.e., metric depth, and head dynamics during the generation, which effectively enhance spatial grounding. Extensive experiments demonstrate that EgoReAct achieves remarkably higher realism, spatial consistency, and generation efficiency compared with prior methods, while maintaining strict causality during generation. We will release code, models, and data upon acceptance. Figure 1 : EgoReAct takes streaming egocentric video as input and synthesizes spatially grounded, realistic human reaction motions in real time, enabling responsive full-body behaviors that are tightly coupled with the ongoing egocentric observations. † † footnotetext: § , † denote work completed during Summer 2025 and corresponding authors, respectively. 1 Introduction Faithful modeling human behavior in response to various input conditions like environmental dynamics remains a long-standing and fundamental challenge, with broad implications for robotics 28 ; 62 ; 21 , computer vision 5 ; 11 ; 61 , and computer graphics 25 ; 57 . Different from previous efforts in human motion generation that conditioned on modalities like textual instructions 42 ; 63 ; 17 ; 65 , another person’s motion 9 ; 30 ; 41 ; 52 , or spatial audio 56 . This paper focuses on high-quality reaction motion synthesis with ego-centric perception, i.e . RGB streaming video 1 1 1 By RGB streaming video, we emphasize causal, per-frame processing, i.e., the model has no access to future context. , as humans inherently perceive and react to the world from their own first-person viewpoint to a large extent–one of the most intuitive and general paradigm for modeling reactions. However, collecting large-scale real world ego-centric based reaction data is inherently difficult, as it requires precise synchronization between camera motion and the actor’s body movement, especially the head trajectory which is a principal cue for the egocentric view. This remains a major obstacle in learn a model to generate human reactions from ego-centric perception. A very recent effort, the ViMo dataset 59 , provides manually paired videos and human reaction motions. However, the quality and diversity of the data are highly limited, which restricts their effectiveness for model training. For example, its videos are mostly captured or generated with a fixed camera. In contrast, the reaction motions involve dynamic movement, which causes ambiguity and lacks the spatial alignment that characterises true ego-centric perception, as illustrated in Fig. 5 . In this paper, we create a more spatially aligned dataset named HRD through an automatic data generation pipeline ( Section 3 ). This scalable synthesis strategy enables the creation of a large, diverse, and spatially aligned dataset that supports data-driven learning of realistic, spatially grounded reactions. Leveraging HRD, we introduce EgoReAct, the first autoregressive framework for this task, capable of generating spatially grounded human reaction motions from egocentric videos in real time. To achieve both efficient and high-quality 3D motion synthesis, we formulate the generation process within a causal autoregressive paradigm. Specifically, we adopt a two-stage design. In the first stage, a VQ-VAE is learned to discretize continuous motion sequences into compact code indices. In the second stage, we train an autoregressive Transformer to predict the code index of the current motion frame based on historical context and the latest visual observation. This design enables the model to generate reaction motions sequentially and causally, updating its predictions as new visual observations arrive in a streaming manner. Note that prior work 59 assumes access to the entire video—including future frames—to one-shot generate the reaction sequence, which violates causality in egocentric settings requiring online synthesis. Instead of relying solely on 2D visual semantics from egocentric videos, which neglects the crucial spatial alignment between the generated reaction motions and the ego perception, we further enhance the generation process with 3D dynamic features by leveraging the power of vision foundation models. Specifically, given a steaming RGB video, our method first extracts rich 2D visual semantics using a pretrained foundation vision model 34 , which guide the autoregressive generation of plausible human reactions. Building upon this 2D perception, we further incorporate 3D dynamic cues—including the metric depth estimated from the video 8 and head dynamics as an ego-motion signal—to achieve spatially grounded and geometrically consistent motion synthesis. These complementary geometric signals enable the synthesized motions to remain consistent with the egocentric visual context within a unified 3D coordinate frame. Based on qualitative and quantitative results, our framework produces realistic, time-consistent, and spatially grounded human reactions that more faithfully emulate human perceptual behavior in dynamic real-world environments, achieving 0.456 vs. 0.661 on FID and 21.33ms vs 400.56 ms on first frame latency compared to the SOTA baseline 59 . Our contributions are summarized as follows: • We construct HRD dataset that contains diverse and spatially aligned egocentric video–reaction motion pairs, providing geometry-consistent supervision for training and evaluation. • We propose EgoReAct, the first autoregressive framework for real-time, 3D-aligned human reaction generation from egocentric videos. • Comprehensive experiments demonstrate the effectiveness of our framework, achieving more efficient and spatially grounded reaction motion generation. 2 Related Work Figure 2 : The automated pipeline for generating the Spatially Aligned Human Reaction Dataset (HRD). Given a scene caption, we first employ LLMs to produce video and motion prompts, then generate egocentric videos and reaction motions through text-driven generation, followed by spatial alignment via camera trajectory control. 2.1 Conditional Motion Generation Conditional motion generation aims to synthesize human motions guided by external signals such as text 42 ; 63 ; 65 ; 17 ; 46 ; 20 ; 31 ; 14 ; 10 ; 46 , audio 27 ; 44 ; 1 ; 56 , action 19 ; 54 ; 6 , motion history 15 ; 7 ; 40 ; 50 ; 4 , etc. Among various conditioning modalities, text-to-motion has been most extensively explored. Human Motion Diffusion Model 42 first introduce diffusion models into motion synthesis, achieving high realism and temporal coherence but at the cost of slow sampling. EMDM 65 further improve efficiency by accelerating the diffusion process, enabling fast yet high-quality motion generation. T2M-GPT 63 reformulate this task in a discrete latent space using VQ-VAE and a GPT-style transformer, enabling autoregressive text-conditioned motion generation with improved semantic alignment. Furthermore, recent approaches 17 ; 24 leverage masked motion

Read the complete text