Rewind : Real-Time Egocentric Whole-Body Motion Diffusion with Exemplar-Based Identity Conditioning

Jihyun Lee, Weipeng Xu, Alexander Richard, Shih-En Wei, Shunsuke Saito, Shaojie Bai, Te-Li Wang, Minhyuk Sung, Tae-Kyun Kim, Jason Saragih · 2025

Read Rewind : Real-Time Egocentric Whole-Body Motion Diffusion with Exemplar-Based Identity Conditioning by Jihyun Lee, Weipeng Xu, Alexander Richard, Shih-En Wei, Shunsuke Saito, Shaojie Bai, Te-Li Wang, Minhyuk Sung, Tae-Kyun Kim, Jason Saragih (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-04956.txt.

Introduction

Jihyun Lee and co-authors present REWIND in a 2025 paper, a one step diffusion model that reads whole body motion from head worn views. Past best methods refine motion in many slow passes, which breaks real time use. A causal, single pass model would open live uses such as telepresence.

Two tricks make the speed possible in their design. A cascaded scheme denoises body then hands, keeping their link in a fast feed forward way. Distillation then squeezes quality into a single denoising step per frame. A tweaked transformer backbone handles motion streams of unseen lengths with grace. Optional identity cues from a few sample poses sharpen the fit further.

Catalog record

CREATORJihyun Lee, Weipeng Xu, Alexander Richard, Shih-En Wei, Shunsuke Saito, Shaojie Bai, Te-Li Wang, Minhyuk Sung, Tae-Kyun Kim, Jason Saragih
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-04956

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Rewind : Real-Time Egocentric Whole-Body Motion Diffusion with Exemplar-Based Identity Conditioning Jihyun Lee, Weipeng Xu, Alexander Richard, Shih-En Wei, Shunsuke Saito, Shaojie Bai, Te-Li Wang, Minhyuk Sung, Tae-Kyun Kim, Jason Saragih Abstract We present Rewind ( R eal-Time E gocentric W hole-Body Mot i o n D iffusion), a one-step diffusion model for real-time, high-fidelity human motion estimation from egocentric image inputs. While an existing method for egocentric whole-body (i.e., body and hands) motion estimation is non-real-time and acausal due to diffusion-based iterative motion refinement to capture correlations between body and hand poses, Rewind operates in a fully causal and real-time manner. To enable real-time inference, we introduce (1) cascaded body-hand denoising diffusion, which effectively models the correlation between egocentric body and hand motions in a fast, feed-forward manner, and (2) diffusion distillation, which enables high-quality motion estimation with a single denoising step. Our denoising diffusion model is based on a modified Transformer architecture, designed to causally model output motions while enhancing generalizability to unseen motion lengths. Additionally, Rewind optionally supports identity-conditioned motion estimation when identity prior is available. To this end, we propose a novel identity conditioning method based on a small set of pose exemplars of the target identity, which further enhances motion estimation quality. Through extensive experiments, we demonstrate that Rewind significantly outperforms the existing baselines both with and without exemplar-based identity conditioning. Figure 1 : Real-time and high-fidelity whole-body motion estimation from stereo egocentric images. We propose Rewind , a novel egocentric image-conditioned diffusion model for high-quality 3D whole-body motion estimation. Rewind is real-time, causal, and generalizable to unseen motion lengths, making it seamlessly applicable for driving photorealistic avatars or meshes. Please refer to the supplementary video, which demonstrates that Rewind estimates significantly more plausible motions compared to existing methods. 1 Introduction Egocentric human motion estimation is essential for delivering immersive and realistic experiences in AR/VR applications, such as gaming and telepresence. For instance, imagine engaging in a conversation with your friend in a virtual environment. The quality of estimated whole-body (i.e., body and hands) motion is crucial in creating a realistic experience in interpersonal communication, while subtle pose changes (e.g., finger poses in Fig. 1b ) can significantly impact the intended message. Developing a live-drivable method that enables accurate and realistic egocentric whole-body motion estimation is thus essential. However, existing egocentric pose or motion estimation methods fall short in achieving the accuracy and speed needed for highly realistic VR/AR experiences. They typically focus on tracking body-only motions [ 59 , 7 , 2 , 56 , 55 , 3 , 26 ] , neglecting the importance of hands in fully capturing the intricacies of human motions [ 57 ] . Directly extending the existing body-only estimation methods for whole-body estimation yields suboptimal results, as body and hands significantly differ in scale in both input images and output motions [ 43 , 6 , 68 , 12 , 36 , 62 ] . To address this challenge, EgoWholeMocap [ 57 ] , the first method for whole-body motion estimation from egocentric images, proposes to leverage specialist models for body and hands. It first performs per-frame pose estimation for body and hands separately, and then refines the output poses using an unconditional whole-body motion prior to model correlations between different body parts. While this approach improves whole-body motion estimation performance, it is non-real-time and acausal (i.e., depends on future information) due to iterative refinement steps using an acausal diffusion-based motion prior. Thus, it cannot be used for real-time egocentric motion tracking applications. In this work, we introduce Rewind ( R eal-Time E gocentric W hole-Body Mot i o n D iffusion), a one-step diffusion model for real-time, high-fidelity human motion estimation from egocentric image inputs. To achieve both fast inference speed and high whole-body motion accuracy, we first introduce cascaded body-hand denoising diffusion (Sec. 3.1 ) , where body motion is sampled first and then hand motion is sampled conditioned on the previously sampled upper body motion. This cascading scheme approximately models the correlation between body and hands in a fast, feed-forward manner (cf. iterative whole-body refinement used in EgoWholeMocap [ 57 ] ) while still inheriting the advantages of specialized body and hand estimation (e.g., effectively handling domain differences). We further argue that this approach is particularly effective for our targeted egocentric image inputs, where hands are often placed outside the field of view or occluded. As hands and upper body poses are known to have meaningful correlations [ 38 ] , conditioning on estimated upper body motion – often provided with more reliable input egocentric observations (e.g., Fig. 1a ) – can effectively reduce hand motion ambiguities. We build these specialist denoising diffusion models based on causal relative-temporal Transformer (Sec. 3.2 ) , which is fully causal and generalizable to unseen motion lengths. We use windowed relative-temporal attention to learn temporal motion features that are invariant to the total sequence length or absolute timesteps, in contrast to motion diffusion models based on vanilla attention (e.g., motion prior used in EgoWholeMocap [ 57 ] ). During network training, we employ diffusion distillation (Sec. 3.3 ) to enable real-time inference ( > 30 FPS) with a single denoising step , while preserving high output motion quality. Not only introducing an effective real-time framework, we take a step further and explore identity -aware motion estimation to further enhance output quality when an additional identity prior is available. To this end, we propose novel exemplar-based identity conditioning (Sec. 3.4 ) , where motion estimation is conditioned on the target identity parameterized by a small set of pose exemplars. While this identity parameterization has not yet been considered in existing works on human pose or motion estimation, we empirically find it to be the most effective compared to widely used identity parameterizations (e.g., height, bone lengths, shape parameters). In the experiments, Rewind achieves state-of-the-art whole-body motion estimation results, both with and without additional identity priors. Please also refer to our supplementary video, where Rewind estimates significantly more plausible motions than the baselines [ 59 , 57 ] , even from challenging egocentric input observations (e.g., occluded or truncated views). 2 Related Work 2.1 Egocentric Body Pose Estimation Recently, various egocentric body pose or motion estimation methods have been proposed for different input domains (e.g., sensors [ 24 , 25 , 9 ] or images [ 30 , 60 , 57 , 59 ] ). Here, we focus on existing methods for estimating the pose of a head-mounted device wearer from image inputs, which are most relevant to our work. These methods can be broadly categorized into two groups based on whether they utilize front-facing or down-facing egocentric cameras. Methods using front-facing egocentric cameras [ 30 , 60 , 37 , 34 , 29 , 16 ] assume that the wearer’s body is not visible from the input viewpoint. Thus, they formulate the problem as a motion generation or inpainting task, conditioned on head-mounted camera poses [ 30 , 60 ] , hand poses [ 60 ] , or the body poses of social interactees [ 37 ] . On the other hand, methods using down-facing egocentric cameras [ 57 , 59 , 7 , 2 , 56 , 55 ,

Read the complete text