PlayerOne: Egocentric World Simulator

Yuanpeng Tu, Hao Luo, Xi Chen, Xiang Bai, Fan Wang, Hengshuang Zhao, [2pt] HKU · 2025

Read PlayerOne: Egocentric World Simulator by Yuanpeng Tu, Hao Luo, Xi Chen, Xiang Bai, Fan Wang, Hengshuang Zhao, [2pt] HKU (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-09995.txt.

Introduction

Yuanpeng Tu and five co-authors build PlayerOne, a system that turns a first-person photo into a world the user can walk through on video. Their paper appeared in two thousand twenty-five. The system takes a scene image from the user's view. A second camera records the user's real body motion from outside.

PlayerOne then grows the scene into a full setting and renders first-person video that matches the motion. Training ran in two stages. Broad video-text pairs taught the basics of first-person views. Paired inside-outside footage then taught the motion link, using an automatic pipeline to build the pairs. A special step feeds in body-part motion with fine control. A joint rebuild of scene and frames keeps long clips consistent. Tests show strong control across many scenes and moves.

Catalog record

CREATORYuanpeng Tu, Hao Luo, Xi Chen, Xiang Bai, Fan Wang, Hengshuang Zhao, [2pt] HKU
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-09995

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PlayerOne: Egocentric World Simulator Yuanpeng Tu, Hao Luo, Xi Chen, Xiang Bai, Fan Wang, Hengshuang Zhao, [2pt] HKU Abstract We introduce PlayerOne, the first egocentric realistic world simulator, facilitating immersive and unrestricted exploration within vividly dynamic environments. Given an egocentric scene image from the user, PlayerOne can accurately construct the corresponding world and generate egocentric videos that are strictly aligned with the real-scene human motion of the user captured by an exocentric camera. PlayerOne is trained in a coarse-to-fine pipeline that first performs pretraining on large-scale egocentric text-video pairs for coarse-level egocentric understanding, followed by finetuning on synchronous motion-video data extracted from egocentric-exocentric video datasets with our automatic construction pipeline. Besides, considering the varying importance of different components, we design a part-disentangled motion injection scheme, enabling precise control of part-level movements. In addition, we devise a joint reconstruction framework that progressively models both the 4D scene and video frames, ensuring scene consistency in the long-form video generation. Experimental results demonstrate its great generalization ability in precise control of varying human movements and world-consistent modeling of diverse scenarios. It marks the first endeavor into egocentric real-world simulation and can pave the way for the community to delve into fresh frontiers of world modeling and its diverse applications. Figure 1: Simulated videos of our PlayerOne . Given an egocentric image as the scene to be explored, we can simulate egocentric immersive videos that are accurately aligned with the user’s motion sequence captured by an exocentric camera. All the users have been anonymized and action videos are shot with the front camera. 1 Introduction World models xiao2025worldmemlongtermconsistentworld ; aether ; wu2022daydreamer ; hafner2019dreamer ; nvidia2025cosmosworldfoundationmodel ; muzero-general have undergone extensive research due to their ability to model environmental dynamics and predict long-term outcomes. Recent breakthroughs in video diffusion models wan2025 ; HaCohen2024LTXVideo ; kong2024hunyuanvideo have revolutionized this domain, enabling the synthesis of high-fidelity, action-conditioned simulations that forecast intricate future states. These advancements empower applications ranging from autonomous navigation in dynamic real-world environments to the creation of immersive, responsive virtual worlds in AAA game development. By bridging the gap between predictive modeling and interactive realism, world simulators are emerging as critical infrastructure for next-generation autonomous systems and game engines, particularly in scenarios requiring real-time adaptation to complex, evolving interactions. Despite significant progress, this topic remains underexplored in existing research. Prior studies guo2025mineworldrealtimeopensourceinteractive ; xiao2025worldmemlongtermconsistentworld ; feng2024matrixinfinitehorizonworldgeneration predominantly focused on simulations within game-like environments, falling short of replicating realistic scenarios. Additionally, in their simulated environments, users are limited to performing predetermined actions ( i.e. , directional movements). Operating within the confines of a constructed world restricts the execution of unrestricted movements as in real-world scenarios. While some initial efforts liang2024wonderlandnavigating3dscenes ; ren2025gen3c3dinformedworldconsistentvideo ; nvidia2025cosmosworldfoundationmodel have been made toward real-world simulation, they mainly contribute to world-consistent generation without human movement control. Consequently, users are reduced to passive spectators within the environment, rather than being active participants. This limitation significantly impacts the user experience, as it prevents the establishment of a genuine connection between the user and the simulated environment. Faced with these challenges, we aim to design an egocentric world foundational framework that enables the user being a freeform adventurer. Given a user-provided egocentric image as the world to be explored, it can enable the user to perform unrestricted human movements real-time captured by an exocentric camera and consistent 4D scene modeling in the simulated world. Specifically, we propose the first realistic egocentric world simulator termed PlayerOne. Starting from a diffusion transformer (DiT) model Peebles2022DiT , we first extract the latent of an egocentric user-input image. Meanwhile, we select the real-world human motions ( i.e. , human pose or keypoints) as our motion representation. Considering the varying importance of different body parts in our task, the human motion sequence is partitioned into three groups ( i.e. , head, hands, feet and body) and fed into our part-disentangled motion injection to generate latents that can enable precise part-wise control. Additionally, we developed a joint scene-frame reconstruction framework that can progressively complete scene point maps during the video generation process to enable scene-consistent generation. The DiT model takes the concatenation of the first frame latent, motion latent, video latent, and the point map latent as input and conducts noising and denoising on both the video and point map latent. Notably, the point map sequence is not required during inference, ensuring practical efficiency. Moreover, to overcome the absence of publicly available datasets, we curate required motion-video pairs from existing egocentric-exocentric datasets using an automated pipeline designed to filter and retain high-quality data. A coarse-to-fine training strategy is also designed to compensate for the data scarcity. The base model is fine-tuned on large-scale egocentric text-video data for coarse-level generation, then refined on our curated dataset to achieve precise motion control and scene modeling. Finally, we distill our trained model yin2025causvid to achieve real-time generation. By integrating these innovations, PlayerOne advances the field of dynamic world modeling. Our contributions are summarized as follows: • We introduce PlayerOne, the first egocentric foundational simulator for realistic worlds, capable of generating video streams with precise control of highly free human motions and world consistency in real-time and exhibiting strong generalization in diverse scenarios. • We design a novel part-disentangled motion injection scheme to enhance fine-grained motion alignment, where a joint scene-frame reconstruction framework is introduced to guarantee world-consistent modeling in long-term video generation as well. • We construct an effective automatic dataset construction pipeline to extract high-quality motion-video pairs from existing egocentric-exocentric datasets, where a coarse-to-fine training scheme is also introduced to compensate for the data scarcity. 2 Related Work Video generation. The rapid development of diffusion models rombach2021highresolution ; ho2020denoising ; zhang2023adding ; latentdiffusion2022 has driven substantial advancements in video generation. Early researchers guo2023animatediff ; chen2023controlavideo adapted existing text-to-image models to enable text-to-video generation to compensate for the limited availability of high-quality video-text datasets. Subsequently, diffusion transformers based frameworks xie2024sana ; Peebles2022DiT ; kong2024hunyuanvideo ; HaCohen2024LTXVideo ; wan2025 ; genmo2024mochi are proposed. When scaling-up training, they enable more highly realistic and temporally coherent generation results. Among them, HunyuanVideo kong2024hunyuanvideo substitutes T5 with a Multimodal Large Language Model. LTX-Video HaCohen2024LTXVideo modifies the VAE decoder to handle the final denoising step and convert latents into pixels. Wan wan2025 introduces a full spatial-temporal attention to ensure computational efficiency. World models. Existing world models hafner2019dreamer ; wu2022daydreamer ; xiao2025worldmemlongtermconsistentworld ; feng2024matrixinfinitehorizonworldgeneration ; nvidia2025cosmosworldfoundationmodel ; muzero-general can be roughly divided into two categories: 1) Agent learning targeted models, 2) World simulation models. For the former, they hafner2019dreamer ; wu2022daydreamer ; muzero-general aim at enhancing policy learning within simulated environments. Among them, Dreamer hafner2019dreamer and DayDreamer wu2022daydreamer tackle long-horizon tasks by leveraging latent imagination. Meanwhile, MuZero muzero-general employs self-play Monte Carlo Tree Search (MCTS). In contrast, world simulation approaches focus on modeling environments by explicitly predicting next states given the current state and action. These methods prioritize enabling human-neural network interaction through high-fidelity rendering, robust control mechanisms. Recent advancements in video generation have further enabled the development of high-quality, controllable world simulations, sparking significant progress in the field of world simulation guo2025mineworldrealtimeopensourceinteractive ; ren2025gen3c3dinformedworldconsistentvideo ; liang2024wonderlandnavigating3dscenes ; xiao2025worldmemlongtermconsistentworld ; feng2024matrixinfinitehorizonworldgeneration ; nvidia2025cosmosworldfoundationmodel ; aether ; cheng2025animegamerinfiniteanimelife . Among these works, WORLDMEM xiao2025worldmemlongtermconsistentworld . The Matrix feng2024matrixinfinitehorizonworldgeneration proposes the first

Read the complete text