DuoMo: Dual Motion Diffusion for World-Space Human Reconstruction

Yufu Wang, Evonne Ng, Soyong Shin, Rawal Khirodkar, Yuan Dong, Zhaoen Su, Jinhyung Park, Kris Kitani, Alexander Richard, Fabian Prada, Michael Zollhöfer, [0.5em] Meta Reality Labs University of Pennsylvania Carnegie Mellon University · 2026

Read DuoMo: Dual Motion Diffusion for World-Space Human Reconstruction by Yufu Wang, Evonne Ng, Soyong Shin, Rawal Khirodkar, Yuan Dong, Zhaoen Su, Jinhyung Park, Kris Kitani, Alexander Richard, Fabian Prada, Michael Zollhöfer, [0.5em] Meta Reality Labs University of Pennsylvania Carnegie Mellon University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-03265.txt.

Introduction

Yufu Wang, Evonne Ng, Soyong Shin, and a large team reconstruct human motion in world coordinates from ordinary, messy video. Casual footage brings noise, gaps, and moving cameras, so recovered motion must generalize while staying globally consistent. Their 2026 method, DuoMo, splits the job between two diffusion models.

The first model estimates motion in camera space from the video. The second lifts that estimate into world space and refines it for global coherence. The system outputs mesh vertices directly, skipping parametric body models entirely. On two standard benchmarks, world-space reconstruction error fell by sixteen and thirty percent against prior state of the art, while foot skating stayed low. The factorization also completes motion hidden by occlusion.

Catalog record

CREATORYufu Wang, Evonne Ng, Soyong Shin, Rawal Khirodkar, Yuan Dong, Zhaoen Su, Jinhyung Park, Kris Kitani, Alexander Richard, Fabian Prada, Michael Zollhöfer, [0.5em] Meta Reality Labs University of Pennsylvania Carnegie Mellon University
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47261
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.03265

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DuoMo: Dual Motion Diffusion for World-Space Human Reconstruction Yufu Wang, Evonne Ng, Soyong Shin, Rawal Khirodkar, Yuan Dong, Zhaoen Su, Jinhyung Park, Kris Kitani, Alexander Richard, Fabian Prada, Michael Zollhöfer, [0.5em] Meta Reality Labs University of Pennsylvania Carnegie Mellon University Abstract We present DuoMo, a generative method that recovers human motion in world-space coordinates from unconstrained videos with noisy or incomplete observations. Reconstructing such motion requires solving a fundamental trade-off: generalizing from diverse and noisy video inputs while maintaining global motion consistency . Our approach addresses this problem by factorizing motion learning into two diffusion models. The camera-space model first estimates motion from videos in camera coordinates. The world-space model then lifts this initial estimate into world coordinates and refines it to be globally consistent. Together, the two models can reconstruct motion across diverse scenes and trajectories, even from highly noisy or incomplete observations. Moreover, our formulation is general, generating the motion of mesh vertices directly and bypassing parametric models. DuoMo achieves state-of-the-art performance. On EMDB, our method obtains a 16% reduction in world-space reconstruction error while maintaining low foot skating. On RICH, it obtains a 30% reduction in world-space error. Project page: yufu-wang.github.io/duomo/ Figure 1 : Dual Motion Diffusion (DuoMo) recovers world-space human motion from unconstrained monocular videos. Our two-prior approach reconstructs accurate world-space motion (left), completes motion under occlusion or missing observation (middle), and generates scene-consistent motion with guided sampling (right). Moreover, our method outputs the mesh vertices directly without a SMPL model. 1 Introduction Perceiving human motion within the 3D world is a fundamental aspect of understanding human behavior Blake and Shiffrar 2007 ; McConnell 1974 . When viewing a video, humans effortlessly build a coherent mental model of the scene Peng et al. 2018 : we distinguish the subject’s movement from the camera’s motion, track their trajectories through the environment, and infer their interaction with the world. This perception involves seamlessly integrating a local, view-dependent understanding of a person’s pose with a global, persistent model of the world and the motion in it. Replicating this capability in computer vision, however, remains a significant challenge, especially when dealing with unconstrained videos captured by moving cameras. The focus of human motion modeling in computer vision has recently shifted from analyzing isolated pose sequences to recovering motion grounded in a consistent world coordinate system. This advancement is supported by progress in camera motion estimation and world-space motion capture data. However, current methods face a fundamental trade-off between generalizability and global consistency. On one hand, end-to-end models Shin et al. 2023 ; Shen et al. 2024 ; Li et al. 2025 can learn powerful priors to predict human motion in world coordinates. However, they often fail to generalize in complex scenes, as they are limited by the motion diversity of the studio-captured data. On the other hand, approaches that first model pose in the camera coordinates and then “lift” the motion into the world using camera parameters are more robust to diverse motion in the wild Wang et al. 2024 ; Wang et al. 2025 . However, their motion priors are inherently local and often too weak to enforce physical plausibility and consistency in world coordinates. To address this trade-off, we introduce Dual Motion Diffusion (DuoMo), a method that factors the problem into a two-stage generative process. Instead of committing to a single representation, our approach leverages the strengths of both camera-space and world-space motion modeling by training two independent generative models: — a model for camera-space motion learns a generalizable prior that reconstructs human motion from the camera’s perspective, and — a model for world-space motion learns a prior of plausible and consistent human motion in a global coordinate system. Our key insight lies in how we connect these two models. Instead of forcing a single, end-to-end model to learn this complex geometric relationship, we inject this known geometric principle directly into the pipeline. Specifically, the camera-space model’s output is explicitly lifted into the world coordinates using the estimated camera motion. This design choice allows us to effectively structure our system as a two-stage generative process. The camera-space model acts as the initial estimation stage, focusing on interpreting video frames and reconstructing motion with respect to the viewpoint. The explicit lifting step transforms its output into world coordinates, which become a noisy proposal for the world-space motion. All sources of error — from camera pose estimates, depth ambiguity, and the first model’s own imperfections — are exposed as noise in the proposal. The world-space model then functions as a global refinement stage, whose task is to take this noisy, incomplete proposal and refine it to be physically plausible and globally consistent. Our second insight lies in how we define the coordinate system for our world-space model. Many existing methods Rempe et al. 2021 ; Tevet et al. 2023 ; Zhang et al. 2024b are trained to operate directly in the canonical coordinate system provided by studio capture datasets (e.g., a lab space where the Y-axis is ‘up’ and the ground is flat at zero height). To use such a model, a reconstruction system must align its camera-space human motion with this canonical space. This alignment requires estimating the camera’s pose relative to the scene’s ground plane—a step that is ambiguous, error-prone, and often impossible for in-the-wild videos on varied terrain like hills or stairs. We sidestep this alignment problem entirely. DuoMo’s world-space model is not tied to a fixed, canonical coordinate system. Instead, we define the world coordinates for each video relative to its starting camera pose. Our world-space model is then trained to denoise motion directly within these diverse, per-video coordinate systems, without any canonical transformation. This approach is not just simpler; it allows our model to reconstruct plausible motion in complex, in-the-wild scenes. Finally, our approach is general with respect to the motion representation. Rather than learning to generate the low-dimensional parameters of a body model like SMPL Loper et al. 2023 , our diffusion models generate the motion of the mesh vertices themselves. This shows that our architecture can model geometric motion directly, suggesting a more general path for modeling motion in other categories Sárándi and Pons-Moll 2024 . We conduct experiments on EMDB Kaufmann et al. 2023b , RICH Huang et al. 2022a and Egobody Zhang et al. 2022b , and demonstrate substantial improvement in world-space motion accuracy and robustness. To summarize, our main contributions are: • DuoMo, a two-stage diffusion method that decouples human motion reconstruction into a camera-space estimation stage and a world-space refinement stage; • a world-space motion model trained to denoise motion in per-video coordinate systems, making it robust to in-the-wild scenes; and • an architecture that generates the motion of mesh vertices directly, without relying on parametric body models. 2 Related work Human motion reconstruction. Reconstructing 3D human pose and shape from monocular images Bogo et al. 2016 ; Kanazawa et al. 2018 ; Kolotouros et al. 2019a ; Kocabas et al. 2021 ; Sun et al. 2022 ; Li et al. 2021a ; Zhang et al. 2021 ; Li et al. 2022b and videos Arnab et al. 2019 ; Huang et al. 2017 ; Xiang et al. 2019 ; Kocabas et al. 2020 ; Choi et al. 2021 has

Read the complete text