Link to the Past: Tem poral P r o pagation for Fast 3D Human Reconstruction from Monocular Video
Matthew Marchellus, Nadhira Noor, In Kyu Park · 2025
Read Link to the Past: Tem poral P r o pagation for Fast 3D Human Reconstruction from Monocular Video by Matthew Marchellus, Nadhira Noor, In Kyu Park (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-07333.txt.
Introduction
Rebuilding a clothed person in 3D from one camera's video is slow with current tools. Matthew Marchellus and co-authors speed it up by reusing what earlier frames already learned. Their method, TemPoFast3D, keeps a stored copy of the person's look and updates it as the video runs, instead of starting fresh each frame.
The trick is a clean way to map that stored look onto each new pose. The system plugs into existing image-based builders rather than replacing them. In tests, it matched or beat top methods on standard scores while keeping fine texture across poses and outfits. The 2025 paper reports speeds up to twelve frames per second.
Catalog record
| CREATOR | Matthew Marchellus, Nadhira Noor, In Kyu Park |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-07333
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Link to the Past: Tem poral P r o pagation for Fast 3D Human Reconstruction from Monocular Video Matthew Marchellus, Nadhira Noor, In Kyu Park Abstract Fast 3D clothed human reconstruction from monocular video remains a significant challenge in computer vision, particularly in balancing computational efficiency with reconstruction quality. Current approaches are either focused on static image reconstruction but too computationally intensive, or achieve high quality through per-video optimization that requires minutes to hours of processing, making them unsuitable for real-time applications. To this end, we present TemPoFast3D, a novel method that leverages temporal coherency of human appearance to reduce redundant computation while maintaining reconstruction quality. Our approach is a “plug-and play” solution that uniquely transforms pixel-aligned reconstruction networks to handle continuous video streams by maintaining and refining a canonical appearance representation through efficient coordinate mapping. Extensive experiments demonstrate that TemPoFast3D matches or exceeds state-of-the-art methods across standard metrics while providing high-quality textured reconstruction across diverse pose and appearance, with a maximum speed of 12 FPS. Figure 1 : We propose TemPoFast3D , a novel pipeline to leverage the temporal coherency of human appearance for efficient and accurate 3D human reconstruction from monocular videos. We temporally propagate information from the past frames result by blending the pixel-aligned implicit function and avatar reconstruction method. 1 Introduction Real-time 3D human reconstruction from monocular video streams is a fundamental challenge that could revolutionize virtual reality, telepresence, and human-computer interaction. These applications demand methods that can accurately reconstruct 3D clothed humans from single images or video streams, capturing both detailed geometry and realistic appearance. However, existing approaches often struggle to balance the computational efficiency required for video with the high fidelity demanded by real-world applications. Recent approaches to 3D clothed human reconstruction primarily follow two distinct paradigms. (i) Single-image reconstruction methods rely on pixel-aligned features Saito et al. 2019 ; Saito et al. 2020 ; Zheng et al. 2022 ; Alldieck et al. 2022 ; He et al. 2020 ; Xiu et al. 2022 ; Xiu et al. 2023 ; Zhang et al. 2023 to capture detailed geometry and textures through implicit functions. (ii) Video-based reconstruction methods leverage pose deformation with either implicit neural fields Weng et al. 2022 ; Guo et al. 2023 ; Jiang et al. 2023 ; Jiang et al. 2022b ; Jiang et al. 2022a ; Peng et al. 2021 ; Su et al. 2021 ; Su et al. 2022 or Gaussian splatting Hu et al. 2024 ; Qian et al. 2024 ; Shen et al. 2023 ; Moon et al. 2024 for articulated 3D modeling. While producing high-quality results, these methods require extensive optimization ( i.e. minutes to hours) and multiple passes over the video, making them unsuitable for real-time applications. Recent attempts at real-time reconstruction Feng et al. 2022 either sacrifice quality, require additional inputs (templates Habermann et al. 2019 ; Xu et al. 2018 , multi-view Collet et al. 2015 ; Zhang et al. 2021b , depth Newcombe et al. 2011 ; Newcombe et al. 2015 ; Dong et al. 2022 ), or lack true 3D capability Li et al. 2020 ; Feng et al. 2022 ; Rocco et al. 2023 , leaving fast, high-quality reconstruction from video an unsolved challenge. Our key insight is that while human poses change rapidly across video streams, the underlying body shape and clothing geometry remain largely consistent over short time periods. Video-based methods exploit this temporal coherence, but require multiple passes over the entire sequence for global optimization. We observe that this temporal consistency can be leveraged even further for faster reconstruction through progressive canonical shape learning for sequential frame-by-frame processing. However, the challenge of learning a canonical shape through sequential frame-by-frame processing remains largely unexplored in existing literature. We address this limitation by combining the reconstruction accuracy of pixel-aligned methods Saito et al. 2019 ; Saito et al. 2020 ; Zheng et al. 2022 ; Alldieck et al. 2022 ; He et al. 2020 ; Xiu et al. 2022 ; Xiu et al. 2023 ; Zhang et al. 2023 with bidirectional canonical-posed space mapping Weng et al. 2022 ; Guo et al. 2023 ; Jiang et al. 2023 ; Jiang et al. 2022b ; Jiang et al. 2022a ; Peng et al. 2021 ; Su et al. 2021 ; Su et al. 2022 , enabling efficient shape learning while maintaining high reconstruction quality from video streams. To that end, we propose TemPoFast3D , a novel fast frame-by-frame 3D reconstruction approach that temporally propagates canonical shape information across video frames. Our method combines pixel-aligned reconstruction with SMPL-based coordinate mapping to maintain a consistent canonical representation while accurately capturing pose variations. This combination naturally extends to multi-view settings when additional views are available. Furthermore, we develop optimization strategies including adaptive coordinate sampling and visibility-guided filtering that significantly reduce per-frame computation. Our framework is designed as a “plug-and-play” solution that accelerates existing SMPL-guided pixel-aligned reconstruction methods, reaching maximum speed of 12 frames per second while maintaining reconstruction quality. In summary, our contributions are: • We introduce TemPoFast3D, a novel framework combining canonical space inference with efficient pose deformation, enabling faster 3D clothed human reconstruction from monocular video without additional inputs or templates • A suite of optimization strategies including adaptive coordinate sampling and visibility-guided filtering mechanisms that significantly reduce per-frame computation while preserving reconstruction quality • A plug-and-play design that accelerates existing SMPL-aligned reconstruction methods while enabling additional capabilities such as multi-view reconstruction for enhanced quality Figure 2 : Overview of our TemPoFast3D pipeline. Given an input RGB frame I_{t} , our method combines efficient canonical space processing with coordinate mapping for fast 3D human reconstruction. The pipeline consists of: (Section 3.1 ) Feature extraction and SMPL params regression, (Section 3.2 ) Mapping canonical coordinates to posed space, (Section 3.2.2 ) Shape and color query, and (Section 3.3 ) Canonical space processing. The canonical space representation X_{c} is continuously updated across frames. 2 Related Works Pixel-Aligned Features for Monocular Human Reconstruction. Pixel-aligned reconstruction methods have revolutionized 3D clothed human reconstruction, with PIFu Saito et al. 2019 introducing implicit functions that map 2D pixel features to 3D space for enhanced detail representation. Subsequent approaches like PIFuHD Saito et al. 2020 and Geo-PIFu He et al. 2020 further improved reconstruction quality through multi-resolution designs and geometric priors. While effective for visible details, these methods Saito et al. 2019 ; Saito et al. 2020 ; He et al. 2020 ; Li et al. 2020 struggle with complex clothing and poses due to their reliance on visible data only. Recent advances integrate parametric models through SMPL Loper et al. 2015 framework Zheng et al. 2022 ; Huang et al. 2020 ; He et al. 2021 ; Cao et al. 2023 ; Liao et al. 2023 and improve fidelity using normal maps and SDF prediction Xiu et al. 2022 ; Xiu et al. 2023 ; Alldieck et al. 2022 ; Corona et al. 2023 ; Zhang et al. 2023 ; Zhang et al. 2024 ; Yang et al. 2023 . However, these methods prioritize quality over speed, resulting in high computational costs that limit their applicability for video streams where reconstruction speed is crucial. Pose Deformation for Monocular Video Reconstruction. Leveraging the temporal coherence of human shape