Occlusion-Aware Temporally Consistent Amodal Completion for 3D Human-Object Interaction Reconstruction Conference: Proceedings of the 33rd ACM International Conference on Multimedia; October 27–31, 2025; Dublin, Ireland Proceedings of the 33rd ACM International Conference on Multimedia (MM ’25), Oct
Hyungjun Doh, Dong In Lee, Seunggeun Chi, Pin-Hao Huang, Kwonjoon Lee, Sangpil Kim, Karthik Ramani · 2025
Read Occlusion-Aware Temporally Consistent Amodal Completion for 3D Human-Object Interaction Reconstruction Conference: Proceedings of the 33rd ACM International Conference on Multimedia; October 27–31, 2025; Dublin, Ireland Proceedings of the 33rd ACM International Conference on Multimedia (MM ’25), Oct by Hyungjun Doh, Dong In Lee, Seunggeun Chi, Pin-Hao Huang, Kwonjoon Lee, Sangpil Kim, Karthik Ramani (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-08137.txt.
Introduction
Hyungjun Doh and co-authors rebuild hidden body parts in video in a 2025 paper. When people handle objects on camera, hands and objects constantly hide each other, and frame-by-frame reconstruction methods flicker or lose the covered parts. Their framework completes the hidden regions, a task called amodal completion, while enforcing consistency across time.
Rather than fitting a fixed body template, the method lets each new frame refine the reconstruction, so detail builds as the video runs. Tests with a modern splatting renderer on hard single-camera videos beat existing techniques on occlusion handling and stability. The approach targets films, games, and robotics, anywhere a single video must become a dependable three-dimensional record of an interaction.
Catalog record
| CREATOR | Hyungjun Doh, Dong In Lee, Seunggeun Chi, Pin-Hao Huang, Kwonjoon Lee, Sangpil Kim, Karthik Ramani |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-08137
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Occlusion-Aware Temporally Consistent Amodal Completion for 3D Human-Object Interaction Reconstruction Conference: Proceedings of the 33rd ACM International Conference on Multimedia; October 27–31, 2025; Dublin, Ireland Proceedings of the 33rd ACM International Conference on Multimedia (MM ’25), October 27–31, 2025, Dublin, Ireland DOI: 10.1145/3746027.3754769 ISBN: 979-8-4007-2035-2/2025/10 532 CCS: Computing methodologies Shape inference CCS: Computing methodologies Reconstruction Hyungjun Doh, Dong In Lee, Seunggeun Chi, Pin-Hao Huang, Kwonjoon Lee, Sangpil Kim, Karthik Ramani Abstract . We introduce a novel framework for reconstructing dynamic human–object interactions from monocular video that overcomes challenges associated with occlusions and temporal inconsistencies. Traditional 3D reconstruction methods typically assume static objects or full visibility of dynamic subjects, leading to degraded performance when these assumptions are violated—particularly in scenarios where mutual occlusions occur. To address this, our framework leverages amodal completion to infer the complete structure of partially obscured regions. Unlike conventional approaches that operate on individual frames, our method integrates temporal context, enforcing coherence across video sequences to incrementally refine and stabilize reconstructions. This template-free strategy adapts to varying conditions without relying on predefined models, significantly enhancing the recovery of intricate details in dynamic scenes. We validate our approach using 3D Gaussian Splatting on challenging monocular videos, demonstrating superior precision in handling occlusions and maintaining temporal stability compared to existing techniques. Figure 1. ( left ) In human–object interaction (HOI) scenarios, occlusions frequently affect both the human and the object. ( middle ) We inpaint the occluded regions while preserving temporal consistency for both entities across frames. ( right ) Leveraging the temporally consistent image sequences, we reconstruct the human and object using a 3D Gaussian splatting representation, enabling animatable 3D HOI applications. † † cc-license: by 1. Introduction Understanding the complex interplay between humans and objects is a fundamental challenge in computer vision and robotics, with broad implications for both theory and real-world applications. This understanding is critical for enabling technologies such as autonomous robots that navigate complex human environments and augmented or virtual reality systems that rely on precise 3D reconstructions. Despite significant progress, reliably interpreting human-object interactions (HOI) in dynamic, real-world settings remains an open problem. Visual complexities—especially mutual occlusions—often obscure critical details, underscoring the need for enhanced 3D reconstruction techniques that capture and animate these interactions effectively. Multi-view 3D reconstruction ( Kerbl et al. 2023 ; Mildenhall et al. 2021 ; Schonberger and Frahm 2016 ) methods have proven effective in generating detailed models from multiple images. Object reconstruction techniques ( Yang et al. 2024 ; Jain et al. 2021 ; Sun et al. 2022 ; Zhu et al. 2024 ) are generally designed for static scenes, where the object remains stationary. In contrast, human reconstruction methods ( Jiang et al. 2022 ; Kocabas et al. 2024 ; Hu et al. 2024 ) typically handle dynamic subjects assuming full visibility. However, in HOI scenarios, these assumptions break down. Both humans and objects are in motion, leading to frequent and mutual occlusions as illustrated in Figure 1 , resulting in ambiguous visual cues that complicate the reconstruction process. The presence of dynamic and mutual occlusions introduces complexities beyond the scope of static object or dynamic full-visibility human reconstruction methods. Amodal completion ( Zhou et al. 2023b ; Chen et al. 2023 ; Ozguroglu et al. 2024 ; Li et al. 2022 ; Nguyen et al. 2022 ; Wu et al. 2022 ; Xu et al. 2024 ; Chi et al. 2025 ) has emerged as a promising strategy to mitigate these occlusion challenges by inferring the complete structure of partially hidden regions. However, most existing approaches are designed for static scenes. Although they perform well in controlled environments, they often falter in dynamic, real-world settings. A key limitation is their reliance on image-level completion, which overlooks the rich temporal context provided by adjacent sequences in HOI cases. Without leveraging temporal information, reconstructions derived from amodal-completed images can suffer from inconsistencies, ultimately compromising the overall quality and stability of the models. Recent research ( Teed and Deng 2020 ; Wang et al. 2024 ; Zhou et al. 2024 ; Zhang et al. 2024 ; Yang et al. 2023 ; Cong et al. 2023 ; Jeong et al. 2022 ) suggests that incorporating temporal consistency can effectively address these challenges. By enforcing coherence across consecutive video frames, models can integrate information from multiple viewpoints more effectively, leading to more accurate recovery of occluded regions. This strategy also enables the incremental refinement of hidden details over time, yielding reconstructions that more faithfully capture the dynamic nature of human–object interactions. Building on these insights, we propose a novel, temporally consistent amodal completion framework tailored for monocular video input. By integrating temporal context into the amodal completion process, our approach overcomes the limitations of traditional frame-by-frame methods, significantly enhancing both stability and realism. Specifically, our framework consists of three key components. First, we introduce Bidirectional Temporal Feature (BTF) Warping , which leverages optical flow to warp latent features from both past and future frames into the current frame’s latent space, enabling effective propagation of temporal cues and aligns spatially meaningful information across time. Second, we present a Temporal Fusion Attention mechanism that adaptively aggregates these temporally aligned features, producing a coherent and semantically enriched latent representation. Third, we propose a Template-free Occlusion Identification strategy that combines 2D segmentation and 3D projections to localize occlusions without relying on predefined templates. Finally, we apply amodal completion, guided by the temporal-aware latent features and precise occlusion masks, to complete missing regions with high fidelity. This unified pipeline enables robust recovery of occluded structures across time, ensuring temporal consistency and accurate amodal completion. Unlike prior approaches, our method generalizes effectively to dynamic, real-world human–object interaction scenarios. We validate the effectiveness of our method through extensive experiments on two public datasets, BEHAVE ( Bhatnagar et al. 2022 ) and InterCap ( Huang et al. 2022 ; Huang et al. 2024 ) , which present severe occlusions and diverse human–object interactions. Our framework demonstrates consistently superior performance in both quantitative metrics and qualitative comparisons. Moreover, we extend our evaluation to reconstructing 3D human–object interactions from monocular video sequences using 3D Gaussian Splatting ( Kerbl et al. 2023 ) (3DGS), enabling various 3D application of animatable HOI including novel-view synthesis. In summary, our contributions include: • Temporally Consistent Completion Method: We carefully design a Bidirectional Temporal Feature (BTF) Warping and a Temporal Fusion Attention mechanism that incorporate temporal context to amodal completion by adaptive fusion of aligned features across past and future frames. • Template-free Occlusion Identification Strategy: We propose a hybrid approach that integrates 2D segmentation with 3D point cloud projection to accurately localize occluded regions. This method introduces a novel masking technique that identifies occlusions efficiently, without the need for predefined templates. • Application to 3D Reconstruction: To the best of our knowledge, our work is the first approach to reconstruct photo-realistic and animatable 3D human-object interactions from monocular videos. By leveraging 3D Gaussian Splatting ( Kerbl et al. 2023 ) , we demonstrate that our pipeline is applicable to various subtasks, including novel-view and novel-pose synthesis for HOI. 2. Related Work 2.1. 3D Reconstruction 3D reconstruction ( Kerbl et al. 2023 ; Mildenhall et al. 2021 ; Schonberger and Frahm 2016 ) from multi-view images