AHOY! A nimatable H umans under O cclusion from Y ouTube Videos with Gaussian Splatting and Video Diffusion Priors
Aymen Mir 1 Riza Alp Guler 2 Xiangjun Tang 3 Peter Wonka 3 Gerard Pons-Moll 1, [6pt] 1 University of Tübingen 2 Imperial College London 3 KAUST · 2026
Read AHOY! A nimatable H umans under O cclusion from Y ouTube Videos with Gaussian Splatting and Video Diffusion Priors by Aymen Mir 1 Riza Alp Guler 2 Xiangjun Tang 3 Peter Wonka 3 Gerard Pons-Moll 1, [6pt] 1 University of Tübingen 2 Imperial College London 3 KAUST (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-17975.txt.
Introduction
Mir and co-authors present AHOY, a way to rebuild full 3D human avatars from plain YouTube video. The method copes with heavy occlusion, where large body parts stay hidden. It uses tuned diffusion models to hallucinate views of unseen regions as training signal. A two-stage design then builds pose-dependent Gaussian maps. Tests show top reconstruction quality, and the avatars can be posed in new ways.
Catalog record
| CREATOR | Aymen Mir 1 Riza Alp Guler 2 Xiangjun Tang 3 Peter Wonka 3 Gerard Pons-Moll 1, [6pt] 1 University of Tübingen 2 Imperial College London 3 KAUST |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40713 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.17975
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AHOY! A nimatable H umans under O cclusion from Y ouTube Videos with Gaussian Splatting and Video Diffusion Priors Aymen Mir 1 Riza Alp Guler 2 Xiangjun Tang 3 Peter Wonka 3 Gerard Pons-Moll 1, [6pt] 1 University of Tübingen 2 Imperial College London 3 KAUST Abstract We present AHOY, a method for reconstructing complete, animatable 3D Gaussian avatars from in-the-wild monocular video despite heavy occlusion. Existing methods assume unoccluded input—a fully visible subject, often in a canonical pose—excluding the vast majority of real-world footage where people are routinely occluded by furniture, objects, or other people. Reconstructing from such footage poses fundamental challenges: large body regions may never be observed, and multi-view supervision per pose is unavailable. We address these challenges with four contributions: (i) a hallucination-as-supervision pipeline that uses identity-finetuned diffusion models to generate dense supervision for previously unobserved body regions; (ii) a two-stage canonical-to-pose-dependent architecture that bootstraps from sparse observations to full pose-dependent Gaussian maps; (iii) a map-pose/LBS-pose decoupling that absorbs multi-view inconsistencies from the generated data; (iv) a head/body split supervision strategy that preserves facial identity. We evaluate on YouTube videos and on multi-view capture data with significant occlusion and demonstrate state-of-the-art reconstruction quality. We also demonstrate that the resulting avatars are robust enough to be animated with novel poses and composited into 3DGS scenes captured using cell-phone video. Our project page is available at https://miraymen.github.io/ahoy/ . 1 Introduction Reconstructing photorealistic, animatable 3D humans from monocular input is a long-standing goal in computer vision and graphics, with applications spanning gaming, virtual reality, and film production. Recent methods based on NeRFs [ 102 , 130 , 37 ] and 3DGS [ 60 , 104 , 89 , 156 , 105 , 155 , 48 , 26 ] have made remarkable progress, producing high-fidelity avatars that can be re-posed in real time. Yet these methods assume unoccluded monocular input, or a fully visible subject, often in a canonical pose. These assumptions exclude the vast majority of real-world footage. YouTube alone hosts billions of videos of people in everyday activities—cooking, exercising, conversing—where the body is occluded by furniture, other people, or the scene itself. The viewpoints are limited, and large portions of the body may never be observed or cut off in the camera shot. If we could reconstruct complete, animatable avatars from such videos, we would unlock an essentially unlimited source of diverse human appearance—every body type, clothing style, appearance—without controlled capture, multi-view rigs, or cooperation from the subject. In this work, we present AHOY ( A nimatable H umans under O cclusion from Y ouTube Videos), a method that reconstructs complete, animatable 3D Gaussian avatars from in-the-wild monocular video despite heavy occlusion, and animates them within photorealistic 3D scenes (Fig. ). Achieving this requires overcoming several challenges that do not arise when the subject is fully visible. Bootstrapping a complete avatar from partial observations. With large body regions never observed, there is no direct supervision for a significant portion of the avatar. We address this in three stages. First, we build a coarse 3DGS avatar from sparse multi-view images generated from the partially observed subject (Sec. 3.1 ). Second, we render this coarse avatar in structured motion sequences and refine the imperfect renderings via an identity-finetuned video diffusion model and diffusion inversion producing hallucinated supervision videos that cover previously unobserved body surfaces from multiple viewpoints and poses (Sec. 3.2 ). Third, we train a full avatar with pose-dependent deformation using these hallucinated videos as supervision (Sec. 3.3 ). Learning pose-dependent appearance without multi-view capture. High-fidelity avatars require pose-dependent deformation to capture how clothing changes with body configuration. In the first stage, no multi-view supervision for pose-dependent deformation exists. We therefore learn an avatar that does not model pose-dependent deformations. We adapt the 2D Gaussian map design from Animatable Gaussians [ 71 ] to learn a canonical pose-independent Gaussian map that captures the texture and structure of the human. We also adopt the use of per-identity networks that use 2D maps to represent both input pose and output 3DGS properties. To obtain the pose-dependent multi-view supervision missing from the original video, we design the structured motion sequences of the second stage—full-body turns paired with target poses such as sitting and reaching—to provide both pose-dependent deformations and in-effect 360 ∘ multi-view coverage per pose. In the third stage, these hallucinated videos supply the dense multi-view supervision needed to upgrade from canonical to pose-dependent Gaussian maps that capture high-fidelity appearance changes across body configurations. This canonical-to-pose-dependent progression is the architectural insight that makes pose-dependent learning possible (Secs. 3.2.2 and 3.3.1 ). Absorbing multi-view inconsistencies from diffusion-generated data. Each hallucinated video is independently sampled from the diffusion model and is therefore not geometrically consistent across viewpoints and frames. Naively training on these videos produces artifacts. To address this, we decouple per-frame variations by introducing two distinct poses: a ‘map pose’ and an ‘LBS pose.’ Specifically, the map pose, which is taken as input for our per-identity networks, is shared across frames with similar poses to ensure consistent Gaussian attributes prediction for a given pose. In contrast, the LBS pose, which deforms the output Gaussians from our per-identity networks, is optimized per-frame to absorb the geometric inconsistencies inherent in the video diffusion model (Sec. 3.3.2 ). Preserving facial identity under unreliable diffusion outputs. Video diffusion models generate identity-inconsistent faces. We partition the avatar into head and body regions via SMPL-to-FLAME correspondence, mask head pixels from the diffusion-based body supervision, and instead supervise the head through a dedicated multi-view face diffusion model that preserves identity (Sec. 3.3.3 ). We evaluate AHOY on YouTube videos featuring humans under significant occlusion and on multi-view capture data with comparable occlusion, and demonstrate state-of-the-art reconstruction quality. The resulting avatars are robust enough to be animated with novel poses and composited into 3DGS scenes reconstructed from casual cell-phone video—yielding an end-to-end pipeline from commodity RGB input to photorealistic human–scene rendering. In summary, our contributions are: 1) A method for reconstructing complete, animatable 3DGS avatars from in-the-wild monocular video under heavy occlusion—unlocking YouTube-scale footage as a source of 3D human assets. 2) A hallucination-as-supervision approach using identity-finetuned video diffusion and diffusion inversion. 3) A two-stage canonical-to-pose-dependent architecture that allows for learning high-fidelity pose-dependent deformations in 3DGS avatars. 4) A map-pose/LBS-pose decoupling that enables pose-dependent Gaussian maps for high-fidelity avatar learning from multi-view-inconsistent generated data. 5) A head/body split supervision for facial identity preservation. 2 Related Work Neural Rendering. NeRF [ 82 ] and its extensions [ 90 , 9 , 10 , 119 , 134 ] enable high-quality novel-view synthesis but remain computationally expensive. 3DGS [ 58 ] addresses this with explicit Gaussian primitives [ 63 , 131 ] and has been extended to dynamic scenes [ 115 , 76 , 133 , 64 , 69 , 84 ] , SLAM [ 57 ] , mesh extraction [ 46 , 30 ] , and sparse-view reconstruction [ 81 ] . Human Reconstruction and Neural Rendering. Dense body representations such as DensePose [ 31 , 92 , 93 ] provide pixel-level surface correspondences for human understanding. Mesh-based approaches [ 99 , 74 , 13 , 56 , 7 , 6 , 85 , 36 ] and implicit functions [ 80 ,