Hyper Diffusion Avatars: Dynamic Human Avatar Generation using Network Weight Space Diffusion
Dongliang Cao, Guoxing Sun, Marc Habermann, Florian Bernard · 2025
Read Hyper Diffusion Avatars: Dynamic Human Avatar Generation using Network Weight Space Diffusion by Dongliang Cao, Guoxing Sun, Marc Habermann, Florian Bernard (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-04145.txt.
Introduction
Dongliang Cao and three co-authors generate moving digital humans by dreaming up the weights of a neural network in a 2025 research paper. Photorealistic avatars usually need training on one person at a time, which blocks variety. Cartoon-like generators cover many faces but move stiffly and miss details like cloth wrinkles that shift with pose. The team's plan joins both strengths in two stages.
First, one small network is fitted per person, capturing how that body deforms while moving. Next, a diffusion model learns the pattern of those fitted networks' weights. At run time it produces fresh weights for a new person, ready for real-time rendering. Tests on a large multi-person video set beat earlier methods.
Catalog record
| CREATOR | Dongliang Cao, Guoxing Sun, Marc Habermann, Florian Bernard |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-04145
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Hyper Diffusion Avatars: Dynamic Human Avatar Generation using Network Weight Space Diffusion Dongliang Cao, Guoxing Sun, Marc Habermann, Florian Bernard Abstract Creating human avatars is a highly desirable yet challenging task. Recent advancements in radiance field rendering have achieved unprecedented photorealism and real-time performance for personalized dynamic human avatars. However, these approaches are typically limited to person-specific rendering models trained on multi-view video data for a single individual, limiting their ability to generalize across different identities. On the other hand, generative approaches leveraging prior knowledge from pre-trained 2D diffusion models can produce cartoonish, static human avatars, which are animated through simple skeleton-based articulation. Therefore, the avatars generated by these methods suffer from lower rendering quality compared to person-specific rendering methods and fail to capture pose-dependent deformations such as cloth wrinkles. In this paper, we propose a novel approach that unites the strengths of person-specific rendering and diffusion-based generative modeling to enable dynamic human avatar generation with both high photorealism and realistic pose-dependent deformations. Our method follows a two-stage pipeline: first, we optimize a set of person-specific UNets, with each network representing a dynamic human avatar that captures intricate pose-dependent deformations. In the second stage, we train a hyper diffusion model over the optimized network weights. During inference, our method generates network weights for real-time, controllable rendering of dynamic human avatars. Using a large-scale, cross-identity, multi-view video dataset, we demonstrate that our approach outperforms state-of-the-art human avatar generation methods. 1 Introduction Generating high-quality renderings of humans is a crucial challenge in computer vision and computer graphics, with numerous real-world applications in remote communication, movies, gaming, and immersive experiences in augmented as well as virtual reality. Traditionally, generating digital avatars from real-world data requires complicated hardware setups, manual efforts from skilled artists, and advanced physical-based rendering techniques to synthesize the final image Bickel et al. 2007 ; Garrido et al. 2013 . With the advancement of neural radiance fields and subsequent works Mildenhall et al. 2021 ; Wang et al. 2021 ; Wang et al. 2023b ; Kerbl et al. 2023 , recent methods Liu et al. 2021 ; Habermann et al. 2023 ; Kwon et al. 2023 ; Zhu et al. 2024 ; Pang et al. 2024 have focused on learning photorealistic and controllable human avatars directly from calibrated multi-view videos. Although these approaches achieve unprecedented levels of photorealism, they are still person-specific, meaning that for each individual human a dense multi-view video has to be captured, data has to be processed and annotated, and a dedicated neural model has to be trained from scratch. This process is neither scalable nor fast and resource-efficient as these steps can easily take multiple days Liu et al. 2021 ; Habermann et al. 2023 . Meanwhile, recent generative methods Rombach et al. 2022 ; Podell et al. 2023 ; Esser et al. 2024 have made significant progress in generalization quality and scalability, driven by advances in generative diffusion models Song et al. 2020 ; Ho et al. 2020 ; Lipman et al. 2022 . To this end, recent avatar generation methods Kolotouros et al. 2023 ; Liu et al. 2024 ; Cao et al. 2024 ; Huang et al. 2024 ; Liao et al. 2024 attempted to distill prior knowledge from 2D image diffusion models through score distillation sampling Poole et al. 2022 . Despite their compelling results, the rendering quality remains significantly lower than that of person-specific rendering methods. Notably, their rendered videos are unable to capture skeletal pose-dependent deformations like clothing wrinkles and appearance variations, e.g. cast shadows, due to the limitations of simple skeleton-based articulation. To address the limitations mentioned above, for the first time we aim to unify the person-specific rendering method and the diffusion-based generation model to generate photorealistic real-time renderings across different individuals, which faithfully captures pose-dependent deformations. To this end, we represent the digital human as 3D Gaussians Kerbl et al. 2023 that are parameterized in UV space Pang et al. 2024 ; Saito et al. 2024 ; Hu et al. 2024a ; Teotia et al. 2024 . In contrast to person-specific rendering methods relying on individual mesh templates Pang et al. 2024 ; Li et al. 2024 , we use a parametric human body model (i.e. SMPL-X Loper et al. 2015 ; Pavlakos et al. 2019 ) to offer a canonical template and a consistent UV space across individuals Zhang et al. 2024c . However, instead of directly optimizing the 3D Gaussian parameters defined in UV space for each individual, we optimize a UNet Ronneberger et al. 2015 that maps the human pose into the Gaussian parameters defined in UV space. To this end, our method is capable of capturing pose-dependent deformation by predicting motion-aware 3D Gaussian parameters. After optimizing the person-specific network for all individuals, we propose a hyper diffusion model, which generates network weights of the optimized UNet rather than 3D Gaussian parameters directly. The motivation for training a diffusion model in this network weight space is two fold: (1) the single network encodes comprehensive pose-dependent 3D Gaussian parameters, as opposed to a static UV Gaussian map; (2) the network weights provide a shared canonical representation across different individuals, as opposed to person-specific rendering methods. During inference, we can directly use our diffusion model to sample network weights and use the generated network to render dynamic digital avatars with the skeletal pose as the input. Fig. provides an overview of our method. We summarize our main contributions as follows: • For the first time, we unify person-specific rendering and diffusion-based generation to enable dynamic human avatar generation with pose-dependent deformations . • To this end, we encode a dynamic human avatar into a motion-aware network and learn a hyper diffusion model that generates the network weights representing a dynamic avatar. • To train our hyper diffusion model on network weights, we leverage a transformer-based diffusion model that effectively learns the complex structure of these weights. 2 Related work In this work, we focus on unconditional dynamic human avatar generation. As a result, human reconstruction methods that rely on multi-view images and the corresponding human pose as inputs at inference time Remelli et al. 2022 ; Shetty et al. 2024 ; Zheng et al. 2024 ; Sun et al. 2025 ; Zubekhin et al. 2025 ; Xiao et al. 2025 are out of the scope of our paper. 2.1 Personalized 3D human rendering Recent advancements in neural rendering, such as NeRF Mildenhall et al. 2021 and 3DGS Kerbl et al. 2023 , have made it possible to learn human avatars directly from calibrated multi-view video inputs. Starting from NeRF Mildenhall et al. 2021 , various approaches have been proposed to reconstruct the dynamic appearance of 3D humans Bergman et al. 2022 ; Feng et al. 2022 ; Hu et al. 2023 ; Li et al. 2022 ; Weng et al. 2022 ; Liu et al. 2021 . The key idea behind these methods is to introduce deformable human NeRFs that deform the posed space to a shared pose canonical space. Despite producing high-quality renderings, these methods inherit the limitations of NeRF-based approaches, resulting in significantly longer rendering times. To overcome this limitation, more recent methods Pang et al. 2024 ; Saito et al. 2024 ;