MaintaAvatar: A Maintainable Avatar Based on Neural Radiance Fields by Continual Learning

Shengbo Gu, Yu-Kun Qiu, Yu-Ming Tang, Ancong Wu, Wei-Shi Zheng · 2025

Read MaintaAvatar: A Maintainable Avatar Based on Neural Radiance Fields by Continual Learning by Shengbo Gu, Yu-Kun Qiu, Yu-Ming Tang, Ancong Wu, Wei-Shi Zheng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-02372.txt.

Introduction

Five researchers present a digital avatar that can change with its person in a paper from 2025. Avatars built from neural radiance fields render a person in three dimensions from photos. Real people, though, change clothes, hair, and poses over time. Older avatar methods assume a fixed set of photos and cannot update. Their system, MaintaAvatar, learns new looks while keeping old ones intact.

Two added parts do the protective work. One stores global and local traits apart, and one keeps body poses true when new data arrives. Only a small set of new photos is needed for an update. Tests showed the avatar kept past looks without color bleed or pose errors. The authors target long-lived virtual doubles.

Catalog record

CREATORShengbo Gu, Yu-Kun Qiu, Yu-Ming Tang, Ancong Wu, Wei-Shi Zheng
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-02372

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MaintaAvatar: A Maintainable Avatar Based on Neural Radiance Fields by Continual Learning Shengbo Gu, Yu-Kun Qiu, Yu-Ming Tang, Ancong Wu, Wei-Shi Zheng Abstract The generation of a virtual digital avatar is a crucial research topic in the field of computer vision. Many existing works utilize Neural Radiance Fields (NeRF) to address this issue and have achieved impressive results. However, previous works assume the images of the training person are available and fixed while the appearances and poses of a subject could constantly change and increase in real-world scenarios. How to update the human avatar but also maintain the ability to render the old appearance of the person is a practical challenge. One trivial solution is to combine the existing virtual avatar models based on NeRF with continual learning methods. However, there are some critical issues in this approach: learning new appearances and poses can cause the model to forget past information, which in turn leads to a degradation in the rendering quality of past appearances, especially color bleeding issues, and incorrect human body poses. In this work, we propose a maintainable avatar (MaintaAvatar) based on neural radiance fields by continual learning, which resolves the issues by utilizing a Global-Local Joint Storage Module and a Pose Distillation Module. Overall, our model requires only limited data collection to quickly fine-tune the model while avoiding catastrophic forgetting, thus achieving a maintainable virtual avatar. The experimental results validate the effectiveness of our MaintaAvatar model. 1 School of Computer Science and Engineering, Sun Yat-sen University, China 2 Peng Cheng Laboratory, Shenzhen, China 3 Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education, China {gushb3,qiuyk,tangym9}@mail2.sysu.edu.cn, [email protected], [email protected] 1 Introduction Free-viewpoint rendering of 3D scenes has attracted significant academic and industrial attention, with effective solutions such as neural radiance fields (NeRF) and 3D Gaussian Splatting (3DGS) demonstrating impressive performance in this field. ( Mildenhall et al. 2021 ; Barron et al. 2021 ; Martin-Brualla et al. 2018 ; Martin-Brualla et al. 2021 ; Wang et al. 2021 ; Zhang et al. 2020 ; Kerbl et al. 2023 ; Qian et al. 2024 ; Li et al. 2024 ; Zhou et al. 2024 ; Sun, Wu, and Gao 2024 ) . Among them, human-specific methods ( Peng et al. 2021 ; Liu et al. 2020 ; Weng et al. 2022 ; Sun et al. 2022 ; Hu, Hu, and Liu 2024 ; Pan et al. 2023 ; Zhao et al. 2022 ; Li et al. 2024 ) model dynamic human bodies by establishing a deformable radiance field, achieving the driving and free-viewpoint rendering of dynamic human bodies. In this paper, we focus on creating a maintainable avatar that models a certain character with ever-changing poses and appearances (see Figure 1 ). In the real world, the poses and appearances of a person could be updated frequently, and consequently, the corresponding human avatar should also be updatable. Employing conventional static human-nerf methods to track real-world trends will lead to a high demand for computation, training time, and extra storage room, let alone the need for abundant training samples from new styles and poses. We raise such a question: Is it possible to maintain a dynamic human avatar using only a few shots from the updated character, with the ability to trace back to any point of time? Figure 1: In reality, a person’s pose and appearance constantly update. Our maintainable avatar is designed to continuously learn from sequential data, enabling it to render any previously encountered viewpoint, pose, or appearance. The main challenge of such a task lies in the old avatar re-rendering with old poses and appearances, which is related to the well-known catastrophic forgetting ( French 1999 ; Yang et al. 2023 ) of deep neural networks. Specifically, if the model directly learns from new data, it tends to forget the information about past poses and appearances, resulting in poor rendering outcomes. Further, we identify two major issues that employ the popular ‘replay’ strategy that stores or generates (fake) old inputs for future task training ( Chung et al. 2022 ; Wu et al. 2024 ; Cai and Müller 2023 ; Shin et al. 2017 ) into a human avatar: (1) The appearances between different tasks can interfere with each other, resulting in color bleeding and affecting the geometric rendering quality of each task. (2) During the process of continual learning of novel appearances and poses, the model may render incorrect poses in past tasks, consequently affecting the driving of the virtual avatar. To address these problems, we propose two key solutions. First, the Global-Local Joint Storage Module stores the distinctions between global and local information of different appearances separately in global embeddings and Tri-planes. This allows the model to better differentiate the variations among different appearances. Second, the Pose Distillation Module extracts the pose information from past tasks as supervision signals for novel tasks, enabling the model to better retain the pose information of past tasks. Additionally, we create a pretrained model trained on multiple human bodies as the initialization for training, which can adjust the surface of the avatar model with only a few images, and quickly adapt to new human bodies. We summarize our main contributions as follows: • We study the problem of modeling a maintainable virtual avatar, which only needs a few shots to quickly update the character’s appearance and also maintain the ability to render the old appearance. This maintainable virtual avatar can adapt in real-time to updates in the appearance of real-world individuals. • We analyze the two critical issues in the continual learning of maintainable virtual avatars. First, in order to prevent color bleeding between different appearances, we introduce the Global-Local Joint Storage Module that can precisely model the differences between various appearances. • Besides, to alleviate the information loss in the human pose, we introduce the Pose Distillation Module, which is capable of preserving the correct pose information of past appearances. Experimental results in two datasets demonstrate the effectiveness of our model, achieving state-of-the-art performance. 2 Related Work Neural Radiance Fields. NeRF ( Mildenhall et al. 2021 ) is a widely acclaimed 3D technology that primarily focuses on synthesizing novel views. It represents a 3D environment as an implicit neural radiance field. Given a camera pose, it can generate images from any viewpoint using ray tracing. Specifically, NeRF ( Mildenhall et al. 2021 ) utilizes a Multilayer Perceptron (MLP) ( Taud and Mas 2018 ) to map spatial coordinates and view directions to their corresponding colors and opacity. NeRF has sparked significant interest and extensive research across a range of domains, including applications in autonomous driving ( Hu et al. 2024b ; Chen et al. 2024 ; Cheng et al. 2023 ; Feldmann et al. 2024 ) , controllable human avatars ( Weng et al. 2022 ; Weng et al. 2023 ; Peng et al. 2021 ; Sun et al. 2022 ; Jiang et al. 2022 ) , large urban scenes ( Wang et al. 2023 ; Xu et al. 2023 ; Turki et al. 2023 ) , and text-driven 3D scene editing ( Bao et al. 2023 ) and so on. Human-specific Neural Representation. Previous work ( Weng et al. 2022 ; Weng et al.

Read the complete text