MVD-HuGaS: Human Gaussians from a Single Image via 3D Human Multi-view Diffusion Prior

Kaiqiang Xiong, Ying Feng, Qi Zhang, Jianbo Jiao, Yang Zhao, Zhihao Liang, Huachen Gao, Ronggang Wang · 2025

Read MVD-HuGaS: Human Gaussians from a Single Image via 3D Human Multi-view Diffusion Prior by Kaiqiang Xiong, Ying Feng, Qi Zhang, Jianbo Jiao, Yang Zhao, Zhihao Liang, Huachen Gao, Ronggang Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-08218.txt.

Introduction

Rebuilding a person in 3D from one photo, viewable from any angle, is a standing challenge in computer graphics. Kaiqiang Xiong and co-authors present MVD-HuGaS in this 2025 paper, which first imagines the person from many viewpoints using a diffusion model tuned on 3D human data. Those imagined views then guide a set of 3D Gaussians that render the body.

Two fixes handle the weak points. An alignment step jointly adjusts the Gaussians and the guessed camera poses, since sparse invented views come without true angles. A depth-based step then repairs facial regions, which often warp. On two benchmark datasets the method reached top scores for single-image human rendering, the authors report.

Catalog record

CREATORKaiqiang Xiong, Ying Feng, Qi Zhang, Jianbo Jiao, Yang Zhao, Zhihao Liang, Huachen Gao, Ronggang Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-08218

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MVD-HuGaS: Human Gaussians from a Single Image via 3D Human Multi-view Diffusion Prior Kaiqiang Xiong, Ying Feng, Qi Zhang, Jianbo Jiao, Yang Zhao, Zhihao Liang, Huachen Gao, Ronggang Wang Abstract 3D human reconstruction from a single image is a challenging problem and has been exclusively studied in the literature. Recently, some methods have resorted to diffusion models for guidance, optimizing a 3D representation via Score Distillation Sampling(SDS) or generating one back-view image for facilitating reconstruction. However, these methods tend to produce unsatisfactory artifacts ( e.g. flattened human structure or over-smoothing results caused by inconsistent priors from multiple views) and struggle with real-world generalization in the wild. In this work, we present MVD-HuGaS , enabling free-view 3D human rendering from a single image via a multi-view human diffusion model. We first generate multi-view images from the single reference image with an enhanced multi-view diffusion model, which is well fine-tuned on high-quality 3D human datasets to incorporate 3D geometry priors and human structure priors. To infer accurate camera poses from the sparse generated multi-view images for reconstruction, an alignment module is introduced to facilitate joint optimization of 3D Gaussians and camera poses. Furthermore, we propose a depth-based Facial Distortion Mitigation module to refine the generated facial regions, thereby improving the overall fidelity of the reconstruction. Finally, leveraging the refined multi-view images, along with their accurate camera poses, MVD-HuGaS optimizes the 3D Gaussians of the target human for high-fidelity free-view renderings. Extensive experiments on Thuman2.0 and 2K2K datasets show that the proposed MVD-HuGaS achieves state-of-the-art performance on single-view 3D human rendering. 1 Introduction Reconstructing the 3D representation of a clothed human from only a single-view image is a fundamental challenge in 3D computer vision and computer graphics, extensively investigated due to its broad applicability in industries such as VR/AR, video gaming, and movie production. The inference of a 3D structure of the visible regions from a single image is extremely ill-posed, while the estimation of both geometry and appearance of the residual invisible regions further exacerbates this problem. To address these challenges, a surge of learning-based methods Saito et al. 2019 ; Chan et al. 2024 ; Saito et al. 2020 ; Zheng et al. 2021 ; Huang et al. 2024 ; Alldieck et al. 2019 ; Alldieck et al. 2022 ; Hu et al. 2023 ; Huang et al. 2023 ; AlBahar et al. 2023 ; Zhu et al. 2019 ; Jiang et al. 2020 ; He et al. 2020 ; Xiu et al. 2022 ; Huang et al. 2020 ; He et al. 2021 ; Liao et al. 2023 has been developed to advance the effectiveness of single-view 3D human reconstruction. Previous methods Saito et al. 2019 ; Saito et al. 2020 ; Zheng et al. 2021 train 3D generative models to predict 3D neural fields from pixel-aligned features. However, these methods often fail to generalize to various unseen scenarios due to the limited size of 3D human datasets Yu et al. 2021 ; Han et al. 2023 ; Liao et al. 2023 ; Cai et al. 2022 . Recently, motivated by the success of generic image-to-3D, several methods Huang et al. 2024 ; Ho et al. 2024 ; AlBahar et al. 2023 turn to 2D image diffusion models for guidance. Owing to pretraining on the Internet-scale image datasets, 2D diffusion models, such as Stable Diffusion Rombach et al. 2022 and Imagen Saharia et al. 2022 , have acquired a prior that encapsulates the natural distribution of images. Moreover, camera-conditioned diffusion models, including single-view Liu et al. 2023a and multi-view variants Liu et al. 2023b ; Voleti et al. 2024 , incorporate camera embedding conditions to generate images on specified viewpoints, which further promotes the lifting from 2D images to 3D. With a fine-tuned single-view diffusion model, Ho et al. 2024 generates one back-view image via an image-to-image diffusion model, followed by a 3D human reconstruction process via the Skinned Multi-Person Linear model (SMPL) Loper et al. 2023 ; Feng et al. 2021 ; Pavlakos et al. 2019 . Nevertheless, the information provided by just two images is insufficient for accurate 3D human reconstruction, resulting in a flattened appearance on the side view. Besides, Huang et al. 2024 ; Zhang et al. 2024a apply the SDS to optimize a 3D human presentation. However, given an image as a condition, single-view diffusion models cannot guarantee 3D geometric consistency among the generated images, which leads to over-smoothing textures. In essence, existing methods Zhang et al. 2024b ; Ho et al. 2024 face significant challenges in achieving high-fidelity novel view synthesis for a 3D human from a single image. To this end, we propose to leverage the multi-view diffusion prior for single-view 3D human reconstruction. Figure 1 : Camera Misalignment Issue. (a) 3D-GS rendering results from generated multi-view images with conditioned camera poses. (b) 3D-GS rendering results from multi-view images with the optimized camera poses via our Camera Alignment Module. In contrast to single-view diffusion models Liu et al. 2023a ; Liu et al. 2024b that generate multi-view images through multiple iterations, suffering from multi-view inconsistency, recent multi-view diffusion models, such as Stable Video 3D (SV3D) Voleti et al. 2024 and SyncDreamer Liu et al. 2023b , leverage cross-attention mechanisms to facilitate the multi-view information interaction during denoising and become capable of producing more geometrically consistent multi-view images through a single inference. They have also been demonstrated to achieve remarkable results in generic 3D object generation tasks. However, extending generic multi-view diffusion models Long et al. 2024 ; Voleti et al. 2024 ; Liu et al. 2024a to 3D human reconstruction presents several challenges: (1) 3D Human Body Priors Deficiency. These models are often trained on datasets that lack human data, resulting in a deficiency of essential knowledge about human anatomy ( i.e. , structure) and appearance. Roughly applying the priors from these models indeed results in unrealistic human depictions. (2) Camera Misalignment. To our knowledge, all camera-conditioned diffusion models Liu et al. 2024b ; Liu et al. 2024a ; Shi et al. 2023 , including SV3D, cannot ensure perfect alignment with specified camera parameters, leading to mismatches between the camera parameters and the generated images, thus affecting the quality of 3D human reconstruction, as depicted in Fig. 1 . (3) Facial Distortion. These models still struggle to generate accurate human faces, even when fine-tuned on 3D human datasets, as Fig. 2 shows. The dimensionality reduction in latent diffusion models may lead to significant facial detail loss, especially since faces occupy a small image region. Moreover, the human visual system is sensitive to facial details, which makes the removal of facial distortion even more necessary and challenging. Figure 2 : Facial Distortion Issue. Despite being fine-tuned on 3D human datasets, models still struggle to generate multi-view consistent and plausible human faces, even though they can learn the structural priors of the human body. To address these challenges, this paper proposes MVD-HuGaS , a pipeline that consists of the following modules: SV3D-Human: We enhance SV3D with a high-quality 3D human dataset Han et al. 2023 , instilling in the model a comprehensive understanding of human structure knowledge. This adaptation enables the generation of realistic multi-view human images. Camera Alignment Module: We use 3D Gaussians Kerbl

Read the complete text