Gen-3Diffusion: Realistic Image-to-3D Generation via 2D & 3D Diffusion Synergy

Yuxuan Xue Xianghui Xie Riccardo Marin Gerard Pons-Moll University of Tübingen Tübingen AI Center Max Planck Institute for Informatics, Saarland Informatics Campus · 2024

Read Gen-3Diffusion: Realistic Image-to-3D Generation via 2D & 3D Diffusion Synergy by Yuxuan Xue Xianghui Xie Riccardo Marin Gerard Pons-Moll University of Tübingen Tübingen AI Center Max Planck Institute for Informatics, Saarland Informatics Campus (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-06698.txt.

Introduction

Yuxuan Xue and three co-authors generate 3D objects from one photo in a paper from twenty twenty-four. Making realistic 3D objects and clothed avatars from a single image is ill-posed. Recent work borrows priors from 2D diffusion models trained on huge data. Those models generalize well, but their many views of an object may not agree in 3D.

Their system, Gen-3Diffusion, pairs a 2D and a 3D diffusion model. The two are synchronized during both training and sampling. Each covers the other's weakness. The 2D model supplies strong shape priors for unseen images, aiding generalization. The 3D model enforces consistency the 2D side cannot guarantee. Readers get image-to-3D generation that aims for realism and coherence at once.

Catalog record

CREATORYuxuan Xue Xianghui Xie Riccardo Marin Gerard Pons-Moll University of Tübingen Tübingen AI Center Max Planck Institute for Informatics, Saarland Informatics Campus
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-06698

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Gen-3Diffusion: Realistic Image-to-3D Generation via 2D & 3D Diffusion Synergy Yuxuan Xue Xianghui Xie Riccardo Marin Gerard Pons-Moll University of Tübingen Tübingen AI Center Max Planck Institute for Informatics, Saarland Informatics Campus Abstract Creating realistic 3D objects and clothed avatars from a single RGB image is an attractive yet challenging problem. Due to its ill-posed nature, recent works leverage powerful prior from 2D diffusion models pretrained on large datasets. Although 2D diffusion models demonstrate strong generalization capability, they cannot guarantee the generated multi-view images are 3D consistent. In this paper, we propose Gen-3Diffusion : Realistic Image-to-3D Generation via 2D & 3D Diffusion Synergy. We leverage a pre-trained 2D diffusion model and a 3D diffusion model via our elegantly designed process that synchronizes two diffusion models at both training and sampling time. The synergy between the 2D and 3D diffusion models brings two major advantages: 1) 2D helps 3D in generalization : the pretrained 2D model has strong generalization ability to unseen images, providing strong shape priors for the 3D diffusion model; 2) 3D helps 2D in multi-view consistency : the 3D diffusion model enhances the 3D consistency of 2D multi-view sampling process, resulting in more accurate multi-view generation. We validate our idea through extensive experiments in image-based objects and clothed avatar generation tasks. Results show that our method generates realistic 3D avatars and objects with high-fidelity geometry and texture. Extensive ablations also validate our design choices and demonstrate the strong generalization ability to diverse clothing and compositional shapes. Our code and pretrained models will be publicly released on our project page . I Introduction Fig. 1: Given a single image of a person or an object, our method Gen-3Diffusion creates realistic 3D objects or clothed avatars with high-fidelity geometry and texture. We use Gaussian Splatting to flexibly represent various shapes which can be extracted to high-quality textured meshes. Creating realistic 3D content is crucial for numerous applications, including AR/VR, as well as in the movie and gaming industries. Methods for creating a 3D model from a single RGB image are especially important to scale up 3D modelling and make it more consumer-friendly compared to traditional studio-based capture methods. However, this task presents substantial challenges due to the extensive variability in object shapes and appearances. These challenges are further intensified by the inherent ambiguities associated with monocular 2D views. Moreover, beyond general object modeling, the generation of realistic clothed avatars presents a particularly demanding set of challenges. This complexity arises from the diversity of human body shapes and poses, which is compounded by a wide array of clothing, accessories, and occlusion by interacting objects. Such challenges are accentuated by the relative scarcity of large-scale 3D human datasets as compared to those available for objects, highlighting the critical need for advanced modeling techniques that can effectively navigate these complexities. Recent image-to-3D approaches can be categorized into Direct-Reconstruction-based and Multi-View Diffusion-based methods. Direct-Reconstruction-based approaches directly predict a 3D representation that can be rendered from any viewpoint. Due to the explicit 3D representation, these methods produce an arbitrary number of consistent viewpoint renderings. For objects reconstruction, recent approaches such as LRM [ 1 ] and TriplaneGaussian [ 2 ] directly predict the NeRF or 3D Gaussian Splats from the input context view. However, these non-generative models directly regress the 3D representation in a deterministic manner, which easily leads to blurry unseen regions in Fig. 3 . For clothed avatar reconstruction, recent popular approaches obtain the 3D model based on common template [ 3 , 4 , 5 , 6 ] which utilize the SMPL [ 7 ] body model as the shape prior and perform the clothed avatar reconstruction. However, underlying SMPL body template highly limits the 3D representation of challenging human appearance, such as large dress, occlusion by interacting objects, etc. Examples can be found in Fig. 3 . Furthermore, human reconstructors are trained on relative small-scale datasets due to the limited amount of high-quality 3D human data, which further restricts their ability to generalize to diverse shapes and textures. Last but not least, all above mentioned image-to-3D reconstruction works, regardless object-oriented or human-oriented, are typically deterministic which produce blurry textures and geometry in the occluded regions. Multi-view diffusion-based methods [ 8 , 9 , 10 ] are proposed to synthesize desired novel views from single RGB image. These methods distill the inherent 3D structure presented in pretrained 2D diffusion models [ 11 ] . Typically, they fine-tune a large-scale 2D foundation model [ 11 ] on a large 3D dataset of objects [ 12 , 13 , 14 ] , to generate novel views at given camera poses. Thanks to the pertaining on large-scale image datasets, Multi-view diffusion methods show strong generalization capability to unseen objects. However, since these models diffuse images purely in 2D without explicit 3D constraints or representation, the resulting multi-views often lack 3D consistency [ 15 , 16 ] . The 3D inconsistent multi-view images further restrict downstream applications such as sparse-view 3D reconstruction [ 17 ] . To address these challenges, we propose Gen-3Diffusion : Realistic Image-to-3D Generation via 2D & 3D Diffusion Synergy. We design our method based on two key insights: 1). 2D multi-view diffusion models (MVDs) provide strong shape priors that help 3D reconstruction; 2). Explicit 3D representation produces guaranteed 3D consistent multi-views that improve the accuracy of sampled 2D multi-view images. To leverage the benefits of both 2D MVD and explicit 3D representation, we propose a novel framework that synchronizes a 3D diffusion model with a 2D MVD model at each diffusion sampling step. Specifically, we first introduce a novel 3D diffusion model that directly regresses 3D Gaussian Splatting (3D-GS [ 18 ] ) from intermediately denoised multi-views images of 2D MVD. The predicted 3D-GS can be rendered into multi-view images with guaranteed 3D consistency. At every iteration, 2D MVD denoises multi-view images conditioned on input view, which are then reconstructed to 3D-GS by our 3D diffusion model. The 3D-GS are then re-rendered to multi-views to continue the diffusion sampling process. This 3D lifting during iterative sampling improves the 3D consistency of the generated 2D multi-view images while leveraging a large-scale foundation model trained on billions of images. In summary, our contributions are: • We propose a novel 3D-GS diffusion model for 3D reconstruction, which bridges large-scale priors from 2D multi-view diffusion models and the efficient explicit 3D-GS representation. • A sophisticated joint diffusion process that incorporates reconstructed 3D-GS to improve the 3D consistency of 2D diffusion models by refining the reverse sampling trajectory. • Our proposed formulation enables us to achieves superior performance and generalization capability than prior works, both in fields of objects reconstruction ( \textit{Gen3D}_{\text{object}} ) and clothed human reconstruction ( \textit{Gen3D}_{\text{avatar}} ). Our code and pretrained models will be publicly released on our project page . II Related Works Fig. 2: Motivation for generative 3D reconstruction design . Unlike methods [ 1 , 19 ] that deterministically regress 3D from single images, our Gen-3Diffusion learns conditional distribution and samples a plausible 3D-GS, resulting in high-fidelity and realistic unseen regions. Fig. 3: Motivation for template-free avatar reconstruction design . Methods [ 5 , 3 ] relying on SMPL [ 7 ] template suffer from inaccurate SMPL estimation and cannot represent

Read the complete text