Sharp-It: A Multi-view to Multi-view Diffusion Model for 3D Synthesis and Manipulation

Yiftach Edelstein, Or Patashnik, Dana Cohen-Bar, Lihi Zelnik-Manor · 2024

Read Sharp-It: A Multi-view to Multi-view Diffusion Model for 3D Synthesis and Manipulation by Yiftach Edelstein, Or Patashnik, Dana Cohen-Bar, Lihi Zelnik-Manor (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-02631.txt.

Introduction

Yiftach Edelstein, Or Patashnik, Dana Cohen-Bar, and Lihi Zelnik-Manor present Sharp-It in a 2024 paper, a multi-view to multi-view diffusion model for 3D work. Text-to-image tools sped up 3D content a lot. A common route makes a set of views of an object, then rebuilds it in 3D from those views.

That route skips a true 3D form, so it can warp shapes and gives weak control. Pure 3D models keep control but lack detail. Sharp-It aims to bridge the two: it takes views in and gives views out, yet holds a native 3D core for clean edits.

Catalog record

CREATORYiftach Edelstein, Or Patashnik, Dana Cohen-Bar, Lihi Zelnik-Manor
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-02631

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Sharp-It: A Multi-view to Multi-view Diffusion Model for 3D Synthesis and Manipulation Yiftach Edelstein, Or Patashnik, Dana Cohen-Bar, Lihi Zelnik-Manor Abstract Advancements in text-to-image diffusion models have led to significant progress in fast 3D content creation. One common approach is to generate a set of multi-view images of an object, and then reconstruct it into a 3D model. However, this approach bypasses the use of a native 3D representation of the object and is hence prone to geometric artifacts and limited in controllability and manipulation capabilities. An alternative approach involves native 3D generative models that directly produce 3D representations. These models, however, are typically limited in their resolution, resulting in lower quality 3D objects. In this work, we bridge the quality gap between methods that directly generate 3D representations and ones that reconstruct 3D objects from multi-view images. We introduce a multi-view to multi-view diffusion model called Sharp-It, which takes a 3D consistent set of multi-view images rendered from a low-quality object and enriches its geometric details and texture. The diffusion model operates on the multi-view set in parallel, in the sense that it shares features across the generated views. A high-quality 3D model can then be reconstructed from the enriched multi-view set. By leveraging the advantages of both 2D and 3D approaches, our method offers an efficient and controllable method for high-quality 3D content creation. We demonstrate that Sharp-It enables various 3D applications, such as fast synthesis, editing, and controlled generation, while attaining high-quality assets. Figure 1 : Sharp-It is a multi-view to multi-view model that enhances low-quality 3D shapes. It corrects fine-grained geometry details and adds appearance features. The top row displays three degraded shapes and their enhancements by Sharp-It. The bottom row demonstrates Sharp-It’s capability to edit the appearance of 3D shapes. 1 Introduction Creating 3D content plays an important role across industries such as gaming, augmented and virtual reality, and animation. These applications demand efficient, controllable processes for generating high-quality, editable 3D assets. In recent years, text-to-image diffusion models have significantly advanced 3D content creation, introducing new approaches for producing complex, detailed visual assets. Early approaches to 3D content generation optimized 3D representations by leveraging 2D diffusion models [ 53 , 67 ] , producing high-quality assets but requiring time-consuming per-asset optimization. Recent techniques have adopted a two-stage process: first, synthesizing consistent multi-view images using 2D diffusion models [ 37 , 69 , 62 , 60 , 36 , 35 ] , and second, performing 3D reconstruction from these multi-view images using feed-forward methods [ 31 , 72 ] . This approach accelerates 3D synthesis while maintaining high-quality output. However, by bypassing native 3D representations, it limits controllability and editability of the resulting assets and tends to produce visual artifacts such as the Janus problem and flat objects. Direct 3D generative models [ 27 , 47 ] offer greater control over creation and editing but are constrained by resolution limitations, resulting in lower-quality assets. This presents a trade-off between the quality of multi-view methods and the controllability of direct 3D generative models. In this work, we address the quality gap between direct 3D generative models and those that reconstruct 3D objects from multi-view images. To bridge this gap, we introduce Sharp-It , a multi-view to multi-view diffusion model that enhances objects generated by 3D generative models, refining geometric details and adding fine-grained texture features. The high-quality multi-view set produced by Sharp-It can be reconstructed into a 3D model using existing sparse-view feed-forward reconstruction methods [ 72 , 26 , 21 , 81 ] . Sharp-It is conditioned on a text prompt and operates on the multi-view set in parallel, sharing features across the generated views [ 62 , 60 ] . We use Shap-E [ 27 ] as our backbone generative model, a text-conditioned 3D latent diffusion model that operates on the parameters of implicit functions. Shap-E has been shown to offer rich generative capabilities [ 9 , 59 ] , but produces low-quality assets. To train Sharp-It, we use pairs of multi-view sets obtained by encoding high-quality 3D objects [ 13 ] into Shap-E’s latent space and rendering them. Our approach offers two key advantages. First, since we start from a coarsely valid 3D object, we inherit its plausible geometric structure, avoiding artifacts like the Janus problem that can occur in direct multi-view synthesis methods. Second, the 3D consistency of the input views facilitates output consistency, allowing the model to focus on generating the fine details necessary for high-quality results. Our method combines the controllability of native 3D generative models with the capabilities of 2D diffusion models in generating highly detailed images. Through extensive experiments, we demonstrate that Sharp-It outperforms existing enhancement methods in both quality and efficiency. We show various applications, including fast generation of high-quality 3D objects from text prompts and geometric controls [ 59 ] . Additionally, we show that Sharp-It enhances 3D asset editing capabilities. 2 Related Work 2.1 Diffusion Models for Content Generation Diffusion models [ 20 , 55 , 56 ] have demonstrated remarkable success in modeling complex data distributions and generating high-fidelity samples. In the realm of 2D image synthesis, these models have achieved state-of-the-art results, producing photorealistic images with precise control. Motivated by this success, recent works have explored the application of diffusion models to 3D content creation. 3D Diffusion Models Some works utilize 3D datasets to train diffusion models for direct 3D asset generation, with different approaches varying in their choice of 3D representations. Some methods employ explicit representations [ 77 , 74 ] such as point clouds [ 57 , 47 , 80 , 43 , 76 ] . Others transform the input data into learned implicit representations such as triplanes [ 63 , 16 , 7 ] or neural networks [ 32 , 25 ] before learning their distributions. A notable example is Shap-E [ 27 ] , which adopts NeRF (Neural Radiance Field) and SDF (Signed Distance Field) as implicit representations and trains a diffusion model to generate the weights of these networks. While these 3D-native diffusion models enable fast generation and provide greater control over the generation process [ 59 ] , they often face challenges due to limited training data and the high computational cost of operating directly on 3D representations. 2D Diffusion Models for 3D Generation Another approach leverages the powerful prior of pretrained text-to-image diffusion models [ 55 ] for 3D generation. Some methods [ 68 , 53 , 45 , 28 , 10 , 70 , 34 , 75 , 65 ] utilize these models to guide the optimization of 3D representations, such as NeRF or Gaussian Splatting, typically through Score Distillation Sampling (SDS) [ 53 ] or its variants. To enhance and accelerate SDS-guided optimization, recent methods fine-tune 2D diffusion models on 3D datasets [ 12 ] . Zero-1-to-3 [ 37 ] fine-tunes a pre-trained diffusion model by incorporating input views and camera parameters to enable novel-view synthesis. Building upon this fine-tuning strategy, some followup works [ 62 , 38 , 64 , 61 ] enable simultaneous generation of multiple views using shared attention mechanisms. While these 3D-aware fine-tuning approaches strengthen the prior and improve both convergence speed and result quality when used in SDS-based optimization, the

Read the complete text