CRAFT: Designing Creative and Functional 3D Objects

Michelle Guo* Mia Tang* Hannah Cha Ruohan Zhang C. Karen Liu Jiajun Wu Stanford University {mguo95,miatang,hcha417,zharu,karenliu,jiajunwu}@cs.stanford.edu · 2024

Read CRAFT: Designing Creative and Functional 3D Objects by Michelle Guo* Mia Tang* Hannah Cha Ruohan Zhang C. Karen Liu Jiajun Wu Stanford University {mguo95,miatang,hcha417,zharu,karenliu,jiajunwu}@cs.stanford.edu (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-03889.txt.

Introduction

Michelle Guo and co-authors present a 2024 paper on a design tool called CRAFT. Everyday objects must fit the human body and match what the user asked for. Current AI design tools struggle with both aims at once. CRAFT starts from a base mesh plus a body shape. Guidance comes as text, an image, or a sketch. A mesh deformation step then reshapes the object. The shaping balances meaning against body contact, rewarding touch and punishing overlap.

Outputs can be worn by virtual characters or fabricated for real use. No manual artist step is needed between prompt and object. Tests across object categories show the method working in both quality and quantity terms. Product designers get body-aware generation for wearables, grips, and furniture-like items. Fit is where generated objects usually fail in the hand. Building contact into the objective makes the results usable rather than merely plausible.

Catalog record

CREATORMichelle Guo* Mia Tang* Hannah Cha Ruohan Zhang C. Karen Liu Jiajun Wu Stanford University {mguo95,miatang,hcha417,zharu,karenliu,jiajunwu}@cs.stanford.edu
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE27 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-03889

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CRAFT: Designing Creative and Functional 3D Objects Michelle Guo* Mia Tang* Hannah Cha Ruohan Zhang C. Karen Liu Jiajun Wu Stanford University {mguo95,miatang,hcha417,zharu,karenliu,jiajunwu}@cs.stanford.edu Abstract For designing a wide range of everyday objects, the design process should be aware of both the human body and the underlying semantics of the design specification. However, these two objectives present significant challenges to the current AI-based designing tools. In this work, we present a method to synthesize body-aware 3D objects from a base mesh given an input body geometry and either text or image as guidance. The generated objects can be simulated on virtual characters, or fabricated for real-world use. We propose to use a mesh deformation procedure that optimizes for both semantic alignment as well as contact and penetration losses. Using our method, users can generate both virtual or real-world objects from text, image, or sketch, without the need for manual artist intervention. We present both qualitative and quantitative results on various object categories, demonstrating the effectiveness of our approach. Figure 1 : ShapeCraft generates 3D shapes given text as input. The objects are optimized to fit on various character body shapes. 1 Introduction Have you struggled to find an everyday object that will fit your body perfectly and match the exact creative concept you have in mind? Recent progress in generative AI models shows promising results in generating 3D objects, which have the potential to facilitate the design process (e.g., help designers rapidly iterate ideas) and enable better customization in industrial design [ 11 , 9 , 32 ] . For designing a wide range of everyday objects, such as glasses, hats, rings, and shoes, the designing process should be aware of both the human body and the object semantics . For these objects that are designed to be used by humans, being body-aware is essential and the design should be primarily optimized for the interaction between the object and the body that it is designed for. In addition, we want to be able to customize the design, styles, or aesthetics of these objects, i.e., we want the design to be semantically-aware in the sense that it aligns with our design specifications, which can be either text descriptions or visual examples. Therefore, we need to provide tools to address individual differences in the different object categories’ demands of body fit and the underlying semantics of the designs. Generative AI models, such as Stable Diffusion [ 42 ] , DALL-E [ 37 ] , and DreamFusion [ 38 ] , can generate semantics-aware 3D assets when given design specifications in the format of natural language, although text-to-image models would require another step that converts 2D designs into 3D objects using image-to-3D models [ 29 ] . However, these approaches typically optimize objects for semantics-related objectives, such as prompt alignment. Meanwhile, designing useful objects requires an understanding of the physical interactions between bodies and objects. It is difficult to use text or image to specify design needs for different body shapes and preferred body contacts, hence the resulted designs are not sufficiently body-aware. Additionally, the generated designs are derivatives of datasets that belong to a specific population of certain body shape and size; therefore, to generate designs for characters of any shape and size, we need to explicitly incorporate the awareness of body shapes and contacts within the generative models. On the other hand, while some previous methods [ 4 , 33 ] have been optimizing body contact or functionality for objects, their methods are usually limited to common objects. The optimization process does not consider semantics-related design specifications, such as text or image prompts. Additionally, optimizing functionality for creative objects, especially for individual human bodies and preferred contacts, is significantly more challenging and not well-addressed. Meanwhile, there are works that address both body- and semantics-aware objectives, however, they are limited to specific object categories, such as garments [ 43 , 51 ] . In this work, we propose a tool to generate customized 3D designs that are both body-aware and semantically-aware. The tools can be applied to a wide range of everyday object categories, without relying on object datasets. We build a flexible system that jointly optimizes for multiple objectives. We define semantically-aware design as the process of designing according to a text or visual concept. Personalized, body-aware design is generating a 3D shape that is well-fitted to an individual body or a specific contact map. As shown in Figure 1 , we showcase a gallery of our generated designs for various digital avatars and object categories. Our qualitative results show that ShapeCraft is effective in generating designs that are simultaneously body-aware and semantics-aware. Additionally, we show that compared to baselines, our joint optimization approach achieves the best results in terms of both objective and subjective metrics. Figure 2 : Our method generates a variety of semantics and body-aware objects from input text prompts. Figure 3 : Our method can deform the same template mesh into different text-specified geometries that are body-fitting. 2 Related Work Text or image-conditioned 3D synthesis. Recent works propose to tackle text-conditioned 3D generation either via text-to-3D [ 7 , 27 , 48 , 63 ] , or image-to-3D [ 29 , 39 , 30 ] where the input image is generated by a text-to-image model such as Stable Diffusion [ 42 ] or DALL-E [ 37 ] . Another line of work directly trains 3D diffusion models for various 3D representations, including point clouds [ 36 ] , meshes [ 13 , 31 ] , or neural fields [ 19 ] . Finally, other works achieve text or image-conditioned mesh generation by deforming a template mesh through text or image guidance [ 34 , 15 , 43 ] . Compared to text-to-image models, text-to-3D is significantly more challenging, partially due to the lack of large-scale training datasets. However, text-to-3D models can leverage pre-trained 2D models, such as CLIP, to synthesize better objects. Guided by a text prompt (embedded using CLIP), Dreamfields [ 18 ] synthesize 3D objects leveraging volume rendering. DreamFusion [ 38 ] and [ 50 ] distill 2D diffusion models as a differentiable image-based loss. Surface-based differentiable rendering can be used to pass views of explicit 3D objects to CLIP, such as Text2Mesh [ 34 ] in which they stylize the template mesh while preserving the initial content. CLIP-Mesh [ 22 ] generates new 3D objects by deforming a sphere at the vertex level, guided by the input text prompt. Magic3D [ 27 ] first optimizes a radiance field, extracts the mesh from the radiance field, and optimizes the mesh via differentiable surface rendering and score distillation. TextDeformer [ 15 ] leverages differentiable rendering and CLIP, but focuses on the problem of deforming explicit geometry rather than generating it from scratch. Body-aware 3D synthesis. The design of 3D objects for human-object interaction is an important research topic. For everyday objects, it is important to consider human bodies, poses, and movements when generating 3D designs for humans [ 44 , 8 ] . To optimize for human interaction, various objective functions and evaluation metrics are defined [ 53 ] . Several previous works have explored this direction, e.g., in 3D room layout generation [ 47 ] ,

Read the complete text