HandMade : Spatial Prompting for Generative 3D Creation with Part-Labeled VR Sketches

Jialin Huang, Rana Hanocka, Ariel Shamir, Yotam Gingold · 2026

Read HandMade : Spatial Prompting for Generative 3D Creation with Part-Labeled VR Sketches by Jialin Huang, Rana Hanocka, Ariel Shamir, Yotam Gingold (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-27738.txt.

Introduction

Text prompts are a weak way to specify a three-dimensional object. They say little about where parts go or how parts relate. Jialin Huang and three colleagues present HandMade in 2026 to give users a spatial channel. HandMade combines virtual-reality sketching with language. Coarse sketches drawn in VR carry part labels. The sketches act as spatial prompts for existing generative models, not as geometry to rebuild directly.

Segmented VR strokes become multi-view part guidance and structured prompts. Users sketch layout and part relations in 3D. Language still names identity, material, style, and local detail. A technical evaluation used twenty varied examples. HandMade preserved user-authored spatial scaffolds better than text-only and sketch-based baselines. A user study with eight participants added detail. People used sketching for layout and language for identity and materials, both when authoring and when revising. The workflow tolerates noisy, incomplete sketches.

Catalog record

CREATORJialin Huang, Rana Hanocka, Ariel Shamir, Yotam Gingold
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE66 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.27738

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

HandMade : Spatial Prompting for Generative 3D Creation with Part-Labeled VR Sketches Jialin Huang, Rana Hanocka, Ariel Shamir, Yotam Gingold Abstract . Text-to-3D generation lowers the barrier to 3D content creation, but text alone is a weak interface for specifying spatial intent: where parts should be placed, how they relate, and how an object should be organized in 3D. We present HandMade , a workflow that combines VR 3D sketching and language for open-domain 3D asset generation. HandMade treats coarse, part-labeled 3D sketches not as incomplete geometry to reconstruct directly, but as spatial prompts for existing generative models. It converts segmented VR strokes into multi-view part guidance and structured prompts, allowing users to specify object layout and part relationships through 3D sketching while using language for identity, material, style, and local details. A technical evaluation shows that HandMade better preserves user-authored spatial scaffolds than text-only and sketch-based baselines on 20 varied examples. A user study with eight participants characterizes how users make use of 3D sketching for spatial layout and language for identity, materials, and details across initial authoring and subsequent revision. HandMade contributes an interaction paradigm and interface-to-generation pipeline for spatially guided 3D creation. Figure 1. From rough spatial intent to textured 3D assets. Each column shows one HandMade example: the top row contains a user-authored VR sketch segmented into system-assigned color channels and paired with text that names and describes the colored parts; only selected labels are shown for readability. The bottom row shows the final mesh generated by our pipeline, with a second-view inset revealing the 3D form. Two-row teaser gallery with five examples. The top row shows user-authored VR sketches segmented into color-coded parts and paired with text that names and describes the colored parts. The bottom row shows generated textured meshes with small second-view insets. 1. Introduction Many designers and “would-be” designers have rich spatial ideas for 3D objects, but turning those ideas into editable assets still requires substantial modeling expertise. Recent generative models have made 3D generation more accessible by using pretrained image diffusion models and multi-view priors as generative backbones ( 40 ; 26 ; 51 ; 9 ; 43 ) . However, text remains an indirect interface for spatial design intent. A prompt can name an object and describe style, but it is awkward for specifying where parts should go, how large they should be relative to each other, how they connect, or what makes a particular design different from a generic instance. In addition, editing or modifying a 3D model using text can also be cumbersome. These limitations are familiar in HCI work on generative AI: prompt-based workflows can reduce the cost of producing candidates, while also shifting effort to prompt formulation, iteration, evaluation, and repair ( 53 ; 47 ; 54 ) . Recent spatial-control work in 3D generation makes a similar argument from the model side: text and images are often insufficiently specific for geometric control, motivating explicit spatial inputs during generation ( 14 ) . Image-to-3D and multi-view-to-3D systems provide a complementary path. They can reconstruct plausible 3D assets from images or multi-view conditions ( 28 ; 29 ; 30 ; 18 ; 49 ; 59 ; 6 ; 20 ; 48 ) . Yet these systems usually assume that the user already has a relatively complete visual reference. For early-stage asset creation, especially for non-experts, the desired input is often less polished: a rough arrangement of parts, a few words about identity and style, and a willingness to iterate once concrete results appear. In contrast, sketching is a natural alternative. Sketch-based modeling has long shown that drawing can support rapid shape ideation, from 2D sketch-to-surface systems such as Teddy ( 21 ) to 3D curve and freeform-surface interfaces ( 2 ; 36 ) . For 3D generation, VR has the advantage that users can sketch directly in the same space where the object will eventually exist, giving them a way to express rough volume, orientation, part placement, and cross-view relationships ( 22 ; 1 ; 5 ) . However, the challenge is that freehand 3D sketches are far from complete 3D models. They are sparse, incomplete, noisy, and often deliberately abstract. A few strokes can indicate a car body, a volcano, or a playground structure to a human collaborator, but those strokes do not comply with the dense visual evidence expected by current image-to-3D systems and are far outside the category assumptions of many sketch-to-3D methods. We present HandMade , a workflow that combines VR 3D sketching and language for open-domain 3D asset generation and editing. Users draw a rough 3D sketch part-by-part and provide a short text description. The 3D sketch anchors the spatial arrangement of parts, while language supplies object identity, materials, and semantic constraints. HandMade ’s pipeline renders 3D strokes into multi-view part guidance, constructs structured prompts that bind user-provided part labels to sketch colors, generates multi-view images, and reconstructs a textured mesh using a modern image-to-3D model (Figure 2 ). HandMade uses existing image-generation and image-to-3D backbones as modular components and leverages their open-domain generality. This choice reflects both data and quality constraints. Training a dedicated sketch-to-3D model would require paired VR sketches, part labels, text descriptions, and target 3D assets at a scale not available for this input setting. Pipelines such as Instant3d ( 25 ) and Hunyuan3D-2mv ( 48 ) expect consistent and precise image-guidance, unlike the inaccurate and imprecise guidance provided by a 3D sketch, whereas pipelines such as Instant3dit ( 4 ) or CompoSE ( 45 ) expect extremely simple bounding shapes. Our contribution is the pipeline that converts rough spatial intent into visual conditions these general-purpose models can use. Figure 1 shows representative examples, pairing user-authored part-labeled 3D sketches with the textured meshes produced by the pipeline. We compare HandMade with text-only and sketch-based baselines, and evaluate its usability with a user study. The technical evaluation tests whether outputs preserve the authored 3D spatial sketches on 20 varied examples. In the user study, eight participants used the VR prototype to author objects of their own imagination, revise inputs after seeing generated results, and reflect on where 3D sketching and language helped or failed. Their feedback supports the intended division of labor: 3D sketches help communicate layout and part relationships, while language helps specify object identity, material, style, and details. To summarize, our contributions are: • A VR authoring workflow that combines 3D sketching, part annotations, and text, and converts them into multi-view visual conditions for open-domain 3D asset generation. • A technical evaluation comparing HandMade with text-only generation and sketch-based baselines, including MeshPad ( 24 ) , LAS-Diffusion ( 66 ) , SketchDream ( 27 ) , and Instant3dit ( 4 ) . • A user study that characterizes how users divide intent between 3D sketching, language, and revision. 2. Related Work Generative 3D systems differ not only in model architecture, but also in the kinds of input they expect from users: text prompts, reference images or views, sketches, spatial controls, or combinations of these signals. This distinction matters for HandMade because our focus is the translation from rough, part-labeled VR 3D sketches and language into the cleaner visual conditions that current open-domain 3D generators can use. Our

Read the complete text