Deep Sketch-Based 3D Modeling: A Survey Volume: 44 Issue: 2
Alberto Tono 1,2,∗ Jiajun Wu 1 Gordon Wetzstein 1 Iro Armeni 1 Hariharan Subramonyam 1 James Landay 1 Martin Fischer 1 1 Stanford University, USA, atono, jiajunw, gordonwz, iarmeni, harihars, landay, [email protected] 2 Computational Design Institute, USA, [email protected] ∗ Corresponding author: [email protected] · 2026
Read Deep Sketch-Based 3D Modeling: A Survey Volume: 44 Issue: 2 by Alberto Tono 1,2,∗ Jiajun Wu 1 Gordon Wetzstein 1 Iro Armeni 1 Hariharan Subramonyam 1 James Landay 1 Martin Fischer 1 1 Stanford University, USA, atono, jiajunw, gordonwz, iarmeni, harihars, landay, [email protected] 2 Computational Design Institute, USA, [email protected] ∗ Corresponding author: [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-03287.txt.
Introduction
Alberto Tono and six Stanford colleagues survey a decade of work on turning sketches into three-dimensional models with deep learning. Sketching is a natural way to design, but sketches are abstract and ambiguous, so older software struggled to read them. Their 2026 survey names the new field Deep Sketch-Based 3D Modeling and maps what has been tried.
The map, called MORPHEUS, sorts systems by their human inputs, the model parts and options they output, and how users view and judge the results. Across the map, the authors flag weak spots and open chances for vision, graphics, and interface researchers. Two needs stand out: outputs that carry more information, and controls that follow a designer's intent more faithfully.
Catalog record
| CREATOR | Alberto Tono 1,2,∗ Jiajun Wu 1 Gordon Wetzstein 1 Iro Armeni 1 Hariharan Subramonyam 1 James Landay 1 Martin Fischer 1 1 Stanford University, USA, atono, jiajunw, gordonwz, iarmeni, harihars, landay, [email protected] 2 Computational Design Institute, USA, [email protected] ∗ Corresponding author: [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 112042 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.03287
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Deep Sketch-Based 3D Modeling: A Survey Volume: 44 Issue: 2 Alberto Tono 1,2,∗ Jiajun Wu 1 Gordon Wetzstein 1 Iro Armeni 1 Hariharan Subramonyam 1 James Landay 1 Martin Fischer 1 1 Stanford University, USA, atono, jiajunw, gordonwz, iarmeni, harihars, landay, [email protected] 2 Computational Design Institute, USA, [email protected] ∗ Corresponding author: [email protected] Abstract In the past decade, advances in artificial intelligence have revolutionized sketch-based 3D modeling, leading to a new paradigm known as Deep Sketch-Based 3D Modeling ( DS-3DM ). DS-3DM offers data-driven methods that address the long-standing challenges of sketch abstraction and ambiguity. DS-3DM keeps humans at the center of the creative process by enhancing the flexibility, usability, faithfulness, and adaptability of sketch-based 3D modeling interfaces. This paper contributes a comprehensive survey of the latest DS-3DM within a novel design space: MORPHEUS . Built upon the Input-Model-Output (IMO) framework, MORPHEUS categorizes M odels outputting O ptions of 3D R epresentations and P arts, derived from H uman-inputs (varying in quantity and modality), and E valuated across diverse U ser-views and S tyles. Throughout MORPHEUS we highlight limitations and identify opportunities for interdisciplinary research in Computer Vision, Computer Graphics, and Human-Computer Interaction, revealing a need for controllability and information-rich outputs. These opportunities align design processes more closely with user’ intent, responding to the growing importance of user-centered approaches. \STAR ccs Computing methodologies Computer vision representations ccs Human-centered computing User interface toolkits ccs Computing methodologies Graphics input devices † † year: 2025 † † year: 2025 † † editors: A. Bousseau and A. Dai † † editors-short: D. Ceylan and T.-M. Li † † editors-star: Y. Gryaditskaya and P. Memari † † editors-tutorial: K. Hildebrandt and R. Mantiuk † † editors-education: R. Kuffner dos Anjos and K. Rodriguez Echavarria † † editors-poster: T. Günther and Z. Montazeri † † editors-dc: L. Liu and G. Singh † † subject: EUROGRAPHICS CGF Vol No † † orcid: 0000-0003-0066-7406 † † orcid: 0000-0002-4176-343X † † orcid: 0000-0002-3450-0447 † † orcid: 0000-0002-4230-5916 † † orcid: 0000-0002-3450-0447 † † orcid: 0000-0003-1520-8894 † † orcid: 0000-0002-5071-017X † † teaser: Illustrates our design space MORPHEUS organized around the Input-Model-Output (IMO) framework used to survey deep sketch-based 3D modeling ( DS-3DM ) methods. The Input is categorized into the amount of sketches, their type of viewpoint, and sketch styles with their respective subcategories. The Model is categorized based on the architecture type: neural networks, implicit functions, diffusion models, transformers, differentiable renderers, and foundation models. The Output is categorized by the 3D output compositions, such as part-based semantic, number of options, and geometric topology as well as their corresponding subcategories. The background displays the qualitative and quantitative metrics used to evaluate DS-3DM . The figure encapsulates the paper’s goal: enabling a user-controlled process driven by human-centric metrics to support informed design. 1 Introduction In the iconic scene from The Matrix Reloaded movie, when Neo (Thomas A. Anderson) meets ‘The Architect,’ the camera transitions from a panoramic view of the galaxy to a pencil. This cinematic moment encapsulates the essence of sketch modeling: a universal method capable of translating the Architect’s intent from the physical world to the digital one. Indeed, 2D sketches are a simple yet effective tool for rapidly communicating complex and abstract concepts [ HE17b ] . Today, the challenge lies in bridging the gap between the physical and digital medium, especially for 3D modeling, by removing intrinsic sketch ambiguities and ensuring that the user’s intent is accurately captured and conveyed to the final 3D digital representation. Addressing this gap will be a significant step toward democratizing the design process, thereby empowering individuals to articulate their concepts through sketches. A critical component of this effort involves evaluating the alignment between the user’s intent and the output. This alignment requires the development and iteration of human-centric metrics that assess how well the generated 3D models reflect the user’s original vision. Key questions arise: How can we quantify the fidelity of this alignment? What specific metrics should be used to evaluate both the semantic richness and geometric accuracy of the output? How do we ensure that the information embedded in the 3D model, such as annotations, context, or material properties, enhances usability and supports decision-making? Addressing these questions is essential for ensuring an intuitive and effective transition from physical sketches to digital 3D representations while fostering a more inclusive and user-centered approach to 3D modeling. To support this endeavor, we provide insights into these metrics and potential research directions in Section 6.3 , and in Table 6 . Sketch-Based Interfaces for Modeling (SBIM) [ LM01 , LM95a , LM95b , SG00 , Sut63 , IMT99 , ZHH96 , NJC∗22 , OSCSJ09 , DL16 , BAC∗19 , FAZ21 , LEL∗25 ] predominantly utilize 2D hand-drawn sketches as input, with limited exploration of 3D sketching interfaces [ LCX∗23 , GSL∗25 , XKK∗24 ] . The input modalities include hand-drawn doodles (abstract drawings made by amateur designers in a few seconds), scaffolding-based sketches [ HLW∗17 ] , one-, two-, and three-point perspectives [ THAF24 ] , axonometric sketches [ LPBM22 ] , freehand sketches [ WLY∗20 ] , scribbles [ SJR∗23 ] , contour-based sketches [ JFD20 ] , and design sketches [ ZQG∗20 ] . These modalities leverage the human’s innate ability to communicate design ideas [ DDS∗09 ] visually, which is deeply rooted in human cognition and culture [ OSCSJ09 ] . However, while sketches are a powerful tool for expression, they capture only partial representations of the users’ comprehensive design intent, as illustrated in Figure 1 . These inherently ambiguous and incomplete representations impede current SBIM methods from accurately translating users’ intent into precise 3D geometric outcomes. As researchers, we might wonder how SBIM can solve this problem, capturing the unsketched knowledge and adequately communicating the user’s intent to an information-rich 3D representation [ YSR∗20 ] . To address this, modern approaches are increasingly relying on data-driven methods [ LOM∗18 ] to infer missing information from datasets of paired shapes and sketches. To ensure that designers’ intentions are effectively translated and understood by all, this paper explores how SBIM can capture unsketched knowledge and predict the user’s intent. Figure 1: The initial sketch in the architectural design process contains partial massing, volumetric, and geometric information [ THAF24 ] . This early representation is incomplete because it represents a building only from its front view, conveying only partial information lacking a comprehensive 3D understanding. This single perspective leaves out other buildings’ details, particularly the rear [ YSR∗20 ] . Moreover, this missing information is influenced by extrinsic and intrinsic factors. Extrinsic factors rely on contextual information such as location, surrounding neighborhood, and local climate. For example, the building will have different shapes if it is designed in New York, Padua, or Palo Alto. Intrinsic factors, such as building typology (office, school, house), design constraints, materials, textures, and appearance reside in the architect’s vision of how to translate client requirements into physical spaces. Some SBIMs ensure that the designers’ intentions are translated in the 3D space by learning the relationship between sketches and 3D shapes, modeled as a conditional distribution between these two modalities in the training dataset. This learned mapping enables the model to infer, predict, and complete the overall form