Reconstruction of a 3D wireframe from a single line drawing via generative depth estimation

Elton Cao, Hod Lipson · 2026

Read Reconstruction of a 3D wireframe from a single line drawing via generative depth estimation by Elton Cao, Hod Lipson (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-13549.txt.

Introduction

Cao and Lipson turn a single engineering line drawing into a three-dimensional wireframe by estimating depth for every drawn pixel. Their latent diffusion model respects the visible strokes instead of inventing unseen structure, and it can accept partial depth hints for step-by-step rebuilding. Trained on over three million synthetic pairs from about 90,000 shapes, it roughly halves its depth error when resolution doubles, reaching 3.9 percent error at best.

Catalog record

CREATORElton Cao, Hod Lipson
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42944
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.13549

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Reconstruction of a 3D wireframe from a single line drawing via generative depth estimation Elton Cao, Hod Lipson Abstract The conversion of 2D freehand sketches into 3D models remains a pivotal challenge in computer vision, bridging the gap between human creativity and digital fabrication. Traditional line drawing reconstruction relies on brittle symbolic logic, while modern approaches are constrained by rigid parametric modeling, limiting users to predefined CAD primitives. We propose a generative approach by framing reconstruction as a conditional dense depth estimation task. To achieve this, we implement a Latent Diffusion Model (LDM) with a ControlNet-style conditioning framework to resolve the inherent ambiguities of orthographic projections. To support an iterative “sketch-reconstruct-sketch” workflow, we introduce a graph-based BFS masking strategy to simulate partial depth cues. We train and evaluate our approach using a massive dataset of over one million image-depth pairs derived from the ABC Dataset. Our framework demonstrates robust performance across varying shape complexities, providing a scalable pipeline for converting sparse 2D line drawings into dense 3D representations, effectively allowing users to “draw in 3D” without the rigid constraints of traditional CAD. Project page: https://eltonc01.github.io/sketch-depth-diffusion Figure 1 : Line drawing reconstruction graphic. a) A wireframe sketch, in this case a cube, presents numerous “possible” reconstructions. b) Actual 2D input line drawing, predicted depth map, and 3D reconstruction output. 1 Introduction Sketching has long been an intuitive and effective form of interaction for humans to express ideas and design eissen2008 . Within the process of computer-aided design (CAD), these sketches are often then converted into 3D models to further applications in engineering, manufacturing, and other domains. Humans can intuitively understand this conversion from 2D to 3D mitra2013 . However, implementing this conversion computationally has been a long standing issue; it is unintuitive to draw in 3D, and CAD tools are difficult to use, requiring significant skill to convert complex sketches into 3D li2022 . Recent years have shown a massive surge in deep learning-based 3D reconstruction models xiang2025 ; however, the wireframe line drawings that outline CAD models are inherently incompatible with the natural images that deep learning vision models expect koley2025 . A few differences include the sparseness of sketches, lack of shading, and the explicit geometric constraints associated with CAD models. Thus, the development of a reliable system that can convert such 2D wireframe drawings into 3D is crucial for any design process in manufacturing and engineering. Indeed, approaches designed specifically to work with wireframe line drawings have emerged. Wang et al. wang2022 built specialized transformers to identify wireframe faces, and Li et al. li2022 developed a framework to convert a sketch directly into a sequence of CAD commands, both serving as successful avenues of 3D wireframe sketch reconstruction. The increasing success of models designed specifically for wireframe line drawings has carved out a space dedicated to CAD-related computer vision zhou2023 . However, these approaches unfortunately suffer from the structural constraints of parametric modeling. Approaches working directly in CAD constrain the output to a fixed vocabulary of CAD primitives/mesh topologies wu2021 . Face identification methodologies require sketches to have a set of connecting faces, which many rough or intermediary sketches may lack wang2022 . Generally, current methods require users to sketch out complete structural components before reconstructing, lacking the ability to quickly sketch, reconstruct, then sketch again. In these paradigms, a user is unable to truly “draw in 3D.” As a result, we instead take a much more direct approach by framing the problem as a dense depth estimation task. Provided a depth map of a wireframe sketch, it is relatively straightforward to reconstruct the sketch into 3D. By predicting the per-pixel depth of each sketched stroke, our system can adapt to arbitrary topologies and organic forms that cannot be represented with standard CAD operations. In doing so, any intermediary processing is removed, enabling a smooth and iterative sketch-reconstruct-sketch workflow to truly “draw in 3D” (Figure 2 ). Additionally, by framing the problem in such a way, our approach can tap into a large existing body of techniques and foundational models designed for both image generation and depth estimation rombach2022 ; yang2024 . While per-pixel depth estimation has traditionally been a deterministic learning task, we instead frame it as a generative one due to the ill-posed nature of orthographic wireframe line drawings ke2024 . Natural images contain photometric and perspective cues that enable deterministic prediction, but orthographic line drawings lack these priors, theoretically resulting in an infinite number of valid 3D interpretations per 2D sketch yan1994 . Thus, we implement a variational autoencoder (VAE) and a conditioned denoising U-Net for generative depth estimation. Our work generates a diverse dataset of one million image-depth pairs to simulate a wide variety of line drawings. To enable the ability to continue sketching onto a reconstructed 3D shape (or the sketch-reconstruct-sketch again pipeline), we further augment our dataset with cases of sketches with “partial depth.” In summary, the main contributions of this paper include: • A simple depth-based approach to wireframe line drawing reconstruction via diffusion-based depth estimation. • The capacity to “draw in 3D,” enabling sequential sketching via partial depth and the potential for infinitely complex sketches. 2 Related Work Table 1 : Summary of related works. We list the work, the technique, their respective weaknesses, their dataset size, whether source code is available, and an example of one of the most complex object they have reconstructed. Reference Technique Weaknesses Dataset Size Source Code Most Complex Huffman (1971) huffman1971 Qualitative labeling of lines Cannot determine specific 3D shape. N/A (Symbolic Logic) N/A Lipson & Shpitalni (1996) lipson1996 ; lipson2007 Optimization using geometric priors. Brittle; requires clean vector graphs; fails on noise. N/A (Analytical) N/A Sketch2CAD (2020) li2020 Generates a sequence of CAD commands. Constrained to specific trained primitives \sim 40,000 Open DeepCAD (2021) wu2021 Generates a sequence of CAD commands. Limits organic shapes. 178,238 Open Free2CAD (2022) li2022 Groups strokes into geometric compounds. Limited by predefined primitives. \sim 210k (synthetic) Open Neural Face Identification (2022) wang2022 Predicts topological connectivity. Requires valid closed graphs. \sim 9,370 (ABC) Open Ours (this paper) Monocular depth estimation. Potential fidelity loss. 1,004,051 (ABC) Open 2.1 Constraint-Based and Optimization Approaches Before the advent of data-driven deep learning, the reconstruction of 3D objects from 2D line drawings was primarily formulated as a problem of symbolic logic and geometric optimization lipson1996 . These methods treated a line drawing as an “inverse problem” solvable by enforcing specific geometric priors, including planarity, orthogonality, and symmetry. While early works like Huffman huffman1971 and Sugihara sugihara1986 focused on qualitative line labeling to validate topology, the dominant paradigm for metric recovery was established by Lipson and Shpitalni lipson1996 . Their approach modeled reconstruction as an energy minimization task, searching for the 3D structure that maximized geometric regularity. While this method produced mathematically precise CAD models with flat faces, it was fundamentally brittle; it relied on clean vector graphs and often failed to converge when applied to noisy sketches containing stroke wobble or vertex drift. This brittleness highlights the necessity for modern data-driven approaches. 2.2 Deep Single-View Reconstruction The advent of deep learning shifted the field from hand-crafted constraints to data-driven priors. Breakthroughs in single-view reconstruction (SVR) focused primarily on natural images. Pixel2Mesh wang2018 introduced a

Read the complete text