CADCrafter: Generating Computer-Aided Design Models from Unconstrained Images

Cheng Chen, Jiacheng Wei, Tianrun Chen, Chi Zhang, Xiaofeng Yang, Shangzhan Zhang, Bingchen Yang, Chuan-Sheng Foo, Guosheng Lin, Qixing Huang, Fayao Liu · 2025

Read CADCrafter: Generating Computer-Aided Design Models from Unconstrained Images by Cheng Chen, Jiacheng Wei, Tianrun Chen, Chi Zhang, Xiaofeng Yang, Shangzhan Zhang, Bingchen Yang, Chuan-Sheng Foo, Guosheng Lin, Qixing Huang, Fayao Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-04753.txt.

Introduction

Cheng Chen and co-authors present a 2025 computing paper on turning photos of real objects into editable CAD models. Factories want digital twins of physical parts, but scanning and cleanup are costly and slow. A photo-based route would let any user capture a part and get a design file.

Their framework, CADCrafter, trains only on synthetic, texture-free CAD data yet works on real photos. A geometry encoder reads the shape features from an image, and those features ignore texture, which helps the jump to real scenes. Since turning CAD parameters into a solid cannot be differentiated, training lacks direct shape feedback. The team fine-tunes with preference optimization driven by an automatic code checker. The result is a friendlier path from snapshot to parametric model.

Catalog record

CREATORCheng Chen, Jiacheng Wei, Tianrun Chen, Chi Zhang, Xiaofeng Yang, Shangzhan Zhang, Bingchen Yang, Chuan-Sheng Foo, Guosheng Lin, Qixing Huang, Fayao Liu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-04753

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CADCrafter: Generating Computer-Aided Design Models from Unconstrained Images Cheng Chen, Jiacheng Wei, Tianrun Chen, Chi Zhang, Xiaofeng Yang, Shangzhan Zhang, Bingchen Yang, Chuan-Sheng Foo, Guosheng Lin, Qixing Huang, Fayao Liu Abstract † † ∗ The first two authors contributed equally to this work. † † † Corresponding authors. Creating CAD digital twins from the physical world is crucial for manufacturing, design, and simulation. However, current methods typically rely on costly 3D scanning with labor-intensive post-processing. To provide a user-friendly design process, we explore the problem of reverse engineering from unconstrained real-world CAD images that can be easily captured by users of all experiences. However, the scarcity of real-world CAD data poses challenges in directly training such models. To tackle these challenges, we propose CADCrafter, an image-to-parametric CAD model generation framework that trains solely on synthetic textureless CAD data while testing on real-world images. To bridge the significant representation disparity between images and parametric CAD models, we introduce a geometry encoder to accurately capture diverse geometric features. Moreover, the texture-invariant properties of the geometric features can also facilitate the generalization to real-world scenarios. Since compiling CAD parameter sequences into explicit CAD models is a non-differentiable process, the network training inherently lacks explicit geometric supervision. To impose geometric validity constraints, we employ direct preference optimization (DPO) to fine-tune our model with the automatic code checker feedback on CAD sequence quality. Furthermore, we collected a real-world dataset, comprised of multi-view images and corresponding CAD command sequence pairs, to evaluate our method. Experimental results demonstrate that our approach can robustly handle real unconstrained CAD images, and even generalize to unseen general objects. 1 Introduction Computer-aided design (CAD) provides fundamental mechanical components that are essential to create shapes and mechanisms in all manufacturing and design applications. Parametric CAD command sequences enable precise control over shapes and facilitate effortless future modifications on size and scales. However, manual creation of CAD command sequences is tedious and time-consuming, leading to reverse engineering studies to recover CAD design procedures from existing CAD models. Current research focuses predominantly on reconstructing CAD command sequences from 3D representations such as B-Reps Willis et al. 2021a ; Xu et al. 2021 , point clouds Ma et al. 2024 ; Wu et al. 2021 , and voxels Li et al. 2023 ; Li et al. 2024 . These forms are typically derived from synthetic digital data or from high-quality 3D reconstructions obtained using costly 3D sensors. This dependency on sophisticated data and expensive technology limits the feasibility of these methods in practical everyday applications. Figure 1 : Our proposed CADCrafter can generate CAD command sequences from unconstrained multi-domain images, including (from left to right) synthetic data renderings, 3D-printed CAD models, and unseen general objects. These generated CAD commands can then be compiled into 3D CAD models. Notably, our model is trained solely on synthetic data renderings. Recent advances in generative models have facilitated the generation and reconstruction of images to 3D Liu et al. 2024a ; Liu et al. 2024b ; Long et al. 2024 ; Hong et al. 2023 with ease. However, these methods often yield 3D shapes with rough surfaces and indistinct, blurred edges, failing to accurately replicate geometric standards such as rectangles or circles. Moreover, the 3D shapes produced by these methods are difficult to edit and lack the precision required for direct use in manufacturing. This prompts us to investigate the feasibility of directly generating editable CAD command sequences from images, as illustrated in Figure 1 . However, the task is particularly difficult due to the significant representational and domain gap between the two modalities, where CAD commands consist of a mix of discrete geometric operations and continuous parameters while images capture raw appearance with limited spatial information. The challenge is amplified with in-the-wild, unconstrained images. These images frequently exhibit variability in camera poses, lighting conditions, and noise, as well as various materials and textures of the objects depicted. Another challenge in developing this system lies in the difficulty of collecting paired CAD commands and image data from real-world scenarios, making it necessary to rely on synthetic datasets for training. However, models trained solely on synthetic data often underperform with real-world data. Therefore, there is a critical need for a method to bridge this gap, ensuring that approaches trained on synthetic data can perform effectively on both synthetic and real-world unconstrained images. To tackle these challenges, we propose CADCrafter, a method engineered to directly generate CAD command sequences from both multi-view and single-view unconstrained images. Specifically, as shown in Figure 2 , we first train a transformer-based autoencoder to map the CAD tokens to a latent space and then reconstruct them. Then, we apply a latent diffusion transformer to denoise the latent CAD codes conditioned on the input images. Unlike traditional latent diffusion architectures Ye et al. 2023 that rely directly on image features as conditions, our approach utilizes geometric features of images, specifically depth and normal maps. There are two main benefits of geometric features: they enhance geometry representations to boost accurate command prediction and are invariant to the textural gap between synthetic data and real-world images. Additionally, different modalities of geometry features capture various perspectives of an object, with each modal providing unique geometric information. To capitalize on this, we designed a geometry encoder that adaptively consolidates geometric data from each modality. When generating CAD commands, the non-differentiable nature of the CAD compiler makes it inherently challenging to directly incorporate geometric constraints. Due to the geometry precision required in CAD models, inaccurate commands may fail to compile into a valid CAD model, as shown in Figure 3 . To implicitly learn the correct shape pattern of CAD models, we enhance the latent diffusion model with additional constraints and regularization, c.f. Dong et al. 2024 . Drawing inspiration from reinforcement learning with human feedback (RLHF) Bai et al. 2022 , we implement a code checker to improve the validity of denoised latent codes. Specifically, we deploy the CAD compiler as an automatic checker to categorize codes as valid or invalid. Subsequently, we fine-tune the diffusion model using these categorized sets through direct preference optimization Rafailov et al. 2024 ; Wallace et al. 2024 to improve the generation quality and accuracy. Since single-view images inherently lack complete information about the unseen part of a 3D object, instead of training separate models for multi-view and single-view inputs, we distill the comprehensive knowledge from our pre-trained multi-view geometry encoder into a single-view geometric encoder by aligning their feature representations. This enables the model to learn the mapping from single-view to multi-view input, thereby enhancing both the accuracy and robustness when processing single-image inputs. To validate our method, we collected RealCAD, a real-world dataset pairing CAD commands with multi-view images, captured freely on CAD models fabricated using 3D printing technology with various materials and textures. Our contributions are: (a) We introduce CADCrafter, a latent diffusion-based framework to generate parametric CAD models from unconstrained images, leveraging geometric features to mitigate the domain gap between synthetic training data and in-the-wild testing data. (b) We introduce an automatic code checker to learn CAD geometry validity by fine-tuning our diffusion model with direct preference optimization (DPO) thereby improving accuracy and reducing invalid outputs. (c) Our proposed CADCrafter

Read the complete text