CAD-GPT: Synthesising CAD Construction Sequence with Spatial Reasoning-Enhanced Multimodal LLMs
Siyu Wang, Cailian Chen, Xinyi Le, Qimin Xu, Lei Xu, Yanzhou Zhang, Jie Yang · 2024
Read CAD-GPT: Synthesising CAD Construction Sequence with Spatial Reasoning-Enhanced Multimodal LLMs by Siyu Wang, Cailian Chen, Xinyi Le, Qimin Xu, Lei Xu, Yanzhou Zhang, Jie Yang (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-19663.txt.
Introduction
Siyu Wang and co-authors present CAD-GPT, a 2024 system that writes the construction steps for a computer-aided design model from a picture or a sentence. Language-driven design tools stumble on exactly where in 3D space each sketch starts and which way it extrudes, so models come out misplaced.
Their fix adds a spatial mechanism inside a multimodal large language model. Three-dimensional positions and sketch plane angles are unfolded into a one-dimensional language-friendly code, while 2D sketch coordinates are discretized onto a grid. With that encoding, the model infers starting points, orientations, and extrusion directions far more precisely.
Catalog record
| CREATOR | Siyu Wang, Cailian Chen, Xinyi Le, Qimin Xu, Lei Xu, Yanzhou Zhang, Jie Yang |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-19663
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CAD-GPT: Synthesising CAD Construction Sequence with Spatial Reasoning-Enhanced Multimodal LLMs Siyu Wang, Cailian Chen, Xinyi Le, Qimin Xu, Lei Xu, Yanzhou Zhang, Jie Yang Abstract Computer-aided design (CAD) significantly enhances the efficiency, accuracy, and innovation of design processes by enabling precise 2D and 3D modeling, extensive analysis, and optimization. Existing methods for creating CAD models rely on latent vectors or point clouds, which are difficult to obtain and costly to store. Recent advances in Multimodal Large Language Models (MLLMs) have inspired researchers to use natural language instructions and images for CAD model construction. However, these models still struggle with inferring accurate 3D spatial location and orientation, leading to inaccuracies in determining the spatial 3D starting points and extrusion directions for constructing geometries. This work introduces CAD-GPT, a CAD synthesis method with spatial reasoning-enhanced MLLM that takes either a single image or a textual description as input. To achieve precise spatial inference, our approach introduces a 3D Modeling Spatial Mechanism. This method maps 3D spatial positions and 3D sketch plane rotation angles into a 1D linguistic feature space using a specialized spatial unfolding mechanism, while discretizing 2D sketch coordinates into an appropriate planar space to enable precise determination of spatial starting position, sketch orientation, and 2D sketch coordinate translations. Extensive experiments demonstrate that CAD-GPT consistently outperforms existing state-of-the-art methods in CAD model synthesis, both quantitatively and qualitatively. Project Page — https://OpenIWIN.github.io/CAD-GPT/ Introduction Figure 1: Demonstration of various CAD models generated by CAD-GPT. The models in the image demonstrate semantic sketch generation capabilities (e.g., a heart shape and the letter ”E”), category-based CAD generation capabilities (e.g., a table, a chair, and a key), spatial reasoning abilities (e.g., a table and mutually perpendicular cylinders), and the capability to generate identical models with varying dimensions (e.g., three connectors with two circular holes of differing sizes). Computer-Aided Design (CAD) has become the standard approach for designing, drafting, and modeling in a wide range of industries ( Robertson and Allen 1993 ; Chen and Olechowski 2024 ) . Almost every manufactured object that exists today started its life in a parametric CAD tool. The CAD command sequence is one type of CAD model representation. It is described as a sequence of operations such as drawing 2d sketches and extruding sketches into 3D solid shapes ( Wu, Xiao, and Zheng 2021 ) . Constructing these CAD models requires domain expertise and spatial inference capabilities, and it can also be time-consuming. Recently, the most popular direction for CAD model generation focused on using generative models like variational autoencoder(VAE) (Wu, Xiao, and Zheng 2021) and vector quantized variational autoencoder(VQ-VAE) (Xu et al. 2022, 2023). These methods map CAD models to vectors or codebooks in a high-dimensional latent space and then reconstruct the original CAD models from these high-dimensional representations. The main limitations of both methods include: 1) The quality of CAD models synthesized by these methods depends not only on the methods’ capabilities but also on the quality of the provided guidance vectors or codebooks, which can inevitably result in cumulative errors. 2) These methods require high dimensional data, similar to the distribution of their vectors or codebooks as inputs, which are difficult to obtain directly. Another line of work directly infers CAD sequences from point clouds(Ma et al. 2023; Khan et al. 2024) or sketches(Li et al. 2022). In practical applications, sketches need to be drawn by professionals, and point clouds require specialized equipment for collection, both of which involve high data acquisition costs. Generative AI tools such as Multimodal Large Language Models (MLLMs) have the potential to remove these barriers. These multimodal models exhibit impressive visual language understanding and generation capabilities ( Achiam et al. 2023 ; Yin et al. 2023 ) . Recently, there have been initial attempts to use state-of-the-art MLLMs for the creation of CAD models ( Makatura et al. 2023 ; Badagabettu, Yarlagadda, and Farimani 2024 ) . Experiments show that these models, such as GPT-4, lack spatial reasoning capabilities ( Makatura et al. 2023 ) and have a low success rate ( Badagabettu, Yarlagadda, and Farimani 2024 ) in generating the desired CAD models. These limitations can manifest as notable challenges in the design and manufacturing domain. For instance, they may generate a car with four horizontally placed wheels or a table with legs that exceed the tabletop and are randomly positioned. Hence, the main question we ask is: How to enhance the 3D spatial reasoning capabilities of multimodal large language models for accurate CAD model synthesis? In this paper, we introduce CAD-GPT, a MLLM with enhanced 3D spatial reasoning capability built upon LLaVA-1.5 7B version ( Liu et al. 2024 ) . For training the model, we constructed a dataset that pairs CAD modeling sequences with natural language descriptions and single fixed-view rendered images of the CAD models. We built our dataset based on the DeepCAD dataset ( Wu, Xiao, and Zheng 2021 ) . To enhance the spatial reasoning capabilities of the model, we developed a 3D spatial localization mechanism specifically tailored for 3D modeling tasks. Concretely, we convert the global spatial 3D coordinates, sketch plane rotation angles into two distinct categories of position tokens by unfolding their characteristics into a 1D linguistic feature space. Additionally, the 2D sketches are discretized and converted into special tokens. These tokens are incorporated into the vocabulary of the base LLM. Simultaneously, we incorporate custom learnable positional embeddings to bridge the gap between language and spatial positions. In summary, our contributions are as follows: • We present CAD-GPT, a MLLM that synthesises CAD modeling sequences precisely from a single image or textual description. To the best of our knowledge, we are the first to develop a MLLM specifically trained for this task. • We designed a novel localization mechanism tailored for the 3D modeling process, enhancing the spatial reasoning capabilities of large-language models by mapping 3D space into 1D through a tokenization method. • Utilizing the DeepCAD dataset, we generated 160k fixed-viewpoint CAD model images and 18k corresponding natural language captions. We plan to release our CAD-GPT model along with the dataset we developed, contributing a valuable resource. • Experiments on the held-out dataset demonstrate that our approach achieves a higher accuracy compared to state-of-the-art baseline models. Figure 2: Overview of our CAD-GPT framework. On the left side, a dashed box contains the overall algorithm framework. The right side provides a detailed view of our 3D Modeling Spatial Localization Mechanism. From top to bottom, it sequentially demonstrates the 3D Orientation, 3D Coordinate Location, showing how they unfold from 3D to 1D along specific directions, as well as the token representation method for the 2D Sketch. Related Work Approximate 3D Representation Accurate and efficient 3D data representation remains a challenge in computer graphics and vision. Point Clouds ( Zhou, Du, and Wu 2021 ; Luo and Hu 2021 ; Nichol et al. 2022 ) capture discrete spatial points, offering simplicity but lacking surface details; Meshes ( Groueix et al. 2018 ; Wang et al. 2018 ; Chen et al. 2024b ; Siddiqui et al. 2024 ) use vertices and edges to form polygons, providing connectivity but facing complexity issues; 3D Gaussians model ( Kerbl et al. 2023 ; Tang et al. 2023 ) use points