Multimodal Refinement of Computer-aided Designs
William P. McCarthy Saujas Vaduguru Karl D. D. Willis Justin Matejka, Judith E. Fan Daniel Fried Yewen Pu · 2025
Read Multimodal Refinement of Computer-aided Designs by William P. McCarthy Saujas Vaduguru Karl D. D. Willis Justin Matejka, Judith E. Fan Daniel Fried Yewen Pu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-20294.txt.
Introduction
William P McCarthy and six co-authors capture something generative AI is bad at, people fixing a design together, step by step, in this 2025 paper. Their mrCAD data set records a communication game. One player, the Designer, sees a target CAD model; the other, the Maker, must build it from instructions sent as text, sketches, or both, refining over rounds until the shapes match.
The corpus holds 6,082 games and 15,163 instruction rounds from 1,092 pairs of players. Analysis shows generation and refinement instructions differ in how they mix drawing and words; fixing leans on different signals than starting. Benchmarked on the task, today's best vision-language models follow fresh generation instructions far better than refinement ones. The authors offer mrCAD as the foundation for studying a multimodal language of revision that earlier data sets simply never recorded.
Catalog record
| CREATOR | William P. McCarthy Saujas Vaduguru Karl D. D. Willis Justin Matejka, Judith E. Fan Daniel Fried Yewen Pu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-20294
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Multimodal Refinement of Computer-aided Designs William P. McCarthy Saujas Vaduguru Karl D. D. Willis Justin Matejka, Judith E. Fan Daniel Fried Yewen Pu Abstract A key feature of human collaboration is the ability to iteratively refine the concepts we have communicated. In contrast, while generative AI excels at the generation of content, it often struggles to make specific language-guided modifications of its prior outputs. To bridge the gap between how humans and machines perform edits, we present mrCAD, a dataset of multimodal instructions in a communication game 1 1 1 dataset repo: https://github.com/AutodeskAILab/mrCAD . In each game, players created computer aided designs (CADs) and refined them over several rounds to match specific target designs. Only one player, the Designer , could see the target, and they must instruct the other player, the Maker , using text, drawing, or a combination of modalities. mrCAD consists of 6,082 communication games, 15,163 instruction-execution rounds, played between 1,092 pairs of human players. We analyze the dataset and find that generation and refinement instructions differ in their composition of drawing and text. Using the mrCAD task as a benchmark, we find that state-of-the-art VLMs are better at following generation instructions than refinement instructions. These results lay a foundation for analyzing and modeling a multimodal language of refinement that is not represented in previous datasets. 1 Introduction Recent advances in generative AI have enabled collaborative design via a natural language interface, with applications such as generating images and code. However, while these models demonstrate impressive results in generating high-quality content, they struggle to modify what they have generated based on further user refinement instructions. The challenge of getting models to perform interactive edits is well-documented in tasks such as image editing Fu et al. (2024) and code generation Cassano et al. (2023) . This often results in frustration with AI-generated content that is “nearly there” but difficult to refine to exactly how the user wanted it. Figure 1: We present a dataset of multimodal instructions for creating and modifying computer aided designs, along accompanying executions of these instructions in a 2D CAD environment. Participants worked together to recreate a target CAD over multiple rounds. Crucially, the target design is known only to the designer, who must instruct the maker on what to create. The maker, in turn, must edit the CAD model based on these instructions. On the other hand, humans regularly instruct each other to perform refinements. Works such as Dingemanse and Enfield (2024) have argued that since human communication is infinitely expressive, repair is a core mechanism for ensuring robust communication. Further, human-given refinements are often multimodal Lawson (2006) ; Williams and Cowdroy (2002) where different modalities interact to make the instruction precise. We present mrCAD, a resource for studying and benchmarking multi-modal refinement in a computer-aided design (CAD) setting. An example data-point is shown in Figure 1 . One player, the Designer , has a target design which they demonstrate via a combination of text and drawing instructions. The other player, the Maker , uses a programmatic CAD language to carry out the Designer’s instructions and attempt to recreate the target. In subsequent rounds, the Designer and Maker iteratively refine the design to get closer to the target. Our task setup produces both (1) natural interaction between modalities: e.g. in Figure 1, the text specifies ‘lines have to be straight’ and the drawing specifies which lines need to be straight and (2) contextual dependence, where refinement instructions need to be interpreted in the context of what has been constructed so far. People naturally use both types of context when generating and interpreting instructions (Sec. 3 ); but we find that models struggle particularly with refinement (Sec. 6 ). Benchmarking As a grounded instruction-following benchmark, mrCAD benefits from the programmatic representation of CAD. CAD reconstruction accuracy can be programmatically defined, in contrast to image creation tasks, which rely on learned metrics such as CLIP Radford et al. (2021) . Additionally, CAD programs can be easily manipulated using a sequence of simple clicks and drags, making it easy to evaluate both tool-using agents and human participants in an evaluation environment. Understanding As a resource for understanding the nature of multimodal refinement in human-human collaboration, mrCAD is large scale, containing 6,082 human-to-human plays of the game, with a total of 15,163 instruction-execution rounds. A large number of CADs in our dataset are recreated multiple times by different dyads, ranging from 2 to 30 games per CAD design. This way, we can understand the variation in refinement strategies across people. The CAD designs in mrCAD are naturalistic, sourced from the SketchGraphs dataset Seff et al. (2020) , consisting of 2D CAD created by designers. A notable feature of CAD is that while it only contains simple curves of lines, circles, and arcs, put together these simple curves form rich semantic objects and sub-parts (like the “flower” and “petals” in Figure 1 ), which must be parsed contextually Ji et al. (2022) . This is in contrast to other scene creation datasets such as Zitnick and Parikh (2013) , where each object in the scene has a canonical, pre-defined name (e.g. a “boy” and a “ball”). The instructions we collect in mrCAD consist of freeform drawings and text, highlighting how humans leverage multimodality and refinement in an open-ended manner. Our work makes the following contributions: • A novel task of multi-modal refinement of CAD designs. • A detailed procedure for collecting a dataset of multi-modal refinement communications using crowd-sourcing. • The dataset and benchmark, consisting of 15,163 instructions and executions, wrapped in an accessible gym environment. • Analyses of human-human communication that reveal differences in the multimodal languages of generation and refinement. • Evaluation of existing VLM models, revealing a severe gap in their refinement ability, particularly compared to their generation ability. 2 The mrCAD Task The mrCAD task (Fig. 1 ) is a two player, multi-turn communication game. Following McCarthy et al. (2024) , a Designer issues multimodal instructions to guide a Maker to take actions in a mutually observable CAD environment to construct a target CAD. The target CAD is known only to the Designer . 2.1 mrCAD environment State The state D is a CAD data structure. ⬇ D -> { Curve , ...} Curve -> Line | Circle | Arc Line -> l ( P , P ) // end points Circle -> c ( P , P ) // points on diameter Arc -> a ( P , P , P ) // start , mid , end Each curve is defined by its control points that the Maker selects. A render function allows players to view the state by rendering the design as an image. Action The Maker can alter the CAD by making, removing, and moving curves via translation. They can also change the shape of a curve by moving its control points. If multiple curves share the same control point, moving the point would modify all the curves, and deleting the point would delete all the curves. ⬇ Action -> make_curve Curve | remove_curve Curve | move_curve Curve Vxy | move_point P P | delete_point P Transition A sequence of actions A=[a_{1},a_{2},\ldots,a_{n}] can be applied to a design, which results