VideoCAD: A Dataset and Model for Learning Long - Horizon 3D CAD UI Interactions from Video
Brandon Man, Ghadi Nehme, Md Ferdous Alam, Faez Ahmed · 2025
Read VideoCAD: A Dataset and Model for Learning Long - Horizon 3D CAD UI Interactions from Video by Brandon Man, Ghadi Nehme, Md Ferdous Alam, Faez Ahmed (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-24838.txt.
Introduction
Brandon Man, Ghadi Nehme, Md Ferdous Alam, and Faez Ahmed present VideoCAD in two thousand twenty-five. VideoCAD is a data set and model for learning CAD tool use from video. Design in CAD is slow and needs long, exact chains of acts. Past UI data sets cover short, simple phone or web tasks.
VideoCAD holds tens of thousands of tagged clips of CAD work. They were made by an auto frame that mines human made designs. Task length runs far past past sets. A new model learns CAD acts direct from video and beats clone based rivals. A quiz benchmark tests spatial thought in big models as well.
Catalog record
| CREATOR | Brandon Man, Ghadi Nehme, Md Ferdous Alam, Faez Ahmed |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 96 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-24838
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
VideoCAD: A Dataset and Model for Learning Long - Horizon 3D CAD UI Interactions from Video Brandon Man, Ghadi Nehme, Md Ferdous Alam, Faez Ahmed Abstract Computer-Aided Design (CAD) is a time-consuming and complex process, requiring precise, long-horizon user interactions with intricate 3D interfaces. While recent advances in AI-driven user interface (UI) agents show promise, most existing datasets and methods focus on short, low-complexity tasks in mobile or web applications, failing to capture the demands of professional engineering tools. In this work, we introduce VideoCAD, the first attempt to model UI interactions for precision engineering tasks. Specifically, VideoCAD is a large-scale synthetic dataset consisting of over 41K annotated video recordings of CAD operations, generated using an automated framework for collecting high-fidelity UI action data from human-made CAD designs. Compared to existing datasets, VideoCAD offers an order - of - magnitude increase in complexity for real - world engineering UI tasks, with time horizons up to 20 × 20\times longer than those in other datasets. We show two important downstream applications of VideoCAD : (1) learning UI interactions from professional 3D CAD tools for precision tasks and (2) a visual question - answering (VQA) benchmark designed to evaluate multimodal large language models (LLMs) on spatial reasoning and video understanding. To learn the UI interactions, we propose VideoCADFormer , a state-of-the-art model for learning CAD interactions directly from video, which outperforms existing behavior cloning baselines. Both VideoCADFormer and the VQA benchmark derived from VideoCAD reveal key challenges in the current state of video-based UI understanding, including the need for precise action grounding, multi-modal and spatial reasoning, and long-horizon dependencies. The dataset and code are available at: https://github.com/ghadinehme/VideoCAD 1 Introduction Designing most physical products—from cars to airplanes—relies on professional engineering CAD software such as SolidWorks, Autodesk Inventor, and PTC Onshape. These platforms, with their hundreds of toolbars and menu options, pose significant challenges for users due to their complex interfaces, intricate workflows, and the high degree of precision required to create accurate 3D geometries Piegl (2005) ; Mathur et al. (2020) . Unlike typical consumer applications, where tasks are completed through simple User Interface (UI) interactions, CAD operations require structured, multi-step processes involving 3D spatial understanding and parametric modeling. Mastery of these tools often requires years of experience, and automating their use remains a formidable challenge Lee-Remond et al. (2025) . While recent AI-based approaches aim to automate parts of the CAD workflow, most are limited to text or image modalities You et al. (2024) ; Khan et al. (2024) and fail to learn from the dynamic, video-based nature of CAD interfaces. The absence of large-scale, annotated datasets capturing such interactions further constrains progress in this area. Prior research on UI navigation has primarily focused on web and mobile applications, where tasks are short, low in complexity, and require no 3D reasoning. Benchmarks such as MiniWob++ Shi et al. (2017) and RICO Deka et al. (2017) have provided valuable insights into learning from user interactions, yet they fall short of capturing the complexity of CAD environments—where tasks involve nested dependencies, geometric constraints, and tool-based operations that go far beyond simple button clicks. While behavior cloning from video has shown promise in robotics and gaming, its application to software UI understanding remains largely unexplored due to the lack of large-scale, video-annotated datasets. To address this gap, we introduce VideoCAD , a large-scale synthetic dataset containing over 41K video demonstrations of CAD modeling tasks with fine-grained action annotations. VideoCAD is generated from a public dataset of parametric CAD models, DeepCAD Wu et al. (2021) , originally authored by human designers in Onshape PTC Inc. () —a professional browser-based CAD platform (see Appendix A ). We map the construction sequences of these CAD models into executable UI actions within Onshape, producing realistic, temporally aligned video–action pairs. To demonstrate the effectiveness of our dataset, we provide two important downstream applications of VideoCAD . First, we develop a transformer-based model, VideoCADFormer , that predicts UI interactions, given a single input image of target CAD model. We benchmark our model against state-of-the-art behavior cloning baselines and show that VideoCADFormer outperforms them by up to 20% in learning from rich user interactions for CAD design, demonstrating superior reasoning over long-horizon UI interaction tasks involving 3D understanding. Second, we also introduce VideoCADQA , a synthetically generated multiple-choice VQA dataset derived from VideoCAD that includes questions that evaluate 3D reasoning and video understanding in LLMs. Our benchmark helps uncover a critical gap in spatial reasoning of current LLMs in precise engineering tasks. VideoCAD has the potential to serve as a valuable benchmark for advancing research in AI-driven UI navigation, software automation, and CAD generation. By providing an open-source dataset with rich annotations, we aim to facilitate breakthroughs in learning from software demonstrations and bridging the gap between computer vision, reinforcement learning, and CAD modeling. More specifically, our contributions can be summarized as follows: • A large-scale, high-fidelity CAD interaction dataset: VideoCAD consists of over 41K annotated videos, capturing diverse CAD workflows with both low-level UI actions and high-level modeling operations. • Behavior cloning benchmarks and a state-of-the-art model: We evaluate multiple baselines on VideoCAD and propose VideoCADFormer , a transformer-based architecture that achieves state-of-the-art performance on long-horizon CAD action prediction. • A VQA benchmark for 3D and spatiotemporal reasoning: We present a case study that extends VideoCAD into a VQA benchmark, introducing 1,200 visual questions that probe LLMs’ fine - grained 3D reasoning and temporal understanding over CAD videos. 2 Related works Research on automating software understanding spans UI navigation, learning from demonstrations, and 3D CAD generation. We situate our work at the intersection of these areas—linking long-horizon UI modeling with spatially grounded CAD reasoning. User Interface Navigation. Automating user interface navigation has been a long-standing challenge in AI research. Works such as MiniWob++ Liu et al. (2018) and Android In The Wild (AITW) Rawles et al. (2023) introduced datasets for web and mobile UI interaction, allowing agents to learn structured policies for performing simple tasks. However, these datasets focus on relatively simple tasks with short time horizons. More recent works, such as WebShop Yao et al. (2023) , have extended these capabilities to complex real-world scenarios where an agent must reason over long sequences of actions to achieve its goal. UI Agents and Behavior Cloning from Videos. Learning from demonstrations has been widely applied to UI interaction. Behavioral cloning, where an agent learns from human demonstrations, has been successfully applied in robotic manipulation Chi et al. (2024) and game environments Baker et al. (2022) . However, its application to GUI-based tasks is still underexplored. Recent studies, such as ActionBert He et al. (2021) and AssistGUIs Gao et al. (2024) , explored using large-scale datasets for predicting UI interactions from multimodal inputs. Our work extends these efforts by introducing a dataset specifically tailored for CAD interactions, where the action space is significantly more complex. CAD Datasets for Learning-Based Modeling. Parametric CAD datasets remain scarce compared to mesh or point cloud collections. The ABC dataset offers 1 million B-rep models with precise geometry but lacks procedural data Koch et al. (2019) . DeepCAD supplements this by providing 178K models with full construction histories, enabling sequence-based generative modeling Wu et al. (2021) . Fusion