RobotDesignGPT: Automated Robot Design Synthesis using Vision Language Models
Nitish Sontakke K. Niranjan Kumar Sehoon Ha · 2026
Read RobotDesignGPT: Automated Robot Design Synthesis using Vision Language Models by Nitish Sontakke K. Niranjan Kumar Sehoon Ha (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-11801.txt.
Introduction
Nitish Sontakke, K. Niranjan Kumar, and Sehoon Ha build a tool that designs robots with the help of vision language models. Robot design is slow work by hand. A design must meet user needs, move in the right way, and look right, all at once.
Most past tools are rule based and need a set grammar of parts. Their tool, named RobotDesignGPT in the work, uses a model that reads both text and images. Such a tool could let non-experts sketch a robot brief and get a first design fast. Human skill would then refine the result.
Catalog record
| CREATOR | Nitish Sontakke K. Niranjan Kumar Sehoon Ha |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-11801
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
RobotDesignGPT: Automated Robot Design Synthesis using Vision Language Models Nitish Sontakke K. Niranjan Kumar Sehoon Ha Abstract Robot design is a nontrivial process that involves careful consideration of multiple criteria, including user specifications, kinematic structures, and visual appearance. Therefore, the design process often relies heavily on domain expertise and significant human effort. The majority of current methods are rule-based, requiring the specification of a grammar or a set of primitive components and modules that can be composed to create a design. We propose a novel automated robot design framework, RobotDesignGPT , that leverages the general knowledge and reasoning capabilities of large pre-trained vision-language models to automate the robot design synthesis process. Our framework synthesizes an initial robot design from a simple user prompt and a reference image. Our novel visual feedback approach allows us to greatly improve the design quality and reduce unnecessary manual feedback. We demonstrate that our framework can design visually appealing and kinematically valid robots inspired by nature, ranging from legged animals to flying creatures. We justify the proposed framework by conducting an ablation study and a user study. I INTRODUCTION Robot design is a fundamental task in robotics, which significantly influences the final performance. This task can be roughly categorized into two subproblems: design optimization and design synthesis. In design optimization, the algorithm receives an initial design as input and then tries to optimize the continuous design parameters, such as link lengths and body dimensions, using gradient-based optimization methods. However, the performance of such an optimization is often heavily dependent on the initial design. This brings us to robot design synthesis, which aims to generate effective designs from scratch based on the user description. Despite its tremendous potential, synthesizing robot designs is a challenging problem because it requires a deep understanding of both mechanical design and motion control. These challenges limit the choice of algorithms to relatively simple approaches, such as evolutionary algorithms [ 8 , 59 ] . Fig. 1 : Flowchart depicting an overview of our method. In this work, we leverage Vision Language Models (VLMs) to advance the state-of-the-art in robot design synthesis. We draw inspiration from recent advances in various robotics domains such as manipulation [ 36 , 14 , 7 ] and navigation [ 33 , 58 , 52 ] that use VLMs for open-ended reasoning. VLMs are well-suited for robot design synthesis and provide a natural language interface for the user to describe the requirements. Our work focuses on two specific features of VLMs that have special significance for robot synthesis. First, their baked-in general knowledge enables them to understand the anatomy of different species and in turn reason about composing complex structures with multiple simple shapes. Second, VLMs can directly translate this knowledge into commonly used robot description formats, such as URDF [ 43 ] and MJCF [ 41 ] . While promising, a naive application of VLMs using few-shot prompts is insufficient for synthesizing valid robot designs. This is because robot synthesis involves long contexts and spatial reasoning which often lead to hallucinations from the VLM that deviate from the space of physically valid designs. We propose a framework for synthesizing kinematically valid robot designs based on simple user inputs. Our approach takes as input a text label (such as “rhinoceros” or “ladybug”) and an optional reference image and first generates a structural design by querying the VLM. It then iterates through the structural elements using our RobotDesignGPT API to synthesize joints and bodies. To ensure connectivity and kinematic validity, we provide compact APIs specifically designed for VLMs. We develop a visual-based automated feedback mechanism that interprets the generated designs, compares them with the user-provided text descriptions and image inputs, and automatically improves on the designs iteratively. Finally, the user has the option to provide stylistic preferences to the VLM to allow for further customization. We demonstrate that the proposed framework generates a diverse range of robot designs, such crab, dragonfly, rabbit, penguin, ostrich, and more, using just 2.15 iterations of human feedback on average. The generated designs then can be further integrated with motion planning modules to generate plausible robot motions. To evaluate the quality of our results with independent viewers, we conducted a user study with 44 respondents who were asked to rate the generated robot designs in terms of how closely they resemble the reference creature, achieving an average user satisfaction score of 2.93 out of 5 . We also conducted an ablation study to show that automated visual feedback is crucial for generating convincing robot designs. To the best of our knowledge, this is the first framework capable of synthesizing kinematically valid articulated rigid body designs from user inputs without relying on manually defined rules or templates. Our technical contributions are as follows. 1. We propose a VLM-based design synthesis framework capable of generating diverse kinematic robot designs, from legged robots to flying creatures, 2. We further enhance design quality by incorporating automated visual feedback from a reference image and limited human feedback, 3. We demonstrate that the generated designs are kinematically valid and capable of producing compelling motions using off-the-shelf motion planners. II RELATED WORK II-A Robot Co-Design via Optimization The performance of a robot on a given task depends on its motion controller, which is in turn tightly coupled with the robot’s design. Automated robot design aims to maximize the performance with respect to the given task by simultaneously optimizing the robot’s design parameters and its motion controller. Current state-of-the-art methods include evolutionary algorithms [ 8 , 59 ] , guided optimization with user-specified motion trajectories [ 10 , 9 , 11 ] , grammar-guided approaches [ 57 , 56 , 13 ] , and Bayesian optimization [ 4 , 28 ] . More recently, the research community has also observed an increase in learning-based approaches, including deep reinforcement learning [ 32 , 19 , 12 ] , meta reinforcement learning [ 3 ] , hierarchical reinforcement learning [ 38 ] , transformer-based methods [ 53 ] , and diffusion-based methods [ 44 , 50 ] . The use of natural language as an input modality, such as in Text2Robot [ 30 ] , is also being actively explored. II-B Large Language and Vision Models in Robotics Large Language Models (LLMs) and Vision Language Models (VLMs) have emerged as innovative solutions in numerous research domains. These models have also been effectively leveraged for various robotic use cases, such as manipulation [ 36 , 14 , 7 ] , navigation [ 33 , 58 , 52 ] , mobile manipulation [ 51 , 47 , 27 ] , and locomotion [ 45 , 40 ] , by offering baked-in general knowledge coupled with commonsense and chain-of-thought reasoning capabilities. This ability to reason over long horizons makes them particularly effective as task and motion planners. To this end, we have seen LLMs and VLMs being successfully utilized in task and motion planning (TAMP) [ 34 , 6 , 39 ] . In addition, LLMs and VLMs have also been integrated into various parts of the developmental pipeline itself, including reward engineering [ 22 , 54 , 48 ] , code generation [ 20 , 2 ] , motion generation [ 16 ] , curriculum