VLMgineer : Vision Language Models as Robotic Toolsmiths
George Jiayuan Gao, Tianyu Li, Junyao Shi, Yihan Li, Zizhe Zhang, Nadia Figueroa, Dinesh Jayaraman vlmgineer.github.io/release, Email correspondence: {gegao, tianyu}@seas.upenn.edu. \chadded [id=js]and denote equal contribution · 2025
Read VLMgineer : Vision Language Models as Robotic Toolsmiths by George Jiayuan Gao, Tianyu Li, Junyao Shi, Yihan Li, Zizhe Zhang, Nadia Figueroa, Dinesh Jayaraman vlmgineer.github.io/release, Email correspondence: {gegao, tianyu}@seas.upenn.edu. \chadded [id=js]and denote equal contribution (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-12644.txt.
Introduction
George Jiayuan Gao and co-authors present VLMgineer, a system that lets a vision-language model invent tools for a robot. Good tools turn hard tasks into easy ones; animals and humans both exploit this. Their framework writes code for a tool's shape, then evolves designs over rounds. It also drafts the action plan that wields each candidate.
A new benchmark of daily manipulation scenes tests the ideas. The system's inventions beat both human-crafted tools and one-shot model designs. Tasks that stumped plain grippers became routine with the right invented aid. Designs were often surprising yet effective. Code and benchmark will be released for others to build on.
Catalog record
| CREATOR | George Jiayuan Gao, Tianyu Li, Junyao Shi, Yihan Li, Zizhe Zhang, Nadia Figueroa, Dinesh Jayaraman vlmgineer.github.io/release, Email correspondence: {gegao, tianyu}@seas.upenn.edu. \chadded [id=js]and denote equal contribution |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-12644
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
VLMgineer : Vision Language Models as Robotic Toolsmiths George Jiayuan Gao, Tianyu Li, Junyao Shi, Yihan Li, Zizhe Zhang, Nadia Figueroa, Dinesh Jayaraman vlmgineer.github.io/release, Email correspondence: {gegao, tianyu}@seas.upenn.edu. \chadded [id=js]and denote equal contribution Abstract Tool design and use reflect the ability to understand and manipulate the physical world through creativity, planning, and foresight. As such, these capabilities are often regarded as measurable indicators of intelligence across biological species. While much of today’s research on robotic intelligence focuses on generating better controllers, inventing smarter tools offers a complementary form of physical intelligence: shifting the onus of problem-solving onto the tool’s design. Given the vast and impressive common-sense, reasoning, and creative capabilities of today’s foundation models, we investigate whether these models can provide useful priors to automatically design and effectively wield such tools? We present VLMgineer , a framework that harnesses the code generation abilities of vision language models (VLMs) together with evolutionary search to iteratively co-design physical tools and the action plans that operate them to perform a task. We evaluate VLMgineer on a diverse new benchmark of everyday manipulation scenarios that demand creative tool design and use. Across this suite, VLMgineer consistently discovers tools and policies that solve tasks more effectively and innovatively, transforming challenging robotics problems into straightforward executions. It also outperforms VLM-generated designs from human specifications and existing human-crafted tools for everyday tasks. To facilitate future research on automated tool invention, we will release our benchmark and code. Project Website: vlmgineer.github.io . Fig. 1 : Given a manipulation task that lies outside the robot’s capabilities, VLMgineer first prompts a vision language model to generate a tool and action. We then employ evolutionary search in simulation to refine the tool’s geometry and synthesize the corresponding robot motion plan. Finally, the robot, equipped with the automatically designed tool, successfully completes the task. I Introduction Humans exhibit a remarkable ability to design and utilize tools, fundamentally extending their capabilities to accomplish tasks otherwise beyond their reach through creativity, planning, and foresight. This capacity for tool creation and usage represents one of our most distinctive cognitive adaptations, and therefore is widely regarded as a marker of cognitive complexity. \chadded [id=js]Much of today’s robotics research concentrates on enabling complex robot motions to utilize standard pre-designed robot end-effectors or tools [ 1 , 2 , 3 , 4 , 5 ] \chcomment [id=nf]Do not cite papers on arxiv not peer-reviewed. \chadded [id=js]In this work, we pursue an alternative form of physical intelligence: inventing tools that simplify downstream control, thereby shifting the primary problem-solving burden from devising control strategies to designing the tool’s geometry. This requires solving a complex co-design problem over tool designs and the motions that wield those tools. The search space is dauntingly large, and it is clear that the design procedure would benefit from suitable priors to guide this search. \chadded [id=js]One potential source of such priors might be large vision-language models (VLMs) pre-trained on web-scale data. State-of-the-art VLMs have been shown to possess impressive common-sense, reasoning, and creative abilities, alongside extraordinary capabilities in code generation, visual comprehension, and in-context learning. When combined with evolutionary search methods, VLMs have successfully crafted human-level reward functions for reinforcement learning [ 6 , 7 ] , 3D graphics [ 8 ] , articulations of in-the-wild objects [ 9 ] , intricate 3D sculptural designs [ 10 ] , and developing advanced algorithms to solve mathematics and science problems [ 11 , 12 , 13 ] . So, can today’s VLMs also guide the design of innovative and action-efficient physical tools for robots ? We introduce VLMgineer , a \chadded [id=js]fully autonomous framework that leverages VLMs to jointly evolve both tool design and manipulation strategies for robots. Our method demonstrates unprecedented efficacy in developing specialized tools for diverse manipulation tasks, through an evolutionary search process guided by VLM-generated tool geometries and action plans. Compared to prior more limited investigations of tool design \chcomment [id=nf]Which one? add citations, that largely consider parameter optimization for a manually designed parametric template, VLMgineer works off-the-shelf for new tasks without task-specific templates, prompts, or examples. \chadded [id=js]Furthermore, compared to prior works that use VLMs, VLMgineer does not require in-context few-shot examples. To facilitate future research and benchmarking, we also introduce RoboToolBench , a comprehensive simulation suite comprising 12 diverse robotic tool-use \chadded [id=nf]manipulation tasks specifically designed to evaluate tool \chadded [id=nf]design and policy optimization methods. Our fully autonomous approach demonstrates superior task performance over designs generated with human specifications and human-crafted everyday tools. When evaluated on RoboToolBench , VLMgineer achieves an average normalized improvement of \chadded [id=nf]64.7% over VLM-generated designs from human language specifications and outperforms existing human-crafted tools by an average normalized improvement of \chadded [id=nf]24.3%. Our results serve to validate both the physical design intelligence enshrined in VLMs pre-trained on web-scale data, and also present the promise of more adaptable and capable robotics systems that can ingeniously create and use tools. II Related Work \chadded [id=js]Task-Specific Computational Agent and Tool Design. Previous research has extensively investigated methods for optimizing robot morphology, end-effectors, and tool designs for robot manipulation through various computational approaches, ranging from model-based optimization [ 14 ] , reinforcement learning (RL) [ 15 , 16 , 17 ] , data-driven generative models [ 18 , 19 , 20 , 21 ] , and differentiable simulation [ 22 ] . Others have explored robot design for locomotion using evolutionary algorithms [ 23 , 24 , 25 , 26 , 27 , 28 ] , stochastic optimization [ 29 ] , and graph search [ 30 ] . However, these existing approaches typically require manual \chadded [id=js]task-specific pre-definition of a handful of optimization parameters, rely on fixed trajectories or pre-defined control policies, and tend to suffer from low sample efficiency. In contrast, we introduce a VLM-driven approach that simultaneously optimizes both tool design and manipulation strategy, enabling generalization across diverse manipulation tasks without requiring manual parameter specifications. Robot Learning for Tool-Based Tasks. To learn effective tool usage, some have employed learned or simulated dynamics models for tool manipulation optimization [ 31 , 32 , 33 , 34 , 35 ] . Another prevalent approach involves learning tool and object affordances — understanding the functions of objects and tool-object interactions [ 36 , 37 , 38 , 39 , 40 , 1 ] . Recently, large language models have been leveraged for creative tool use [ 41 ] . While these methods all assume that suitable tools already exist in the environment, we instead address the more challenging scenario where a general-purpose robot must concurrently invent the right tool design and the appropriate way to use it for a task. Joint Optimization of Morphology and Control. A common strategy for jointly addressing tool design and manipulation problems is to formulate them as nonlinear programs within a predefined parameter space including both design and manipulation variables, a method that has demonstrated particular effectiveness in sequential manipulation tasks involving extended horizons [ 42 , 43 ] . However, given our objective to deploy VLMgineer in any arbitrary environment without manual specification of design parameters, we rely on the underexplored physical creativity of VLMs. Approaches using RL [ 44 , 45 , 46 , 47 ] , gradient-based optimization [ 48 ]