InstructRobot: A Model-Free Framework for Mapping Natural Language Instructions into Robot Motion
Iury Cleveston, Alana C. Santana, Paula D. P. Costa, Ricardo R. Gudwin, Alexandre S. Simões, Esther L. Colombini · 2025
Read InstructRobot: A Model-Free Framework for Mapping Natural Language Instructions into Robot Motion by Iury Cleveston, Alana C. Santana, Paula D. P. Costa, Ricardo R. Gudwin, Alexandre S. Simões, Esther L. Colombini (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-12861.txt.
Introduction
Iury Cleveston and five co-authors present a 2025 framework that turns spoken or typed orders into robot motion, with no preset body model. Teaching robots from words is a key step for easy human control. Older methods need huge data sets and suit only robots with at most six joints. That leaves complex robots out of reach.
The framework, called InstructRobot, learns word sense and joint motion at the same time through trial and reward. It was tested on a robot with twenty-six turning joints doing object handling tasks. Results held up well in lifelike settings. Since it needs no large data set, the method suits fields where data are scarce.
Catalog record
| CREATOR | Iury Cleveston, Alana C. Santana, Paula D. P. Costa, Ricardo R. Gudwin, Alexandre S. Simões, Esther L. Colombini |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-12861
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
InstructRobot: A Model-Free Framework for Mapping Natural Language Instructions into Robot Motion Iury Cleveston, Alana C. Santana, Paula D. P. Costa, Ricardo R. Gudwin, Alexandre S. Simões, Esther L. Colombini Abstract The ability to communicate with robots using natural language is a significant step forward in human-robot interaction. However, accurately translating verbal commands into physical actions is promising, but still presents challenges. Current approaches require large datasets to train the models and are limited to robots with a maximum of 6 degrees of freedom. To address these issues, we propose a framework called InstructRobot that maps natural language instructions into robot motion without requiring the construction of large datasets or prior knowledge of the robot’s kinematics model. InstructRobot employs a reinforcement learning algorithm that enables joint learning of language representations and inverse kinematics model, simplifying the entire learning process. The proposed framework is validated using a complex robot with 26 revolute joints in object manipulation tasks, demonstrating its robustness and adaptability in realistic environments. The framework can be applied to any task or domain where datasets are scarce and difficult to create, making it an intuitive and accessible solution to the challenges of training robots using linguistic communication. Open source code for the InstructRobot framework and experiments can be accessed at https://github.com/icleveston/InstructRobot . I INTRODUCTION The ability to communicate effectively with robots is essential for completing complex tasks. Accurately translating verbal commands into physical actions is a promising task [ 1 , 2 ] , but still presents challenges [ 3 , 4 , 5 ] . They often come with a high cost in terms of training, requiring vast datasets to train the models. Furthermore, jointly learning language and motion only works for robots with a maximum of 6 degrees of freedom (DoF), where the inverse kinematics model is already known [ 6 , 7 , 8 ] . These limitations hinder the progress of the field, making it impractical to learn, for complex robots, robot motion without a prior kinematics model and limiting the use of natural language to manipulate robots in everyday life. Fig. 1: An illustration of the InstructRobot framework. Our framework enables natural language instructions to be mapped into robot motion without requiring the creation of datasets or knowledge of inverse kinematic models. In this work, we present InstructRobot, a framework that maps natural language instructions into robot motion without requiring the construction of large datasets or prior knowledge of the robot’s inverse kinematics model. InstructRobot employs a reinforcement learning (RL) algorithm that is highly adaptable and easy to implement, making supervised training unnecessary. Our approach simplifies the learning process and offers an intuitive and accessible solution to the challenges of training robots using linguistic communication. Our experiments demonstrate the joint learning of robot motion and language representations, a significant advancement in simplifying the training process. In addition, the framework’s modular approach enhances its flexibility, allowing the inclusion of new perceptual modules and efficient adaptations to various tasks and domains. To validate the effectiveness of the proposed framework (Figure 1 ), this work employs the NAO robot [ 9 ] in object manipulation tasks. NAO is an extremely complex robot with 26 revolute joints that must be controlled in a highly realistic environment. This deliberate choice highlights the robustness and adaptability of the proposed framework, demonstrating its ability to deal with highly demanding environments and tasks. InstructRobot allows multi-modality of input data, and any architecture or reinforcement learning algorithm can be used to formulate the agent. The fundamental aspect of our framework is to assemble an environment in such a way the agent is encouraged to explore and learn an RL policy that better maps natural language instructions into robotic motion. Searching for an RL policy by exploring the environment avoids the task generalization issues frequently present when dataset construction is needed [ 6 ] . The framework can be applied to any task or domain, especially where datasets are scarce and difficult to create. Therefore, the main contributions of our work are: 1. To the best of our knowledge, InstructRobot is the first instructional framework that learns language representations and model-free kinematics motion simultaneously; 2. To the best of our knowledge, this is the first work in the literature to validate an instructional learning approach using a complex 26-revolute joint robot like NAO; 3. To the best of our knowledge, this is the first work that uses reinforcement learning to map instructions to motion, eliminating the need for large datasets; 4. Validation of the framework’s effectiveness through challenging object manipulation experiments using state-of-the-art neural networks trained via reinforcement learning; 5. Open source code for the InstructRobot framework and experiments can be accessed at https://github.com/icleveston/InstructRobot . II RELATED WORK Recent studies have examined the idea of teaching robots through natural language. One approach was proposed by Ahn et al. [ 10 ] and emphasizes a robot’s need to refer to its task history, particularly when executing a series of pick-and-place manipulations through instructions given sequentially. The authors created a history-dependent dataset with pick-and-place operations instructed through natural language. A human provides the robot with linguistic instructions to move certain blocks individually to build a specific structure. The robot’s main objective is to estimate the target block’s position before and after executing the instructions. The dataset provides synthetic RGB images of the workspace, a set of natural language instructions written by six annotators, the objects’ bounding boxes, and heat maps showing the target object. Six annotators describe the same operation for each task, in their own style. The model is trained via supervised learning, using the heatmap as ground truth. Kuo et al. [ 11 ] conducted a study on trajectory prediction systems that utilize linguistic representations, where the model is trained using sample trajectories with labels. However, the study highlights the need for motion prediction datasets with natural language annotation. To address this limitation, the authors expanded the Argoverse [ 12 ] dataset with synthetic language descriptions, which were limited by the types of filters used to generate these descriptions. They also sampled 40,000 trajectories from the Waymo [ 13 ] dataset and employed human labor to annotate their labels. The authors observed that annotated sentences had a much more diverse vocabulary than synthetic language, but the annotation process was quite laborious. Due to the limitations of the available datasets, some studies have been restricted to examining how robots understand natural language through tasks that do not require physical actions in the real world. These tasks can be performed using conventional datasets consisting of pairs of image-instruction previously developed in the literature. For instance, Cheang et al. [ 14 ] focused on inferring the objects’ category and estimating the 6-DoF information for invisible objects of known classes from natural language instructions provided to a robot. They used RefCOCO [ 15 ] as a benchmark because it is a more established benchmark in the literature, and existing ground truth is available. Additionally, Yang et al. [ 16 ] explored object attributes in disambiguation and developed an interactive understanding system that can resolve ambiguities through dialogues. Various robot manipulation tasks require different skills and formats, including following instructions [ 7 ] , one-shot imitation [ 8 ]