RoboTron-Mani : All-in-One Multimodal Large Model for Robotic Manipulation
Feng Yan, Fanfan Liu, Yiyang Huang, Zechao Guan, Liming Zheng, Yufeng Zhong, Chengjian Feng, Lin Ma · 2024
Read RoboTron-Mani : All-in-One Multimodal Large Model for Robotic Manipulation by Feng Yan, Fanfan Liu, Yiyang Huang, Zechao Guan, Liming Zheng, Yufeng Zhong, Chengjian Feng, Lin Ma (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-07215.txt.
Introduction
Feng Yan and co-authors build one model for many robot hands in a 2024 paper. Robot manipulation models usually specialize by body and task, and gathering training data is costly. Their model, RoboTron-Mani, starts from a general multimodal network and adds sharper three-dimensional sense through camera parameters and occupancy supervision, plus masking and decoder blocks that keep the senses from blurring.
A companion data set, RoboData, merges several public collections into the first bundle aligning multi-view images, camera data, depth, actions, and space. Trained on it, the model beat specialist experts on the CALVIN benchmark, more than doubling the average task chain length, and it generalized across robot bodies. The authors call it the first generalist policy scored on all tasks at once.
Catalog record
| CREATOR | Feng Yan, Fanfan Liu, Yiyang Huang, Zechao Guan, Liming Zheng, Yufeng Zhong, Chengjian Feng, Lin Ma |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 67 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-07215
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
RoboTron-Mani : All-in-One Multimodal Large Model for Robotic Manipulation Feng Yan, Fanfan Liu, Yiyang Huang, Zechao Guan, Liming Zheng, Yufeng Zhong, Chengjian Feng, Lin Ma Abstract Recently, robotics has advanced significantly through the integration of larger models and large-scale datasets. However, challenges remain in applying these models to 3D spatial interactions and managing data collection costs. To address these issues, we propose the multimodal robotic manipulation model RoboTron-Mani and the comprehensive dataset RoboData . RoboTron-Mani , on one hand, enhances 3D perception through camera parameters and occupancy supervision. On the other hand, it further incorporates Modality-Isolation-Mask and multimodal decoder blocks based on OpenFlamingo, improving modality fusion and fine-grained perception. RoboData integrats several publicly-available datasets, achieving the first fusion of multi-view images, camera parameters, depth maps, actions, and space alignment, which facilitates comprehensive learning from diverse robotic datasets and offers one complete evaluation system. Trained on RoboData , RoboTron-Mani is the first generalist policy that surpasses expert models, enabling simultaneous evaluation of all tasks across multiple datasets, rather than being limited to specific data or task selections. Specifically, RoboTron-Mani boosts manipulation performance by increasing the average sequence length on CALVIN from 1.7 to 3.5, enabling cross-embodiment generalization, and achieving state-of-the-art results on both simulated and real-world datasets. Figure 1 : Stacked bar chart depicting the performance of various models across five datasets. The “SOTA” label represents the best results achieved by specialized models for each dataset. Notably, RoboTron-Mani is the only generalist policy evaluated on multiple datasets, demonstrating competitive performance relative to “SOTA”. For more details, please refer to Section 4.1 . 1 Introduction In recent years, machine learning has experienced profound advancements, from the advent of CLIP [ 56 , 70 ] to the progression of foundational models like the GPT series [ 10 , 1 , 11 ] , Llama [ 62 , 63 ] , LLaVA [ 39 ] , and Flamingo [ 2 , 3 ] . These strides are largely due to larger transformer-based architectures and the utilization of “internet-scale” datasets [ 38 , 15 , 13 ] . These innovations have not only extended the frontiers of natural language processing [ 17 ] and computer vision [ 23 , 57 ] but also galvanized researchers to integrate these models into Embodied Artificial Intelligence (EAI) [ 51 ] , thereby enabling more complex and varied tasks in real-world environments. Figure 2 : Left: RoboData integrates multiple diverse and complex datasets (CALVIN [ 43 ] , Meta-World [ 68 ] , LIBERO [ 36 ] , RT-1 [ 8 ] , RoboCAS [ 71 ] , ManiSkill2 [ 22 ] , RoboCasa [ 49 ] , RLBench [ 27 ] , and Colosseum [ 55 ] ), covering various robot embodiments, environments, and task types, resulting in a unified dataset with standardized input and output spaces. Right: RoboTron-Mani features comprehensive 3D perception capabilities, flexible multimodal outputs, and significantly enhances the robotic manipulation generalization capabilities. In terms of modeling, there has been a gradual shift from single-task or single-dataset learning [ 60 , 44 , 70 ] towards transfer learning approaches [ 9 , 31 , 37 , 64 , 61 , 26 ] . These models leverage robust foundational models that are pre-trained on extensive “internet-scale” datasets or diverse data sources. Subsequently, they undergo fine-tuning on specific robotic datasets to produce precise control actions. On the data front, researchers collect data through various means to augment models. For example, the Open X-Embodiment [ 53 ] amalgamates diverse datasets containing vision-language-action pairs, whereas the RH20T [ 20 ] gathers data via teleoperation. Despite the impressive robustness of these efforts, they still encounter significant challenges in practical applications. First, is the direct application of multimodal models to EAI the optimal solution? Robots should interact with the physical 3D space. However, current multimodal models predominantly focus on 2D image understanding and generation, limiting their practical applicability, especially in physical-world interaction. Second, is it essential to address the cost and efficiency of dataset construction? For instance, collecting around 130,000 episodes from the RT-1 [ 8 ] took 17 months. Therefore, it is imperative to integrate as many existing multi-platform, multi-robot datasets from the industry as possible to address this urgent issue. For tackling the aforementioned challenges, this paper introduces RoboTron-Mani , the multimodal large model for robotic manipulation, and RoboData , the comprehensive dataset integrating datasets across various platforms and robots for evaluation and training purposes. Figure 2 shows the framework and capabilities of our dataset platform and multimodal robot model. In terms of modeling, RoboTron-Mani combines camera parameters and occupancy supervision to enhance 3D environmental perception. Additionally, leveraging large language models like OpenFlamingo [ 3 ] , we design the plug-and-play efficient Modality-Isolation-Mask (MIM), which flexibly introduces multimodal supervision. This gives the model fine-grained perception and leverages large-scale internet data. On the data side, while Open X-Embodiment [ 53 ] integrates multiple datasets, it lacks critical information such as multi-view images, camera parameters, and depth maps, making it more suitable for 2D multimodal training. Additionally, the lack of data space alignment prevents the robot’s 6D pose from being consistent across different datasets, yielding the results in Open X-Embodiment [ 53 ] Table 1, where direct data fusion results in RT-1-X underperforming RT-1. In contrast, RoboData addresses these limitations by integrating a wide range of well-known industry datasets, including simulated datasets (CALVIN [ 43 ] , Meta-World [ 68 ] , LIBERO [ 36 ] , Robomimic [ 41 ] , RoboCAS [ 71 ] , ManiSkill2 [ 22 ] , RoboCasa [ 49 ] , RLBench [ 27 ] , and Colosseum [ 55 ] ), and RT-1 [ 8 ] . We dedicate hundrads of person-days to collect and organize these data, supplementing missing modalities such as depth maps and camera parameters. Moreover, RoboData unifies the input and output spaces across different robots and platforms, ensuring consistent perception and action representation, and enabling integrated learning from diverse simulated and real-world datasets. Figure 3 : Architecture of RoboTron-Mani . Vision Encoder extracts multi-view features. 3D Perception Adapter leverages occupancy supervision to unify features and enhance spatial perception. Feature Fusion Decoder based on LLMs merges text and visual information. Multimodal Decoders enhance fine-grained perception and understanding through multimodal outputs. Extensive experiments demonstrate that the various components of RoboTron-Mani significantly improve performance in robotic manipulation tasks, enhancing the average sequence length on the CALVIN [ 43 ] benchmark from 1.7 to 3.5. Additionally, RoboTron-Mani ensures cross-embodiment capabilities and the first generalist policy that surpasses expert models, enabling simultaneous evaluation of all tasks across multiple datasets, rather than being limited to specific data or task selections, as shown in Figure 1 . This paper underscores the critical role of advanced models and curated datasets in advancing robotics, highlighting their potential to drive significant improvements in robotic performance. In summary, RoboData aims to provide the research community with a comprehensive and fair evaluation system. And RoboTron-Mani is the first generalist policy to incorporate training and testing across multiple datasets. 2 RoboTron-Mani 2.1 Preliminary Multimodal Large Language Models (MLLMs) typically consist of three main components: the modality encoder (Enc), the adapter (Adapter), and the large language model (LLM), mathematically expressed as follows: \displaystyle O_{T} \displaystyle=\text{MLLM}(I,T)