Robo CO IN : An Open-Sourced Bimanual Robo tic Data CO llection for IN tegrated Manipulation

Shihan Wu 1 , 2 ∗ † 1,2*{\dagger} Xuecheng Liu 1 ∗ † 1*{\dagger} Shaoxuan Xie 1 ∗ † 1*{\dagger} Pengwei Wang 1 ∗ 1* Xinghang Li 1 ∗ 1* Bowen Yang 3 3 Zhe Li 3 3 Kai Zhu 3 3 Hongyu Wu 1 , 4 1,4 Yiheng Liu 1 , 4 1,4 Zhaoye Long 1 , 5 1,5 Yue Wang 1 1 Chong Liu 1 , 4 1,4 Dihan Wang 1 , 4 1,4 Ziqiang Ni · 2025

Read Robo CO IN : An Open-Sourced Bimanual Robo tic Data CO llection for IN tegrated Manipulation by Shihan Wu 1 , 2 ∗ † 1,2*{\dagger} Xuecheng Liu 1 ∗ † 1*{\dagger} Shaoxuan Xie 1 ∗ † 1*{\dagger} Pengwei Wang 1 ∗ 1* Xinghang Li 1 ∗ 1* Bowen Yang 3 3 Zhe Li 3 3 Kai Zhu 3 3 Hongyu Wu 1 , 4 1,4 Yiheng Liu 1 , 4 1,4 Zhaoye Long 1 , 5 1,5 Yue Wang 1 1 Chong Liu 1 , 4 1,4 Dihan Wang 1 , 4 1,4 Ziqiang Ni (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-17441.txt.

Introduction

Shihan Wu and 83 co-authors introduce RoboCOIN, a large open dataset for teaching robots to work with two arms at once. Bimanual manipulation drives human-like dexterity, but datasets for it are scarce because robot hardware varies so widely between platforms. RoboCOIN gathers more than 180,000 demonstrations from 15 distinct robotic platforms, across home, commercial, and industrial settings, covering 421 tasks organized by collaboration action and object.

A layered annotation pyramid labels the data at several levels, from trajectory concepts down through subtasks to frame-level kinematics. A processing pipeline built around a robot trajectory markup language handles quality checks, machine annotation, and unified handling across embodiments. Experiments show training on RoboCOIN improves bimanual manipulation models across many robot bodies. The dataset and code are fully open-sourced for the field.

Catalog record

CREATORShihan Wu 1 , 2 ∗ † 1,2*{\dagger} Xuecheng Liu 1 ∗ † 1*{\dagger} Shaoxuan Xie 1 ∗ † 1*{\dagger} Pengwei Wang 1 ∗ 1* Xinghang Li 1 ∗ 1* Bowen Yang 3 3 Zhe Li 3 3 Kai Zhu 3 3 Hongyu Wu 1 , 4 1,4 Yiheng Liu 1 , 4 1,4 Zhaoye Long 1 , 5 1,5 Yue Wang 1 1 Chong Liu 1 , 4 1,4 Dihan Wang 1 , 4 1,4 Ziqiang Ni
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-17441

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Robo CO IN : An Open-Sourced Bimanual Robo tic Data CO llection for IN tegrated Manipulation Shihan Wu 1 , 2 ∗ † 1,2*{\dagger} Xuecheng Liu 1 ∗ † 1*{\dagger} Shaoxuan Xie 1 ∗ † 1*{\dagger} Pengwei Wang 1 ∗ 1* Xinghang Li 1 ∗ 1* Bowen Yang 3 3 Zhe Li 3 3 Kai Zhu 3 3 Hongyu Wu 1 , 4 1,4 Yiheng Liu 1 , 4 1,4 Zhaoye Long 1 , 5 1,5 Yue Wang 1 1 Chong Liu 1 , 4 1,4 Dihan Wang 1 , 4 1,4 Ziqiang Ni 1 1 Xiang Yang 1 1 You Liu 1 1 Ruoxuan Feng 1 , 6 1,6 Runtian Xu 1 , 4 1,4 Lei Zhang 1 , 7 1,7 Denghang Huang 1 , 8 1,8 Chenghao Jin 1 , 9 1,9 Anlan Yin 1 , 10 1,10 Xinlong Wang 1 1 Zhenguo Sun 1 1 Junkai Zhao 1 1 Mengfei Du 1 1 Mingyu Cao 1 1 Xiansheng Chen 1 1 Hongyang Cheng 1 1 Xiaojie Zhang 1 1 Yankai Fu 1 , 11 1,11 Ning Chen 1 , 11 1,11 Cheng Chi 1 1 Sixiang Chen 1 , 11 1,11 Huaihai Lyu 1 , 7 1,7 Xiaoshuai Hao 1 1 Yequan Wang 1 1 Bo Lei 1 1 Dong Liu 1 1 Xi Yang 1 1 Yance Jiao 1 1 Tengfei Pan 1 1 Yunyan Zhang 1 1 Songjing Wang 1 1 Ziqian Zhang 1 1 Xu Liu 1 1 Ji Zhang 12 12 Caowei Meng 13 13 Zhizheng Zhang 13 , 1 13,1 Jiyang Gao 14 14 Song Wang 15 15 Xiaokun Leng 15 15 Zhiqiang Xie 16 16 Zhenzhen Zhou 16 16 Peng Huang 17 17 Wu Yang 17 17 Yandong Guo 18 18 Yichao Zhu 18 18 Suibing Zheng 19 19 Hao Cheng 20 20 Xinmin Ding 21 21 Yang Yue 22 22 Huanqian Wang 22 22 Chi Chen 22 22 Jingrui Pang 1 , 22 1,22 YuXi Qian 23 23 Haoran Geng 24 24 Lianli Gao 2 2 Haiyuan Li 4 4 Bin Fang 4 , 1 4,1 Gao Huang 22 , 1 22,1 Yaodong Yang 11 , 1 , 25 11,1,25 Hao Dong 11 , 1 11,1 He Wang 11 , 13 , 1 11,13,1 Hang Zhao 22 , 14 22,14 Yadong Mu 11 , 1 11,1 Di Hu 6 , 1 6,1 Hao Zhao 22 , 1 22,1 Tiejun Huang 11 , 1 11,1 Shanghang Zhang 11 , 1 ‡ 11,1{\ddagger} Yonghua Lin 1 ‡ 1{\ddagger} Zhongyuan Wang 1 ‡ 1{\ddagger} and Guocai Yao 1 † ‡ 1{\dagger}{\ddagger} Abstract Bimanual manipulation is essential for achieving human-like dexterity in robots, but the large-scale and diverse bimanual robot datasets remain scarce due to hardware heterogeneity across robotic platforms. To address the challenge, we present RoboCOIN, a comprehensive multi-embodiment bimanual manipulation dataset with over 180,000 demonstrations collected from 15 distinct robotic platforms. The dataset covers 16 scenarios, including residential, commercial, and working environments, with 421 tasks systematically organized by bimanual coordination patterns and object properties. Our key innovation is a hierarchical capability pyramid that provides multi-level annotations, spanning trajectory-level concepts, segment-level subtasks, and frame-level kinematics. We further develop CoRobot, a comprehensive processing framework featuring Robot Trajectory Markup Language (RTML) for quality assessment, automated annotation generation, and unified multi-embodiment management. Extensive experiments demonstrate the reliability and effectiveness of RoboCOIN in multi-embodiment bimanual learning, with significant performance improvements across various model architectures and robotic platforms. The complete dataset and framework are open-sourced and publicly available for further research purposes. https://FlagOpen.github.io/RoboCOIN/ Fig. 1 : Overview of our RoboCOIN dataset. I Introduction Bimanual manipulation stands as a fundamental capability enabling robots to perform complex, human-like tasks in unstructured real-world environments such as manufacturing [ 27 ] , home assistance [ 15 ] , and logistics [ 11 ] . High-quality bimanual demonstration data is critically important for training data-driven robot learning methods-including imitation learning [ 40 ] and reinforcement learning [ 7 ] -that rely on large-scale, diverse datasets to generalize across tasks and environments. Such data enables models to learn complex coordination strategies, understand physical interactions, and develop robust control policies for bimanual systems. Despite significant progress in data-driven robotics, existing datasets remain limited in diversity and structure, particularly for bimanual manipulation tasks. The heterogeneity across bimanual platforms makes it difficult to collect large-scale, diverse bimanual datasets that are broadly applicable. Moreover, existing datasets primarily supply action trajectories for imitation learning, but omit the structural learning of the manipulation process. Consequently, these limitations constrain the development of models that can generalize and adapt to new tasks or physical environments. As depicted in Figure 1 , we introduce RoboCOIN , a large-scale, multi-embodiment bimanual manipulation dataset containing over 180,000 demonstrations across 421 distinct tasks. These demonstrations were collected from 15 distinct robotic platforms, including dual-arm robots and humanoids with both parallel grippers and dexterous hands. All data were acquired via human teleoperation to ensure high-quality, and multi-view observations and language annotations are provided for each demonstration. A two-dimensional task taxonomy based on action coordination and object flexibility organizes the demonstrations into a comprehensive grid, enabling systematic task design and progressive skill acquisition. To enable effective structural learning across diverse embodiments, RoboCOIN introduces a hierarchical capability pyramid with three structured annotation levels: (a) trajectory-level annotations capture global concepts and task objectives for holistic planning; (b) segment-level annotations decompose tasks into executable subtasks with temporal alignment for structured reasoning; and (c) frame-level annotations provide dense details, including kinematic states and action labels for precise control. This multi-resolution framework supports learning from high-level conceptual understanding to low-level control, facilitating advanced reasoning and generalization in bimanual manipulation. To efficiently support the data construction and deployment infrastructure of RoboCOIN, we developed the CoRobot framework comprising three key components: (a) a Robot Trajectory Markup Language (RTML) checker that validates trajectory properties, including motion smoothness and task-stage consistency, to ensure physical and semantic integrity across platforms; (b) a hierarchical annotation toolchain combining visual language models with rule-based methods to automate scene description and state transition labeling; (c) and a unified platform which extends LeRobot [ 6 ] to provide unified multi-embodiment control, data management, and deployment capabilities, establishing a foundational infrastructure for scalable multi-embodiment learning. This integrated infrastructure ensures consistent data quality across diverse hardware platforms while providing the necessary tools for scalable multi-embodiment learning. To validate the RoboCOIN dataset’s multi-embodiment applicability, we developed a general enhancement method that incorporates the hierarchical capability pyramid into diverse architectures while preserving their original network structures and parameters, enabling high-level conceptual understanding and low-level feedback control. Comprehensive evaluation across multiple models and robotic platforms demonstrates consistent performance improvements in bimanual manipulation tasks spanning RoboCOIN’s multi-dimensional task space. Furthermore, analysis of RTML-validated data reveals inherent limitations in human-teleoperated demonstrations, underscoring the need for unified data standards to advance multi-embodiment learning. The key contributions of this work can be summarized as follows: • Large-Scale, Multi-Embodiment Bimanual Dataset. We introduce RoboCOIN, a comprehensive dataset featuring over 180,000 demonstrations across 421 tasks, collected from 15 distinct robotic platforms. • Hierarchical Capability Pyramid. We propose a hierarchical capability pyramid with trajectory-level, segment-level, and frame-level descriptions, enabling multi-resolution learning from high-level global concepts to low-level control. • Integrated Data Processing Framework. We develop

Read the complete text