Adversarial Data Collection: Human-Collaborative Perturbations for Efficient and Robust Robotic Imitation Learning
Siyuan Huang, Yue Liao, Siyuan Feng, Shu Jiang, Si Liu, Hongsheng Li, Maoqing Yao, Guanghui Ren · 2025
Read Adversarial Data Collection: Human-Collaborative Perturbations for Efficient and Robust Robotic Imitation Learning by Siyuan Huang, Yue Liao, Siyuan Feng, Shu Jiang, Si Liu, Hongsheng Li, Maoqing Yao, Guanghui Ren (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-11646.txt.
Introduction
Siyuan Huang and seven co-authors present a 2025 paper on collecting better training data for robots. Robots learn tasks from human demos, but plain demos are costly and show only smooth success. The team argues that one rich demo beats many dull ones, if it packs in recovery moves.
Their method adds a second human whose job is to disrupt the scene mid-task: shifting objects, changing orders, and stirring conditions. The driver must adapt on the fly, so a single run records failures, fixes, and task twists all at once. The aim is tougher robots trained on far fewer runs.
Catalog record
| CREATOR | Siyuan Huang, Yue Liao, Siyuan Feng, Shu Jiang, Si Liu, Hongsheng Li, Maoqing Yao, Guanghui Ren |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-11646
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Adversarial Data Collection: Human-Collaborative Perturbations for Efficient and Robust Robotic Imitation Learning Siyuan Huang, Yue Liao, Siyuan Feng, Shu Jiang, Si Liu, Hongsheng Li, Maoqing Yao, Guanghui Ren Abstract The pursuit of data efficiency, where quality outweighs quantity, has emerged as a cornerstone in robotic manipulation, especially given the high costs associated with real-world data collection. We propose that maximizing the informational density of individual demonstrations can dramatically reduce reliance on large-scale datasets while improving task performance. To this end, we introduce Adversarial Data Collection (ADC), a Human-in-the-Loop (HiL) framework that redefines robotic data acquisition through real-time, bidirectional human-environment interactions. Unlike conventional pipelines that passively record static demonstrations, ADC adopts a collaborative perturbation paradigm: during a single episode, an adversarial operator dynamically alters object states, environmental conditions, and linguistic commands, while the tele-operator adaptively adjusts actions to overcome these evolving challenges. This process compresses diverse failure-recovery behaviors, compositional task variations, and environmental perturbations into minimal demonstrations. Our experiments demonstrate that ADC-trained models achieve superior compositional generalization to unseen task instructions, enhanced robustness to perceptual perturbations, and emergent error recovery capabilities. Strikingly, models trained with merely 20% of the demonstration volume collected through ADC significantly outperform traditional approaches using full datasets. These advances bridge the gap between data-centric learning paradigms and practical robotic deployment, demonstrating that strategic data acquisition, not merely post-hoc processing, is critical for scalable, real-world robot learning. Additionally, we are curating a large-scale ADC-Robotics dataset comprising real-world manipulation tasks with adversarial perturbations. This benchmark will be open-sourced to facilitate advancements in robotic imitation learning. More information can be found on our project page: https://sites.google.com/view/adc-robot . I Introduction In real-world robotic manipulation, collecting real-scene imitation learning data requires humans to teleoperate robotic arms to perform tasks based on linguistic instructions, which is a labor-intensive and costly process [ 1 , 2 ] . Each demonstration demands significant human effort ( e.g. , expert operation, precise annotation), making it economically infeasible to scale data quantity for diverse environmental coverage. Real-world complexity—varying object configurations, lighting, and task constraints—further exacerbates the need for massive data diversity to ensure robust generalization [ 3 ] . While recent approaches attempt to address this by generating diversity through simulated environments [ 4 , 5 ] or low-cost Universal Manipulation Interface (UMI) grippers [ 3 , 6 ] (sampling one variation per trial, such as object poses), these methods face two fundamental limitations: • Synthetic data, despite its scalability, suffers from a domain gap due to idealized physics/kinematics, hindering transfer to real-robot dynamics; • Extending such diversity to tele-operated systems remains impractical, as even minor environmental changes ( e.g. , repositioning objects) require costly human-in-the-loop resampling. Fig. 1: Comparative Analysis of the Real-Data Collection Loop in Robotic Manipulation. (a) Traditional Approach: A tele-operator executes tasks via fixed linguistic instructions in static visual environments. (b) Adversarial Data Collection (ADC) Framework: Employs a Two-Humans-in-the-Loop approach, where a secondary operator intervenes to perturb the primary’s execution dynamically when the tele-operator is executing a task. (c) ADC Loop: The adversarial operator introduces visual (backgrounds, object positions/poses) and linguistic (task goals) perturbations, shifting environmental context and target objects within a single episode. To contextualize our approach, we first analyze the data requirements and structural constraints of state-of-the-art robotic imitation frameworks, particularly Vision-Language-Action (VLA) models. VLAs integrate Multimodal Large Language Models (MLLMs) to unify reasoning over visual inputs, language instructions, and robotic actions. Most VLA [ 7 , 8 ] frameworks inherit the Markov assumption, predicting actions based solely on the current visual observation. This design necessitates training data in the form of triplets: current visual scene, language instruction, and action supervision. As shown in Fig. 1 (a), Conventional data collection, however, treats an entire task ( e.g. , ”grasping the orange”) as a single demonstration, slicing it into sequential units. These units suffer from inherent inefficiencies—visually redundant frames, repetitive language instructions, and highly correlated actions—effectively diluting the informational value of each collected demonstration. For instance, a 30-second ”pick-and-place” task might yield hundreds of near-identical training triplets, with minimal diversity in scenes or commands. To overcome these limitations, we propose Adversarial Data Collection (ADC), a collaborative Human-in-the-Loop (HiL) framework that systematically injects controlled perturbations during data acquisition to maximize per-demonstration diversity and robustness. As illustrated in Fig. 1 , the ADC framework operates through two synchronized human roles: a tele-operator executes primary manipulation tasks via standard robotic interfaces, while an adversarial operator dynamically modulates task parameters through real-time perturbations across perceptual, linguistic, and physical dimensions. During a single demonstration episode, the adversarial operator alters scene configurations ( e.g. , repositioning objects, rotating target objects mid-grasp and adjusting background) and paraphrases task instructions ( e.g. , ”grasping the orange” → ”grasping the watermelon”). These perturbations force the tele-operator to adaptively re-plan actions, such as re-grasping strategies or trajectory corrections, under evolving constraints, thereby compressing recovery behaviors, instruction compositional generalization, and environmental variations into a unified demonstration. By systematically coupling human-driven adversity with real-time adaptation, ADC not only elevates demonstration efficiency but also cultivates model robustness against perceptual ambiguities, linguistic variability, and physical uncertainties, which is critical for deploying VLAs in unstructured real-world environments To evaluate the effectiveness of ADC , we constructed a dataset consisting of 200 demonstrations including balanced adversarially perturbed demonstrations ( 96 K frames) from ADC and demonstrations ( 90 K frames). Each demonstration from ADC incorporates multiple layers of real-time perturbations, spanning visual and linguistic dimensions, introduced dynamically during human tele-operation. While marginally more time-intensive per demonstration, ADC-trained models achieve superior generalization and robustness with significantly fewer demonstrations—outperforming models trained on larger-scale conventional datasets in handling ambiguous instructions, environmental perturbations, and unseen object configurations. This efficiency stems from ADC ’s capacity to encode real-world complexity through intentional adversity, transforming data quality into a substitute for quantity—a critical advancement for scalable robotic learning. Additionally, we are curating a large-scale ADC-Robotics dataset comprising real-world manipulation tasks with adversarial perturbations. Upon completion, this benchmark will be open-sourced to facilitate advancements in robotic imitation learning. II Related Work Fig. 2: The overview of ADC . During training data collection, we introduce several adversarial perturbations, including dynamic visual perturbations and adaptive linguistic challenges. These perturbations increase information density, expand state space coverage, and provide more complete observations of target objects. The resulting high-quality dataset enables the trained policy model to achieve strong robustness and generalization, outperforming models trained with conventional data collection strategies. II-A Data Collection in Robot Learning Data collection methods in robot learning have been extensively studied to improve generalization, generally with a focus on scaling up robot datasets [ 9 , 10 , 2 , 1 ] . These efforts have shown that greater object and task diversity can significantly improve policy robustness, highlighting the importance of collecting diverse datasets. For instance, Droid [ 2 ] , one of the largest existing datasets, contains 76k demonstration trajectories collected across more than 500 scenes. More recently, studies [ 11 , 3 ] have explored strategies for efficient data collection to enhance policy generalization; however, their discussions are largely constrained to compositional generalization with respect to visual factors. In this paper, we extend the scope of data collection by considering both visual and language dimensions, aiming to improve generalization across a broader range