A Study on Enhancing the Generalization Ability of Visuomotor Policies via Data Augmentation
Hanwen Wang · 2025
Read A Study on Enhancing the Generalization Ability of Visuomotor Policies via Data Augmentation by Hanwen Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-09932.txt.
Introduction
Hanwen Wang, writing alone, studies how data augmentation can make robot visuomotor policies generalize to new scenes. Policies trained by imitation often stumble when object placement and scene conditions change, and existing data-adding pipelines generate data that still lacks diversity. Wang automates data generation around the scene layout factors that most affect how broadly policies work, building a heavily randomized data set from little human demonstration.
The data set spans five manipulator types, two grippers, and six manipulation tasks, randomizing camera pose, lighting, tabletop texture, and table height. Every factor influenced generalization, and any randomization helped, with diverse trajectories best at closing the visual gap. Tests on a low-cost manipulator tested how the randomization supports zero-shot transfer from simulation to the real robot.
Catalog record
| CREATOR | Hanwen Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-09932
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Study on Enhancing the Generalization Ability of Visuomotor Policies via Data Augmentation Hanwen Wang Abstract The generalization ability of visuomotor policy is crucial, as a good policy should be deployable across diverse scenarios. Some methods can collect large amounts of trajectory augmentation data to train more generalizable imitation learning policies, aimed at handling the random placement of objects on the scene’s horizontal plane. However, the data generated by these methods still lack diversity, which limits the generalization ability of the trained policy. To address this, we investigate the performance of policies trained by existing methods across different scene layout factors via automate the data generation for those factors that significantly impact generalization. We have created a more extensively randomized dataset that can be efficiently and automatically generated with only a small amount of human demonstration. The dataset covers five types of manipulators and two types of grippers, incorporating extensive randomization factors such as camera pose, lighting conditions, tabletop texture, and table height across six manipulation tasks. We found that all of these factors influence the generalization ability of the policy. Applying any form of randomization enhances policy generalization, with diverse trajectories particularly effective in bridging visual gap. Notably, we investigated on low-cost manipulator the effect of the scene randomization proposed in this work on enhancing the generalization capability of visuomotor policies for zero-shot sim-to-real transfer. I INTRODUCTION The generalization ability of visuomotor policies is crucial for robotic manipulation, as it determines whether the model can be effectively deployed. A good policy should be able to handle variations in object layouts, scene changes, and even enable cross-embodiment deployment. Existing approaches enhance the generalization of policies by collecting diverse datasets. Imitation learning methods based on human demonstrations have been proven effective in robotic unstructured manipulation tasks. Typically, human expert demonstration data is collected, which includes action signals generated by human teleoperation [ 3 , 5 , 12 , 13 , 16 , 22 , 26 , 38 , 42 ] . robot observation information such as RGB images from different viewpoints, and the robot’s state (usually represented by the end-effector pose). Some studies have also collected task-related language instructions, enabling the policy to generalize object categories using pre-trained encoders from Vision-Language Models (VLM). Generally, imitation learning policies take visual images and the robot’s state as inputs and output the action the robot will perform for closed-loop position control [ 9 , 35 , 2 , 6 , 5 , 24 , 23 , 27 , 43 , 41 ] . The cost in terms of both human effort and time for data collection is high; therefore, some methods attempt to leverage a small number of human demonstrations to generate large amounts of data for training more generalizable policies across scenes. These methods increase the diversity of trajectory data, enabling position generalization for objects on a desktop, and have achieved notable performance in tasks such as pick-and-place and simple assembly [ 28 , 8 , 33 , 34 , 19 ] . Fig. 1: Previous work has primarily focused on trajectory augmentation methods. However, to obtain a generalizable visuomotor policy, it is necessary to address other factors such as visual discrepancies. As the diversity of the scene increases, such as variations in camera poses, lighting colors, tabletop textures, table heights and cross-embodiment using data augmented by a single method becomes insufficient. However, prior work has primarily focused on automating the generation of diverse trajectories without accounting for visual discrepancies [ 28 , 34 , 19 ] . For instance, methods like MimicGen [ 28 ] record the relative positions between the robot’s end effector and the object to be manipulated based on a small number of human demonstrations. Even when the object’s placement pose on a horizontal tabletop changes arbitrarily, the robot can still perform the task by continuously tracking both the object and the end effector poses. Nonetheless, relying on this single-type data augmentation proves to be quite limited in enhancing the generalization capability of the policy. Some studies have explored the policy’s performance under extensive domain randomization, aiming to bridge the visual gap and potentially facilitate sim-to-real policy deployment. Yet, these studies have not systematically assessed the impact of various domain randomization factors, and the conditions they investigate lack sufficient diversity. Moreover, cross-embodiment deployment using a single policy trained with such data has not been sufficiently explored. Therefore, we systematically studied the impact of different domain randomization factors on the generalization ability of the policy. We focused on factors such as random camera poses, random lighting colors, random tabletop textures, and random table heights, and identified these as key factors influencing the policy’s generalization capability. The data generated by existing methods does not cover these randomization factors. In addition, we explore how cross-embodiment data contribute to improving the generalization of visuomotor policies. For lighting and tabletop textures, we randomly sample between them. For camera poses, we ensured that each configuration sampled the same number of poses to maintain a balanced distribution. For cross-embodiment data, we used five manipulators with similar workspaces. To evaluate whether these scene randomization factors facilitate zero-shot sim-to-real deployment in real-world manipulation, we conducted experiments using a low-cost SO-101 manipulator. The results show that policies trained with scene randomization achieve stronger performance when objects are placed in more diverse configurations, effectively reducing the sim-to-real gap. Our contribution can be summarized as follows: (1) The limitations of different randomization factors on the generalization ability of the policy trained by previous methods were systematically studied. (2) We systematically investigate whether different randomization augmentation factors can mutually enhance each other to improve the generalization performance of the policy. (3) We conducted real-world experiments, confirming that policies trained with randomization augmentation facilitate zero-shot sim-to-real transfer. II RELATED WORK II-A Manipulation data collection In the early stages, some works focused on collecting large-scale data for robotic learning. These efforts concentrated on self-supervised learning through trial and error to acquire robot grasping data [ 25 , 20 , 7 , 36 , 21 ] . Later, some works, such as GraspNet, did not collect robot trajectories but instead used analytical methods to label grasp poses directly [ 11 , 40 ] . The goal of robotic learning is not only to perform grasps but also to manipulate objects. Some related studies have collected large-scale operation datasets over extended periods of time using a large number of human operators [ 30 , 18 , 1 , 10 ] , but these methods still consume a significant amount of human and material resources. Some works on automatic data generation aim to address the problem of low-cost collection of large-scale trajectory data [ 28 , 19 , 34 , 33 ] . For example, MimicGen [ 28 ] efficiently generates a large number of trajectory data with only 10 human demonstration samples for a single task, which is essential for policy training. However, these automatic data generation methods focus solely on the diversity of trajectories, which still limits the generalization capability of imitation learning policies. The method we propose aims to expand the diversity of the data by adding variations such as different camera poses, lighting colors, and textures to address the lack of diversity in