Bridging the Sim2Real Gap: Vision Encoder Pre-Training for Visuomotor Policy Transfer
Samuel Biruduganti Yash Yardi Lars Ankile, {sbiruduganti, yyardi}@imsa.edu [email protected] · 2025
Read Bridging the Sim2Real Gap: Vision Encoder Pre-Training for Visuomotor Policy Transfer by Samuel Biruduganti Yash Yardi Lars Ankile, {sbiruduganti, yyardi}@imsa.edu [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-16389.txt.
Introduction
Yardi, Biruduganti, and Ankile grade vision encoders for moving robot skills from simulation to reality in a 2025 paper. Robots trained in simulators often fail in real rooms because the view changes. A strong pre-trained vision encoder might extract features that survive the jump.
The team scored 23 encoders on two axes: whether their features carry what control needs, and whether they ignore irrelevant scene changes. Encoders pre-trained on manipulation data led on the action score. Convolutional models held up better across domain shifts than transformer models. The best encoders combined both strengths, which the authors offer as a practical selection guide.
Catalog record
| CREATOR | Samuel Biruduganti Yash Yardi Lars Ankile, {sbiruduganti, yyardi}@imsa.edu [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-16389
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Bridging the Sim2Real Gap: Vision Encoder Pre-Training for Visuomotor Policy Transfer Samuel Biruduganti Yash Yardi Lars Ankile, {sbiruduganti, yyardi}@imsa.edu [email protected] Abstract Simulation offers a scalable and efficient alternative to real-world data collection for learning visuomotor robotic policies. However, the simulation-to-reality, or “Sim2Real” distribution shift—introduced by employing simulation-trained policies in real-world environments—frequently prevents successful policy transfer. This study explores the potential of using large-scale pre-training of vision encoders to address the Sim2Real gap. We examine a diverse collection of encoders, evaluating their ability to (1) extract features necessary for robot control while (2) remaining invariant to task-irrelevant environmental variations. We quantitatively measure the encoder’s feature extraction capabilities through linear probing and its domain invariance by computing distances between simulation and real-world embedding centroids. Additional qualitative insights are provided through t-SNE plots and GradCAM saliency maps. Findings suggest that encoders pre-trained on manipulation-specific datasets generally outperform those trained on generic datasets in bridging the Sim2Real gap. https://github.com/yyardi/Bridging-the-Sim2Real-Gap I Introduction Robots that can reliably function in unstructured environments, like a home or construction site, operating from vision alone, have the potential to transform daily life and industry [ 1 ] . Visuomotor robotic policies—algorithms that enable robots to perform tasks solely using image observations are necessary to accomplish this. Recent advancements have been made using Neural Networks (NNs) and machine learning to learn these policies from data [ 2 , 3 , 4 ] . However, gathering the vast data necessary for robots to learn robust policies on physical robots is often impractical and can pose safety risks. Simulation is a scalable alternative that has allowed researchers to replicate years of robot training in a fraction of the time [ 5 ] . Despite the incredible benefits, the simulation will inevitably differ from the real environment. This difference introduces a distribution shift known as the Sim2Real gap, which, if not adequately addressed, results in poorly performing simulation-trained policies when deployed in real settings. A promising way to improve the Sim2Real gap is to pre-train visual representations [ 6 ] . These neural networks, trained on enormous datasets, convert raw images into compact, information-rich representation vectors that improve transferability by preferably extracting task-relevant information. This study assesses various encoders’ potential for transfer learning of visuomotor policies. We hypothesize that effective vision encoders capable of bridging the Sim2Real gap should (1) pick out relevant features from an image to infer actions (e.g., the objects and their position) and (2) ignore task-irrelevant parts of the image that change between simulated and real environments (e.g., the table color and clutter objects). To comprehensively understand the potential for the vision encoders to (1) predict the correct actions based on the image and (2) to neglect domain-specific features that vary between simulated and real environments, we employ various quantitative and qualitative metrics described in detail in Section 3. We find that the type of pre-training data used, either manipulation-specific data or general image data, tends to be the most important factor in determining an encoder’s overall performance. We also found that CNN-based encoders generally outperformed ViT-based encoders in domain invariance. We present our findings as a 2-dimensional plot according to our two criteria. II Related Works Given the significant potential of AI-enabled robots capable of real-world unsupervised operation, many researchers have explored the nature of and potential solutions to the Sim2Real gap. Training machine learning models to help robots learn visuomotor policies traditionally requires enormous datasets that are costly and time-consuming to produce. Thus, it is crucial to solve the problem of increasing the viability of robots with diversely applicable visuomotor policies in the future. II-A Pre-Trained Visual Representations The robotics research community has investigated pre-trained visual representations (PVRs) for training downstream policies in real-world tasks, comparing their performance across simulated and real environments. PVRs produce embeddings (output features of 1-dimensional vectors) extracted from visual data using pre-trained vision encoders designed for visuomotor tasks. Silwal et al. conducted a comprehensive investigation into using PVRs for training policies, evaluating their effectiveness in simulated and real environments [ 7 ] . The study demonstrated that PVRs significantly improved Sim2Real transfer in the ImageNav task compared to previous methods. Burns et al. further studied the success of PVRs in robotic manipulation tasks, focusing on the attributes of pre-trained vision models robust to distribution shifts, including subtle lighting and texture changes [ 8 ] . The study evaluates fifteen models to identify which features made them more reliable under these conditions. The authors found that models pre-trained on manipulation-relevant data could consistently generalize better than models trained on standard pre-training datasets like ImageNet [ 9 ] . Notably, they found that a model’s strong emergent segmentation ability—the capacity to separate essential parts of an image from irrelevant details naturally—showcased its consistent success in generalization despite distribution shifts. Given the significant relevance of these distribution shifts to the Sim2Real gap, this finding applies directly to this inquiry’s application of the PVRs in examining whether the nature of the model’s pre-training indicates its performance. II-B Custom Frameworks for Learning to Generalize To combat the distribution shift, a study introduced Maniwhere, a framework to improve generalization across diverse visual disturbances [ 10 ] . Maniwhere leverages multi-view representation learning with a Spatial Transformer Network to extract meaningful information from varying viewpoints. By combining a fixed image with a randomly perturbed one, the model learns correspondences that enhance its ability to generalize semantic features. Then, reinforcement learning is employed to train the robot through environmental interaction, optimizing its decision-making process. Evaluated on eight manipulation tasks, the unique framework demonstrated superior visual generalization and Sim2Real transfer compared to other designs. This study’s multi-view approach highlights a promising avenue for addressing the Sim2Real gap in future research. II-C Probing to Evaluate Quality of Learning The robotics community continues to explore cost-efficient methods for evaluating learned representations in training visuomotor policies. Linear probing is an affordable alternative to experimental assessments with reinforcement learning (RL) algorithms introduced in [ 11 ] . Researchers applied this approach in the Arcade Learning Environment, testing nine Atari games by comparing actions predicted by the probes to those taken by RL algorithms. The probes estimated task-specific rewards and identified optimal actions, with results showing a strong correlation between probe predictions and RL performance. This method proved to be up to 600 times more computationally efficient than RL-based evaluations. These findings validate this research’s novel probing to predict the action-inference performance of surveyed models without requiring direct experimentation on robots. III Methods We evaluate several pre-trained vision encoders’ ability to learn effective visuomotor policies transferable to real-world applications. We posit that to tackle the Sim2Real gap, effective encoders should exhibit high action-inference accuracy—indicating relevant feature extraction ability—and low source determination ability, or high domain invariance, thus minimizing the Sim2Real gap. Encoder attributes are assessed by generating embeddings, using linear probes and distance metrics for quantitative comparison, and performing qualitative analyses with t-SNE plots and Grad-CAM saliency maps on image samples [ 12 , 13 ] . With matching action and domain labels, the image dataset includes more than 50,000 frames of a robot arm trying to construct a small table—a challenging task—from simulated and real—world settings [ 14 , 15 ] . This large collection offers