DexSim2Real: Foundation Model-Guided Sim-to-Real Transfer for Generalizable Dexterous Manipulation
Zijian Zeng, Fei Ding, Huiming Yang, Xianwei Li, Yuhao Liao · 2026
Read DexSim2Real: Foundation Model-Guided Sim-to-Real Transfer for Generalizable Dexterous Manipulation by Zijian Zeng, Fei Ding, Huiming Yang, Xianwei Li, Yuhao Liao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-05241.txt.
Introduction
Zijian Zeng and four co-authors present DexSim2Real, a framework that moves dexterous robot hand skills from simulation into the real world with help from vision-language foundation models. Sim-to-real transfer usually leans on hand-designed randomisation or task-specific tuning, which limits how widely it generalises. Their system instead lets a foundation model act as a visual realism critic for the simulation itself.
Three parts combine: guided domain randomisation tuned in a closed loop, a tactile-visual attention policy built for zero-shot transfer, and a progressive skill curriculum based on language-model task breakdowns. Across six hard manipulation tasks with blinded scoring, the system reached a high average real-world success rate and a small simulation-to-reality gap.
Catalog record
| CREATOR | Zijian Zeng, Fei Ding, Huiming Yang, Xianwei Li, Yuhao Liao |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.05241
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DexSim2Real: Foundation Model-Guided Sim-to-Real Transfer for Generalizable Dexterous Manipulation Zijian Zeng, Fei Ding, Huiming Yang, Xianwei Li, Yuhao Liao Abstract Sim-to-real transfer remains a critical bottleneck for deploying dexterous manipulation policies learned in simulation to real-world robots. Existing approaches rely on manually designed domain randomization or task-specific adaptation, limiting their generalizability across diverse manipulation scenarios. We present DexSim2Real , an integrated framework that leverages vision-language foundation models to bridge the sim-to-real gap for dexterous manipulation. Our system combines three components: (1) Foundation Model-Guided Domain Randomization (FM-DR), which uses a vision-language model as a visual realism critic to optimize simulation parameters via closed-loop CMA-ES, complementing text-based approaches like DrEureka with direct visual feedback; (2) a Tactile-Visual Cross-Attention Policy (TVCAP) that adapts cross-attention visuo-tactile fusion to zero-shot sim-to-real RL; and (3) a Progressive Skill Curriculum (PSC) that builds on LLM-based task decomposition with a difficulty scheduler tailored to contact-rich dexterous tasks. Extensive experiments on six challenging manipulation tasks with blinded evaluation demonstrate that DexSim2Real achieves a 78.2% average real-world success rate, outperforming DrEureka and DeXtreme while reducing the sim-to-real performance gap to only 8.3%. 1 Introduction Dexterous robotic manipulation—the ability to grasp, reorient, and precisely control objects using multi-fingered hands—is a long-standing challenge in robotics [ 18 ] . While reinforcement learning (RL) has shown remarkable success in acquiring complex manipulation skills in simulation [ 16 , 9 ] , transferring these policies to physical robots remains notoriously difficult due to the sim-to-real gap : discrepancies in visual appearance, physics dynamics, and sensor characteristics between simulated and real environments. Domain randomization (DR) [ 25 ] and its variants such as active domain randomization (ADR) [ 17 ] have been the dominant paradigm for sim-to-real transfer. These methods randomize simulation parameters during training to produce policies robust to domain shift. However, they suffer from two fundamental limitations: (1) the randomization ranges are typically set manually or tuned through expensive trial-and-error, and (2) uniform randomization over a large parameter space leads to training on many unrealistic configurations, degrading policy performance. Meanwhile, the emergence of vision-language foundation models (VLMs) such as CLIP [ 21 ] and large vision-language models has opened new possibilities for robot learning [ 2 , 15 ] . These models encode rich semantic and visual priors about the physical world. Recent works such as DrEureka [ 14 ] and Eureka [ 13 ] have demonstrated that large language models can automate reward design and domain randomization generation for sim-to-real transfer. Similarly, LLM-based curriculum design [ 22 , 5 ] and cross-modal visuo-tactile fusion [ 10 , 11 ] have shown promise individually. However, no existing work integrates foundation-model-guided domain randomization, multi-modal policy learning, and automatic curriculum generation into a unified sim-to-real pipeline for dexterous manipulation. In this paper, we present DexSim2Real , an integrated system that bridges the sim-to-real gap for dexterous manipulation by leveraging foundation models at multiple stages of the learning pipeline. Our key insight is that VLMs can serve as visual realism critics : given rendered simulation images and real-world reference images, a VLM can assess visual fidelity and guide parameter optimization via closed-loop CMA-ES, complementing text-based approaches like DrEureka [ 14 ] with direct visual feedback. Our contributions are as follows: • We present DexSim2Real , an integrated sim-to-real system that combines foundation-model-guided domain randomization, multi-modal policy learning, and automatic curriculum generation for zero-shot dexterous manipulation transfer. • We propose Foundation Model-Guided Domain Randomization (FM-DR) , which uses a VLM as a visual realism critic with closed-loop CMA-ES optimization. Unlike DrEureka’s text-based LLM approach, FM-DR directly evaluates rendered images against real references, providing complementary visual feedback for parameter optimization. • We introduce a Tactile-Visual Cross-Attention Policy (TVCAP) that adapts cross-attention visuo-tactile fusion [ 10 , 11 ] to the zero-shot sim-to-real RL setting without requiring real-world demonstrations. • We design a Progressive Skill Curriculum (PSC) that builds on LLM-based curriculum ideas [ 22 , 5 ] with a \delta -based difficulty scheduler and success-rate-thresholded skill chaining tailored to contact-rich dexterous tasks. • We conduct extensive real-world experiments on six dexterous manipulation tasks with blinded evaluation, demonstrating state-of-the-art sim-to-real transfer performance with a 78.2% average success rate and only 8.3% sim-to-real gap, outperforming DrEureka and DeXtreme on shared tasks. 2 Related Work Sim-to-Real Transfer for Manipulation. Domain randomization [ 25 ] trains policies across randomized simulation parameters to achieve robustness to the reality gap. Peng et al. [ 19 ] extended this to dynamics randomization for locomotion and manipulation. Active domain randomization [ 17 ] learns to select informative randomization parameters but still operates within predefined ranges. DeXtreme [ 9 ] demonstrated impressive sim-to-real transfer for in-hand manipulation using massive-scale DR in Isaac Gym, but required extensive manual tuning of randomization distributions. DexPoint [ 20 ] proposed point cloud-based RL for generalizable dexterous manipulation. More recently, Eureka [ 13 ] showed that LLMs can generate human-level reward functions, and DrEureka [ 14 ] extended this to LLM-guided automatic generation of domain randomization distributions for sim-to-real transfer. GenSim [ 28 ] uses LLMs to generate simulation tasks. Our FM-DR approach is complementary to DrEureka: while DrEureka uses text-based LLM reasoning to propose DR distributions, FM-DR uses a VLM as a visual realism critic with closed-loop CMA-ES optimization over rendered images. Foundation Models for Robotics. Recent works have explored leveraging foundation models for robot learning [ 15 ] . RT-2 [ 2 ] directly fine-tunes VLMs for robotic control, while \pi_{0} [ 1 ] proposes vision-language-action flow models for general robot control. Diffusion Policy [ 4 ] and 3D Diffusion Policy [ 30 ] use diffusion models for visuomotor policy learning. Unlike these approaches that use foundation models as policy backbones, we leverage VLMs as simulation critics to optimize the training environment itself, which is complementary to policy architecture choices. Multi-Modal Policy Learning. Combining visual and tactile sensing for manipulation has gained increasing attention [ 12 ] . PerAct [ 24 ] and RVT [ 8 ] use transformer architectures for multi-task manipulation from visual observations. Act3D [ 7 ] introduces 3D feature fields for manipulation. DexCap [ 27 ] collects multi-modal demonstration data for dexterous manipulation. Cross-attention for visuo-tactile fusion has been explored in Visuo-Tactile Transformers [ 3 ] , 3D-ViTac [ 10 ] , and ViTacFormer [ 11 ] , which applies cross-attention at every stage on a dexterous bimanual platform. Our TVCAP builds on this line of work and adapts bidirectional cross-attention with residual connections specifically for the zero-shot sim-to-real RL setting, where no real demonstrations are available. Curriculum Learning for RL. Curriculum strategies have been shown to improve RL training efficiency [ 18 ] . RAPS [ 6 ] uses parameterized action primitives to structure the action space. DayDreamer [ 29 ] learns world models for efficient robot learning. CurricuLLM [ 22 ] uses LLMs to automatically design task curricula for learning complex robot skills. Plan-Seq-Learn [ 5 ] leverages language models for task decomposition in long-horizon manipulation. Our PSC builds on these ideas, contributing a \delta -based difficulty scheduler and success-rate-thresholded skill chaining specifically designed for contact-rich dexterous manipulation tasks. 3 Method We present DexSim2Real, a three-stage framework for sim-to-real dexterous manipulation (Fig. 1 ). Given a target manipulation task specified in natural language, our