Towards Human-Like Manipulation through RL-Augmented Teleoperation and Mixture-of-Dexterous-Experts VLA
Tutian Tang, Xingyu Ji, Wanli XING, Ce Hao, Wenqiang Xu, Lin Shao, Cewu Lu, Qiaojun Yu, Jiangmiao Pang, Kaifeng Zhang · 2026
Read Towards Human-Like Manipulation through RL-Augmented Teleoperation and Mixture-of-Dexterous-Experts VLA by Tutian Tang, Xingyu Ji, Wanli XING, Ce Hao, Wenqiang Xu, Lin Shao, Cewu Lu, Qiaojun Yu, Jiangmiao Pang, Kaifeng Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-08122.txt.
Introduction
Tang and colleagues push Vision-Language-Action models toward human-like, bimanual dexterous manipulation. Current VLA models excel at simple pick-and-place tasks but struggle with contact-rich in-hand operations, which demand high-fidelity data, multi-skill learning, and force and touch signals. Their framework adds IMCopilot, reinforcement-trained atomic skills that assist teleoperation and serve as low-level tools. It also adds MoDE-VLA, which feeds force and touch signals into a pretrained VLA backbone without eroding prior knowledge.
Catalog record
| CREATOR | Tutian Tang, Xingyu Ji, Wanli XING, Ce Hao, Wenqiang Xu, Lin Shao, Cewu Lu, Qiaojun Yu, Jiangmiao Pang, Kaifeng Zhang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36915 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.08122
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Towards Human-Like Manipulation through RL-Augmented Teleoperation and Mixture-of-Dexterous-Experts VLA Tutian Tang, Xingyu Ji, Wanli XING, Ce Hao, Wenqiang Xu, Lin Shao, Cewu Lu, Qiaojun Yu, Jiangmiao Pang, Kaifeng Zhang Abstract While Vision-Language-Action (VLA) models have demonstrated remarkable success in robotic manipulation, their application has largely been confined to low-degree-of-freedom end-effectors performing simple, vision-guided pick-and-place tasks. Extending these models to human-like, bimanual dexterous manipulation—specifically contact-rich in-hand operations—introduces critical challenges in high-fidelity data acquisition, multi-skill learning, and multimodal sensory fusion. In this paper, we propose an integrated framework to address these bottlenecks, built upon two components. First, we introduce IMCopilot (In-hand Manipulation Copilot), a suite of reinforcement learning-trained atomic skills that plays a dual role: it acts as a shared-autonomy assistant to simplify teleoperation data collection, and it serves as a callable low-level execution primitive for the VLA. Second, we present MoDE-VLA (Mixture-of-Dexterous-Experts VLA), an architecture that seamlessly integrates heterogeneous force and tactile modalities into a pretrained VLA backbone. By utilizing a residual injection mechanism, MoDE-VLA enables contact-aware refinement without degrading the model’s pretrained knowledge. We validate our approach on four tasks of escalating complexity, demonstrating doubled success rate improvement over the baseline in dexterous contact-rich tasks. I Introduction Dexterous manipulation in humans arises from the seamless integration of hierarchical decision-making, rich tactile feedback, and well-practiced manipulation skills [ 1 , 2 ] . For example, when peeling an apple, humans coordinate both hands while using vision to guide the peeler, fingertip tactile sensing to detect slippage and regulate grip force, and in-hand manipulation skills to rotate the apple between successive cuts. Such capabilities highlight the importance of combining multimodal sensing with hierarchical manipulation skills for executing contact-rich interactions. Replicating these abilities in robots requires not only high-degree-of-freedom (DoF) hardware, but also learning frameworks that acquire hierarchical manipulation skills grounded in multimodal sensory feedback. To enable robots to acquire such capabilities, recent work has explored Vision-Language-Action (VLA) models [ 3 , 4 , 5 ] , which leverage large-scale vision and language pretraining to learn general-purpose manipulation policies. However, the success of existing VLAs has largely been confined to low-DoF end-effectors such as parallel-jaw grippers [ 3 , 6 ] , where manipulation is typically reduced to simple pick-and-place primitives that rely primarily on vision rather than force or tactile sensing. This paradigm breaks down when manipulation transitions from grasping to in-hand manipulation , where fingers must continuously regulate contact states to reposition, rotate, or manipulate objects and tools. Extending VLA models to human-like bimanual dexterous manipulation introduces three key challenges. First, a severe data acquisition bottleneck : teleoperating a bimanual system with 63 DoFs is highly challenging, and collecting high-quality demonstrations for precise multi-finger coordination in in-hand manipulation often exceeds the capabilities of even expert operators [ 7 , 8 , 9 , 10 ] . Second, the multi-skill learning challenge : tasks such as apple peeling span vision-guided gross motions (approaching and aligning), force-guided fine motions (cutting), and tactile-guided in-hand routines (rotating the apple). These distinct phases require different skills, making them difficult for a single policy to master within a high-dimensional action space [ 11 , 12 ] . Third, the modality heterogeneity challenge : directly concatenating force and tactile observations into a pretrained VLA backbone fails to account for their different temporal dynamics and physical semantics, often degrading rather than improving performance [ 13 , 14 , 15 ] . To address these challenges, we propose an integrated framework for human-like bimanual dexterous manipulation built on two synergistic components. First, IMCopilot ( I n-hand M anipulation Copilot ) is a set of RL-trained atomic in-hand manipulation skills that serves a dual role . During data collection, it functions as a shared-autonomy copilot [ 16 ] : the operator performs gross motions via exoskeleton teleoperation while delegating challenging in-hand manipulation phases to IMCopilot through foot pedals, enabling efficient acquisition of high-fidelity demonstrations. During autonomous execution, IMCopilot acts as a callable low-level skill invoked by the VLA for in-hand manipulation, forming a hierarchical architecture analogous to human motor control. Second, MoDE-VLA , a M ixture- o f- D exterous- E xperts VLA, addresses modality heterogeneity by introducing a dedicated pathway for force and tactile inputs. Force–tactile tokens interact with backbone representations through self-attention, are refined via sparse MoE routing [ 17 ] for per-timestep expert specialization, and are injected as residual corrections, enabling contact-aware refinement while preserving pretrained knowledge. Our contributions are summarized as follows: 1. IMCopilot : an RL-trained in-hand manipulation primitive suite that unifies teleoperation assistance and autonomous low-level control, yielding a hierarchical framework in which the VLA handles vision-language-guided planning while IMCopilot provides reactive in-hand dexterity. 2. MoDE-VLA : a Mixture-of-Dexterous-Experts VLA that fuses force and tactile modalities into a pretrained VLA backbone through dedicated self-attention, sparse expert routing, and residual injection, enabling contact-aware action generation without degrading the backbone’s pretrained capabilities. 3. Experimental validation : we evaluate MoDE-VLA on four tasks of escalating contact complexity (gear assembling, charger plugging, test tube rearranging, and apple peeling) and demonstrate significant improvements over baselines. To our knowledge, we present the first autonomous dual-dexterous-hand apple peeling, a task requiring the full synergy of all proposed components. II Related Work II-A Teleoperation System for Dexterous Manipulation Teleoperation is fundamental to data-driven dexterous manipulation, providing demonstrations for learning high-DoF, contact-rich in-hand manipulation skills, and existing data-collection pipelines can be broadly categorized into vision-based , glove-based , and exoskeleton-based approaches. Vision-based teleoperation leverages cameras to estimate human hand pose and retarget it to robotic hands [ 18 , 7 , 19 ] . While lightweight and scalable, such systems suffer from occlusion and depth ambiguity, and rely primarily on kinematic retargeting without explicit force or contact awareness, limiting their effectiveness in contact-rich manipulation. Glove-based teleoperation employs data gloves with bend sensors or inertial units to directly measure finger joint configurations, improving retargeting accuracy and temporal consistency over vision-based methods [ 9 , 8 , 20 , 21 , 22 ] . However, despite enabling scalable and high-fidelity kinematic capture, the lack of force feedback and human–robot morphological mismatch still limit their applications in contact-rich manipulation. Exoskeleton-based teleoperation interfaces mechanically couple human and robot joints, reducing retargeting error and enabling bidirectional force feedback [ 10 , 23 , 24 ] . While this tight embodiment favors contact-rich manipulation, such systems are complex and cognitively demanding, and precise multi-finger in-hand coordination often exceeds human teleoperation capability, leading to inconsistent demonstrations. In summary, existing pipelines leave the most challenging contact-rich in-hand manipulation to human operators, motivating skill-level shared autonomy via IMCopilot for improved performance and adaptability. II-B Imitation learning with Force/Tactile Observation Prior imitation learning for robotic manipulation relies primarily on vision and proprioception, learning policies that map observed states to actions without direct force or tactile feedback for contact perception. To address this gap, recent work integrates force or tactile observation into imitation learning frameworks for contact-rich manipulation tasks [ 25 , 26 , 13 , 27 , 28 , 29 , 30 ] .Among these, RDP [ 30 ] incorporates force or tactile signals as conditional inputs within a fast–slow architecture to refine actions, demonstrating strong performance in contact-rich manipulation. TA-VLA [ 26 ] investigates torque-aware Vision-Language-Action models, exploring architectural mechanisms for integrating torque