DexHiL: A Human-in-the-Loop Framework for Vision-Language-Action Model Post-Training in Dexterous Manipulation
Yifan Han, Zhongxi Chen, Yuxuan Zhao, Congsheng Xu, Yanming Shao, Yichuan Peng, Yao Mu, Wenzhao Lian · 2026
Read DexHiL: A Human-in-the-Loop Framework for Vision-Language-Action Model Post-Training in Dexterous Manipulation by Yifan Han, Zhongxi Chen, Yuxuan Zhao, Congsheng Xu, Yanming Shao, Yichuan Peng, Yao Mu, Wenzhao Lian (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-09121.txt.
Introduction
Yifan Han and colleagues present DexHiL, a human-in-the-loop framework for post-training vision-language-action models on dexterous robot tasks. Multi-finger control is high-dimensional and contact-heavy, so past dexterous systems stayed unreliable when moved to new jobs. DexHiL lets a person correct both arm and hand within one system through a light teleoperation interface. Its sampling favors corrective segments, and real-robot tests beat offline-only fine-tuning by a wide margin in success rates.
Catalog record
| CREATOR | Yifan Han, Zhongxi Chen, Yuxuan Zhao, Congsheng Xu, Yanming Shao, Yichuan Peng, Yao Mu, Wenzhao Lian |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33799 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.09121
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DexHiL: A Human-in-the-Loop Framework for Vision-Language-Action Model Post-Training in Dexterous Manipulation Yifan Han, Zhongxi Chen, Yuxuan Zhao, Congsheng Xu, Yanming Shao, Yichuan Peng, Yao Mu, Wenzhao Lian Abstract While Vision-Language-Action (VLA) model has demonstrated promising generalization capabilities in robotic manipulation, deploying them on specific and complex downstream tasks still demands effective post-training. In parallel, Human-in-the-Loop (HiL) learning has proven to be a powerful mechanism for refining robot policies. However, extending this paradigm to dexterous manipulation remains challenging: multi-finger control is high-dimensional, contact-intensive, and exhibits execution distributions that differ markedly from arm motion, leaving existing dexterous VLA systems limited in reliability and adaptability. We present DexHiL, the first integrated arm–hand human-in-the-loop framework for dexterous manipulation VLA , enabling coordinated interventions over the arm and the dexterous hand within a single system. DexHiL introduces an intervention-aware data sampling strategy that prioritizes corrective segments for post-training, together with a lightweight teleoperation interface that supports instant human corrections during execution. Real-robot experiments demonstrate that DexHiL serves as an effective post-training framework, yielding a substantial performance leap that outperforming standard offline-only finetuning baselines by an average of 25% in success rates across distinct tasks. Project page: https://chenzhongxi-sjtu.github.io/dexhil/ I Introduction Vision-Language-Action (VLA) models represent a paradigm shift in robot learning, enabling cross-scenario semantic understanding and spatial perception by directly mapping multimodal inputs to control outputs [ 1 , 2 , 3 , 4 , 5 ] . While VLA models have demonstrated significant potential in general manipulation, their application to high-degree-of-freedom (DOF) dexterous hands remains challenging, particularly in the post-training and downstream adaptation phases. Existing VLA post-training strategies, which typically rely on Supervised Fine-Tuning (SFT) over offline datasets [ 3 , 6 , 7 ] , struggle to bridge the gap between high-dimensional end-effector control and the intricate, contact-rich requirements of multi-fingered manipulation. The primary bottleneck in adapting VLA models to dexterous tasks partially stems from hardware-level kinematic misalignment. Traditional teleoperation interfaces, such as exoskeletons and leader-follower arms [ 8 , 9 , 10 ] , often fail to precisely map human hand motions to the complex joint configurations of robotic hands. This misalignment, coupled with the sharp discontinuities inherent in dexterous contact states, precludes the collection of high-quality, fine-grained demonstration data. Consequently, existing VLA paradigms lack the high-fidelity guidance necessary to optimize the nuanced motion manifold required for robust dexterous performance. Fig. 1 : While scaling offline data for VLA models yields slow accuracy gains and performance plateaus, DexHiL integrates offline training with online Human-in-the-Loop interventions. By strategically reweighting offline and corrective online data, our approach achieves high data efficiency and rapid accuracy growth. Fig. 2 : The DexHiL Framework. Below we will introduce our overall framework of DexHiL from data acquisition system, human-in-the-loop intervention paradigm, dexterous manipulation VLA model structure to offline-to-online training process. ( a ) We propose an arm-hand data collection system both supporting teleoperation offline data collection and online human-in-the-loop policy intervention data collection. We also propose a two-stage training method for precise hand joint retargeting. ( b ) We propose an asynchronous human-in-the-loop policy intervention mechanism for online data collection, here we examplify the “Plush Toy Grasping” case. ( c ) Our dexterous manipulation VLA policy follows Being-H0.5 [ 11 ] structure, which utilizing MoT (Mixture of Transformer) to relate understanding model with action expert for multi-modal reasoning and action generation and inherit the open-source pretrained weights of it. ( d ) We propose a two-phase training framework that ultilize both offline dataset and online dataset. In the first warm-up phase, we finetuned the pretrained weights into warm-up model. In the DAgger loop, we utilize the system above to acquire online dataset and use reweighting training to update the policy which will be applied in the next DAgger loop. Beyond hardware limitations, the algorithmic post-training process for dexterous VLAs faces three core systemic challenges: (i) Convergence difficulties in high-dimensional action spaces: The expansive action manifold of dexterous hands, compounded by complex contact dynamics, makes stable policy convergence exceptionally difficult; (ii) Sample efficiency bottlenecks: Offline datasets are frequently dominated by repetitive success data, forcing the model to expend gradients on consolidating learned behaviors rather than exploring the critical transitions necessary for complex tasks; (iii) Covariate shift and error accumulation: During real-robot execution, open-loop VLA policies suffer from trajectory drift. Without an effective recovery mechanism, minor errors quickly push the system into Out-of-Distribution (OOD) states, leading to failure [ 12 ] . These challenges collectively suggest that relying solely on conventional offline post-training is insufficient for robust dexterous manipulation. To bridge this gap, we propose DexHiL, an interactive online training framework that integrates dexterous robotic hands with Human-in-the-Loop (HiL) intervention [ 13 , 14 ] . DexHiL leverages a human-robot collaboration mechanism to facilitate precise manipulation and rapid error recovery in complex, contact-rich tasks. By incorporating real-time expert interventions and an intervention-aware weighting mechanism, DexHiL significantly enhances sample efficiency and policy convergence, ensuring robust performance in high-DOF real-world applications. Our experimental results demonstrate that for a VLA model pre-trained on large-scale human videos with a cold-start action head, jointly fine-tuning on multiple downstream tasks using the DexHiL framework yields significant performance gains. Through three iterative stages of online refinement, DexHiL achieves 20% and 30% improvements in success rates across two distinct tasks, respectively, compared to a baseline model trained offline with an equivalent volume of data. Ablation studies further validate the necessity of our core components, demonstrating that the intervention-aware weighting mechanism is the primary driver for overcoming sample efficiency bottlenecks in high-DOF manipulation. Our contributions are summarized as follows: • Human-to-Robot Hand Movement Retargeting : We introduce a novel learning-based approach for precise retargeting of human hand movements to dexterous robotic hands. This method effectively maps human hand gestures to the robotic manipulator with high fidelity, ensuring accurate control over complex dexterous hand movements. Unlike traditional optimization methods, our approach provides a more adaptive, real-time mapping, improving the performance of dexterous manipulation tasks. • Integrated and HiL-Enabled Teleoperation System : We propose a seamless human-in-the-loop Teleoperation System that addresses the challenge of intervention discontinuity in high-DOF dexterous hand manipulation. By enabling smooth real-time interventions during post-training, our framework ensures effective and high-quality error correction. • Iterative Human-in-the-Loop Post-Training for VLA : We propose DexHiL, a post-training pipeline for dexterous VLA models that introduces a novel intervention-aware data sampling strategy. This strategy prioritizes corrective segments from expert interventions, ensuring that the most valuable data for error recovery and task refinement is efficiently used. By dynamically re-weighting these corrective samples, DexHiL accelerates convergence, improves sample efficiency, and enhances model performance, particularly in high-DOF, contact-intensive tasks. II Related work II-A Dexterous Manipulation Data Collection System Dexterous manipulation data collection systems typically employ two approaches. The first method directly collects human hand data and generates dexterous hand data in a simulation environment using optimization methods [ 15 ] or reinforcement learning [ 16 , 17 , 18 ] . The other relies on teleoperation with retargeting algorithms, typically utilizing exoskeletons or master-slave arms [ 19 , 20 ] . However, both approaches face limitations when applied to Human-in-the-Loop systems. Exoskeletons or master-slave arms do not align precisely with the current dexterous hand’s robotic arm, preventing the intervention process from being made in an incremental form.