Towards Bridging the Gap between Large-Scale Pretraining and Efficient Finetuning for Humanoid Control

Weidong Huang, Zhehan Li, Hangxin Liu, Biao Hou, Yao Su, Jingwen Zhang · 2026

Read Towards Bridging the Gap between Large-Scale Pretraining and Efficient Finetuning for Humanoid Control by Weidong Huang, Zhehan Li, Hangxin Liu, Biao Hou, Yao Su, Jingwen Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-21363.txt.

Introduction

Weidong Huang, Zhehan Li, Hangxin Liu, Biao Hou, Yao Su, and Jingwen Zhang bridge large-scale pretraining and cheap fine-tuning for humanoid robots. Trial-and-error training in huge parallel simulations can produce sturdy control. Some policies even move to real robots without extra work.

The catch is that the standard methods learn slowly from each trial. Slow learning makes it risky and costly to adapt a robot to a new setting. The authors aim to keep the strengths of big pretraining while adapting with far fewer trials. That balance is key to humanoids that can adjust safely outside the lab.

Catalog record

CREATORWeidong Huang, Zhehan Li, Hangxin Liu, Biao Hou, Yao Su, Jingwen Zhang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-21363

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Towards Bridging the Gap between Large-Scale Pretraining and Efficient Finetuning for Humanoid Control Weidong Huang, Zhehan Li, Hangxin Liu, Biao Hou, Yao Su, Jingwen Zhang Abstract rl ( rl ) is widely used for humanoid control, with on-policy methods such as ppo ( ppo ) enabling robust training via large-scale parallel simulation and, in some cases, zero-shot deployment to real robots. However, the low sample efficiency of on-policy algorithms limits safe adaptation to new environments. Although off-policy RL and model-based RL have shown improved sample efficiency, the gap between large-scale pretraining and efficient finetuning on humanoids still exists. In this paper, we find that off-policy sac ( sac ), with large-batch update and a high utd ( utd ) ratio, reliably supports large-scale pretraining of humanoid locomotion policies, achieving zero-shot deployment on real robots. For adaptation, we demonstrate that these sac -pretrained policies can be finetuned in new environments and out-of-distribution tasks using model-based methods. Data collection in the new environment executes a deterministic policy while stochastic exploration is instead confined to a physics-informed world model. This separation mitigates the risks of random exploration during adaptation while preserving exploratory coverage for improvement. Overall, the approach couples the wall-clock efficiency of large-scale simulation during pretraining with the sample efficiency of model-based learning during fine-tuning. Code and videos: https://lift-humanoid.github.io Figure 1: L arge-scale pretra I ning and efficient F ine T uning (LIFT) Framework. In stage (i), we implement SAC in JAX to support large-batch update and high utd , achieving fast, robust convergence in massively parallel simulation and zero-shot deployment to a real humanoid in outdoor experiments. In stage (ii), we pretrain a physics-informed world model on the SAC data, combining Lagrangian dynamics with a residual predictor to capture contact forces and other unmodeled effects. In stage (iii), we finetune both the policy and the world model to new environments while executing only deterministic actions in the environment. Stochastic exploration is confined to rollouts within the world model. This framework enhances both the safety and efficiency of finetuning. 1 Introduction Learning generalist physical agents is a long-standing goal in AI, and humanoid robots offer broad task coverage among robotic embodiments. PPO ( 39 ) becomes a mainstream baseline for humanoid robot control due to its robustness and fast wall-clock convergence in massively parallel GPU simulation ( 40 ) . However, when only limited real-world or cross-task data can be collected, on-policy methods are disadvantaged because they discard off-policy experience ( 18 ; 30 ; 12 ) . Moreover, even when a PPO policy achieves zero-shot deployment in new environments, performance metrics can degenerate ( 14 ) (e.g., poorer velocity tracking or reduced execution precision). These limitations highlight the need for frameworks that can reuse prior experience and adapt policies efficiently in new environments. Off-policy algorithms offer an appealing backbone for this paradigm because they can leverage replayed experience for sample-efficient finetuning ( 30 ) . However, these algorithms tend to overfit to replayed data, which can bias exploration during fine-tuning, especially when large-scale parallel simulation with them has received limited attention. During fine-tuning, directly adapting a policy to a new environment often leads to unsafe actions because of stochastic exploration. This risk is especially high for humanoid robots: small support polygons, particularly during single-support phases, make the system highly sensitive to perturbations. Constraining exploration within a learned world model provides a safer and more sample-efficient alternative ( 27 ; 20 ) , though synthetic rollouts may introduce model bias that destabilizes finetuning ( 15 ; 25 ) . A physics-informed world model ( 27 ) that incorporates structural priors from physcis improves the fidelity of synthetic rollouts, reduces model bias, and enables more reliable finetuning even with limited data. However, training such model-based systems from scratch ( 27 ) remains extremely slow in wall-clock time and prone to local minima. Meanwhile, we find that off-policy SAC ( 18 ) integrates more naturally with model-based algorithms ( 27 ) than PPO , achieving robust and sample-efficient improvements. These findings further motivate our pipeline. We introduce L arge-scale pretra I ning and efficient F ine T uning ( LIFT ), a three-stage framework as shown in the Figure 1 : (i) large-scale policy pretraining, (ii) physics-informed world model pretraining, and (iii) efficient finetuning of the policy and world model. We make the following contributions: 1. We provide a scalable JAX implementation of SAC that supports robust convergence in massively parallel simulation and zero-shot deployment to a physical humanoid robot within one hour of wall-clock training on a single NVIDIA RTX 4090 GPU. The resulting SAC policy also serves as the policy module within our model-based finetuning stage. 2. We develop a fine-tuning strategy that executes deterministic actions in new environments while limiting stochastic exploration in a physics-informed world model, which improves sample efficiency and stabilizes convergence. This physics-informed design is pivotal for finetuning, enabling data-efficient in-distribution adaptation and stronger out-of-distribution generalization. 3. We release an open-source pipeline for humanoid control spanning pretraining, zero-shot deployment, and finetuning, providing a practical baseline for the robotics community. 2 Related Works Large-Scale RL Pretraining in Simulation. With the rapid progress of GPU technology, several GPU-based physics simulators have emerged, such as IsaacGym ( 33 ) , Mujoco Playground ( 50 ) , and Brax ( 11 ) . These frameworks enable fast parallel simulation of thousands of robot environments directly on GPUs. ppo ( 39 ) has become the dominant baseline in this setting due to the ease of implementation and robust convergence. By leveraging domain randomization ( 46 ) , ppo policies can achieve fast wall-clock convergence and even zero-shot transfer to physical humanoid robots ( 38 ; 14 ) . However, the low sample efficiency of on-policy methods limits their ability to adapt or continue training in new environments. Off-Policy RL. When data collection is expensive or risky, improving sample efficiency becomes critical. Off-policy algorithms such as TD3 ( 12 ) and SAC ( 18 ) achieve higher sample efficiency by reusing past experiences and leveraging critic gradients to update the policy. SAC has been used to train quadruped robots from scratch with limited real-world data ( 19 ; 16 ; 17 ; 43 ) , and pretrained SAC policies can be finetuned on hardware with the same algorithm ( 42 ; 4 ) . However, these methods typically rely on stochastic exploration in the environment, where injected action noise may damage actuators or induce unsafe states. This risk is amplified for humanoids, whose smaller support polygons (especially in single-support phases) make them sensitive to perturbations compared to quadrupeds. These methods also underutilize large-scale parallel simulation, resulting in high wall-clock training time. Recent large-scale off-policy efforts include Parallel Q-Learning ( 28 ) , which scales DDPG ( 30 ) across massive simulations yet without sim-to-real validation, and FastTD3 ( 41 ) , which achieves humanoid sim-to-real but its finetuning ability in new environments remains unclear. 36 show that SAC can be made stable in massively parallel simulators through extensive parameter tuning, but their work also lacks sim-to-real evaluation and does not examine fine-tuning performance. Model-Based Techniques Model-based methods can further improve sample efficiency, such as MBPO ( 25 ) using synthetic rollouts and Dreamer (

Read the complete text