Transferable Latent-to-Latent Locomotion Policy for Efficient and Versatile Motion Control of Diverse Legged Robots
Ziang Zheng, Guojian Zhan, Bin Shuai, Shengtao Qin, Jiangtao Li, Tao Zhang, Shengbo Eben Li · 2025
Read Transferable Latent-to-Latent Locomotion Policy for Efficient and Versatile Motion Control of Diverse Legged Robots by Ziang Zheng, Guojian Zhan, Bin Shuai, Shengtao Qin, Jiangtao Li, Tao Zhang, Shengbo Eben Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-17626.txt.
Introduction
Ziang Zheng and co-authors wrote a 2025 paper on a transferable walking policy for legged robots of many shapes. Training each robot and task from scratch eats data. Their design pretrains a core policy that works in a hidden code space, not raw senses or motor commands.
Task specific encoders and decoders plug into that core for each robot. A diffusion module guards against losing key detail in the code during pretraining. At fine tune time the core stays frozen and only the light plugs are trained. Simulations and real robots show faster adaptation to new bodies and tasks. Prior skill, even from other robots, carries over.
Catalog record
| CREATOR | Ziang Zheng, Guojian Zhan, Bin Shuai, Shengtao Qin, Jiangtao Li, Tao Zhang, Shengbo Eben Li |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-17626
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Transferable Latent-to-Latent Locomotion Policy for Efficient and Versatile Motion Control of Diverse Legged Robots Ziang Zheng, Guojian Zhan, Bin Shuai, Shengtao Qin, Jiangtao Li, Tao Zhang, Shengbo Eben Li Abstract Reinforcement learning (RL) has demonstrated remarkable capability in acquiring robot skills, but learning each new skill still requires substantial data collection for training. The pretrain-and-finetune paradigm offers a promising approach for efficiently adapting to new robot entities and tasks. Inspired by the idea that acquired knowledge can accelerate learning new tasks with the same robot and help a new robot master a trained task, we propose a latent training framework where a transferable latent-to-latent locomotion policy is pretrained alongside diverse task-specific observation encoders and action decoders. This policy in latent space processes encoded latent observations to generate latent actions to be decoded, with the potential to learn general abstract motion skills. To retain essential information for decision-making and control, we introduce a diffusion recovery module that minimizes information reconstruction loss during pretrain stage. During fine-tune stage, the pretrained latent-to-latent locomotion policy remains fixed, while only the lightweight task-specific encoder and decoder are optimized for efficient adaptation. Our method allows a robot to leverage its own prior experience across different tasks as well as the experience of other morphologically diverse robots to accelerate adaptation. We validate our approach through extensive simulations and real-world experiments, demonstrating that the pretrained latent-to-latent locomotion policy effectively generalizes to new robot entities and tasks with improved efficiency. I Introduction Legged robots are gaining increasing attention due to their ability to navigate complex terrains and perform tasks in environments that are challenging for wheeled or tracked robots. Their biological design allows them to traverse uneven surfaces [ 1 ] , climb obstacles [ 2 ] , and maintain stability, making them ideal for applications in search and rescue, exploration [ 3 ] , and industrial operations [ 4 , 5 ] . While optimization-based controllers such as model predictive control can handle structured environments, they still require meticulous platform-specific tuning and often fail to generalize across diverse morphologies or unstructured terrains. In contrast, learning-based methods have emerged as a promising alternative [ 6 ] . These techniques employ a neural network as a controller, mapping the robot’s observations to control inputs. The most representative paradigm is imitation learning (IL) which trains the neural network to align the observation-action pairs collected from expects [ 7 , 8 , 9 ] . Though there have been several remarkable demos achieved by IL, considerable high-quality data is still needed for continual performance improvement. RL does not rely on labeled data but learn optimal policies through trial and error by interacting with their environments [ 10 , 11 ] . This paradigm has been successfully supported across several domains including Chinese Go [ 12 ] , video games [ 13 ] . In legged robot locomotion field, RL has also demonstrated great potential in developing agile and robust locomotion skills directly from interaction with the environment [ 14 ] in both simulations [ 15 , 16 , 17 , 18 ] and real-world environments [ 19 ] . However, these learning-based locomotion controllers generally face two critical barriers: (1) the resulting policies are tightly coupled to a robot’s specific kinematic design, hindering cross-platform transfer [ 20 ] , and (2) the robot specific status gap like sim2real , different motor conditions and robot loads persists, causing the dynamic mismatches between different entities [ 21 , 22 ] . Figure 1 : Biological inspiration for latent-to-latent locomotion policy. Generally speaking, in many practical scenarios, the requisite training conditions are either inaccessible or insufficient [ 23 ] . For example, it is often impractical to train a robot in every possible complex or unstructured environment. This limitation underscores the need for approaches that can leverage pre-trained models or shared knowledge to improve the adaptation efficiency. We can draw inspiration from humans and animals that can maintain an unconscious, latent model of their body, which is continuously adjusted in the cerebellum and translated into motor actions through neural signals as shown in Fig. 1 . Thereby a natural idea rises: Can we abstract both observation and action into the latent space to enable more efficient policy transfer? Hence we propose a latent training framework, L3P (Latent-to-Latent Locomotion Policy) to enable skill transfer and platform adaptation. Unlike traditional approaches that directly map robot-specific observation to actions, our framework maps raw observations through robot-specific encoder from diverse robots—regardless of their physical differences—into a shared latent space. Then the policy outputs latent actions that serve as prototypes of robot behavior, which can be decoded as real control command by task-specific action decoder. Therefore, the trained latent-to-latent has the potential to contain general abstract motion skills and can be efficiently transferred. To ensure that both the latent observations and latent actions capturing sufficient information for effective decision-making, we integrate a diffusion recovery module that leverages the powerful multi-modal fitting capability of diffusion models. This module guarantees that the latent representations are capable of faithfully recovering the original information, thus enabling seamless encoder-decoder switching and cross-platform policy transfer. Our contributions are summarized as follows: • We propose a Latent-to-Latent Locomotion policy (L3P) framework that decomposes control into observation encoder, latent policy, and action decoder. This design enables the observation encoder to adapt across diverse terrains while the action decoder can be tailored to different robotic individuals, thereby learning core universal skills that are readily reusable. Our design allows a robot to leverage its own prior experience across different tasks as well as the experience of other morphologically diverse robots to accelerate adaptation. • We introduce a diffusion recovery module to ensure accurate reconstruction of latent states, maintaining consistency between the encoder-decoder pairs and ensuring that the latent representations capture sufficient critical information for decision-making. By utilizing a diffusion-based recovery module and joint optimizing policy performance and latent reconstruction loss, our approach ensures that the latent-to-latent policy encapsulates well-established skills and is ready for transfer without information loss. • Extensive experiments in simulation and on real-world platforms show that our approach achieves up to 2 \times higher zero-shot transfer success rates and adapts 3 \times faster to new platforms compared to baseline RL policies. Figure 2 : Overview of the L3P framework, which enables transferable locomotion control across diverse legged robots. The framework consists of three key modules: (1) an observation latent encoder that maps raw sensory inputs into a shared latent space, (2) a latent policy backbone that learns a generalizable control strategy, and (3) a latent action decoder that translates latent actions into robot-specific motor commands. Recovery modules for both observation and action ensure latent space consistency. The framework is trained in three stages: (i) latent space definition and alignment, (ii) transfer from a single entity to diverse robots, and (iii) zero-shot generalization from simple to complex terrains. II Preliminaries II-A Reinforcement Learning We consider the standard Markov Decision Process setting, where an agent interacts with an environment defined by a state space \mathcal{S} , an action space \mathcal{A} , and a reward function r(s,a) , with the goal of maximizing the expected cumulative reward: J(\pi)=\mathbb{E}_{\pi}\left[\sum_{t=0}^{T}\gamma^{t}\,r(s_{t},a_{t})\right], (1) where \gamma\in[0,1) is the discount factor. II-B Diffusion Models for Recovery