Reinforcement Learning from Wild Animal Videos
Elliot Chane-Sane, Constant Roux, Olivier Stasse, Nicolas Mansard · 2024
Read Reinforcement Learning from Wild Animal Videos by Elliot Chane-Sane, Constant Roux, Olivier Stasse, Nicolas Mansard (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-04273.txt.
Introduction
Elliot Chane-Sane and co-authors teach a four-legged robot to move by watching wildlife films. Nature documentaries hold thousands of hours of animals walking, leaping, and resting in the wild, a free library of graceful motion. The hard part is turning that footage into lessons a metal body can use.
First, a classifier learns to name actions in animal clips. Then a control policy trains in simulation, rewarded whenever a camera view of the robot fools the classifier into seeing the animal act. The policy moved straight onto a real Solo quadruped. Despite the gulf between beast and machine, the robot learned to walk, jump, and hold still, with no reference motions or hand-built rewards.
Catalog record
| CREATOR | Elliot Chane-Sane, Constant Roux, Olivier Stasse, Nicolas Mansard |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-04273
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Reinforcement Learning from Wild Animal Videos Elliot Chane-Sane, Constant Roux, Olivier Stasse, Nicolas Mansard Abstract We propose to learn legged robot locomotion skills by watching thousands of wild animal videos from the internet, such as those featured in nature documentaries. Indeed, such videos offer a rich and diverse collection of plausible motion examples, which could inform how robots should move. To achieve this, we introduce Reinforcement Learning from Wild Animal Videos (RLWAV), a method to ground these motions into physical robots. We first train a video classifier on a large-scale animal video dataset to recognize actions from RGB clips of animals in their natural habitats. We then train a multi-skill policy to control a robot in a physics simulator, using the classification score of a third-person camera capturing videos of the robot’s movements as a reward for reinforcement learning. Finally, we directly transfer the learned policy to a real quadruped Solo. Remarkably, despite the extreme gap in both domain and embodiment between animals in the wild and robots, our approach enables the policy to learn diverse skills such as walking, jumping, and keeping still, without relying on reference trajectories nor skill-specific rewards. Figure 1: RLWAV trains a video classifier on 8,791 wild animal videos in natural environments to transfer the learned skills to a real quadruped robot with reinforcement learning, without relying on reference trajectories nor hand-designed rewards for each skill. Here, we present the motions resulting from the transfer of the keeping still, walking, and jumping skills to the Solo-12 robot. 1 Introduction Robot learning for control often involves a repetitive cycle: designing algorithms - such as refining a reward function in reinforcement learning (RL) -, analyzing the resulting behavior through video, and iterating until a satisfactory policy is achieved. Learning directly from videos could streamline this process by directly optimizing for visually successful rollouts, thus aligning the training process with the desired outcome. Internet videos offer a vast and diverse array of motion examples across a wide range of scenarios. Similar to how foundation models in computer vision and natural language processing have achieved impressive generalization by leveraging internet-scale data [ 1 , 2 , 3 , 4 , 5 ] , allowing robots to imitate these motions at scale could lead to new advancements in generalist robots. Just as humans and animals often learn by observing others [ 6 ] , even those with different body morphologies, robots could benefit from similar observational learning strategies. Recent research has demonstrated the potential of leveraging large datasets of videos of humans interacting with objects in everyday activites for robotic manipulation [ 7 , 8 , 9 ] . In this work, we instead ask the following question: Can robots learn locomotion skills by watching thousands of wild animal videos? There exists a wealth of footage showcasing a wide range of animal species, from mammals and reptiles to amphibians and birds [ 10 , 11 , 12 ] . However, compared to learning manipulation tasks from human demonstrations, learning locomotion from these videos presents additional challenges. First, the embodiment gap between animals and robots is wider, as body dynamics play a lead role in defining locomotion and balance behavior. Prior works on behavior transfer from animals to robots has aimed to reduce this gap by tracking keypoints or estimating poses from videos [ 13 , 14 , 15 ] , typically using recordings of animals with similar morphology to the target robot (e.g. transferring motions from dogs to quadruped robots) in controlled lab environments. Yet, this approach limits the range of available video resources and restricts robot designs to specific bio-inspired shapes for which data acquisition is practically feasible. On the other hand, many natural skills, like walking or jumping, emerge across species and can be easily identified by human observers despite these morphological differences. We argue that the embodiment gap between robots and animal species with similar forms is no greater than the gap between different classes in the animal kingdom, for instance between birds and mammals. Cross-embodiment relationships can be learned, but doing so would require large video datasets, encompassing diverse species in their natural habitats, where they freely exhibit their behaviors, yet resulting in poor camera angles, occlusions, or multiple animals in the same frame. Two main limitations prevent the direct application of this idea. First, there is no obvious correspondence between the body mechanics of animals and robots. Second, cross-embodiment visual imitation requires physical grounding - replicating locomotion skills captured from animal videos as faithfully as possible while adhering to the capabilities and limits of the physical morphology of the robot. To overcome these challenges, we introduce Reinforcement Learning from Wild Animal Videos ( RLWAV ), a method for grounding skill concepts from videos of animals in their natural environments into the behaviors of legged robots. First, we train a video encoder network using the Animal Kingdom dataset [ 11 ] , a large and diverse collection of labeled animal videos spanning various species, sourced from the internet such as wildlife documentaries. This network learns to recognize actions directly from video pixels, namely ”Keeping still”, ”Walking”, ”Running” and ”Jumping”. By training on a wide range of scenarios and embodiments, we intend for the network to generalize to robot behaviors in a zero-shot manner, i.e without having been trained on robot videos. Then, we train a multi-skill policy in a physics simulator [ 16 , 17 ] to ground these learned action concepts in realistic robot behaviors. Using constrained RL [ 18 , 19 , 20 ] , we optimize the robot policy to maximize, as a reward, the classification score of the corresponding skill label obtained on videos of the robot movements captured from a third person view in simulation. We impose task-agnostic constraints related to the robot embodiments, similarly for each transferred skill. The physics simulator, along with the constraints, ensures that the robot behaviors remain physically plausible and transferable to real-world scenarios. Despite the significant gap in domain and embodiment between animals and robots, our approach successfully enables robots to acquire distinct skills corresponding to the considered action classes in the animal video dataset, without the need for reference trajectories or skill-specific reward functions. We validate our approach in simulation as well as on a real Solo-12 quadruped robot, demonstrating the emergence and successful transfer of multiple skills which include keeping still, walking with two different styles and jumping on the spot (see Figure 1 ). To the best of our knowledge, this is the first demonstration of successful transfer from a large, diverse dataset of wild animal videos across various species to physical robot locomotion. 2 Related Work Over the past few years, training locomotion policies in physics simulators [ 21 , 22 , 16 ] using reinforcement learning, then transferring them to real robots [ 23 , 24 , 25 , 17 , 26 , 27 ] has proven remarkably effective for acquiring diverse skills for legged robots. These include high-speed running [ 28 , 29 , 30 , 31 ] , jumping [ 32 , 33 , 26 , 34 , 35 ] , and traversing challenging terrains [ 36 , 37 , 38 , 39 , 40 , 41 ,