Meta-Controller: Few-Shot Imitation of Unseen Embodiments and Tasks in Continuous Control

Seongwoong Cho Donggyun Kim Jinwoo Lee Seunghoon Hong · 2024

Read Meta-Controller: Few-Shot Imitation of Unseen Embodiments and Tasks in Continuous Control by Seongwoong Cho Donggyun Kim Jinwoo Lee Seunghoon Hong (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-12147.txt.

Introduction

Seongwoong Cho, Donggyun Kim, Jinwoo Lee, and Seunghoon Hong present Meta-Controller, a framework for robot imitation learning, in a paper published in two thousand twenty-four. A useful robot policy should transfer to new robot bodies and new tasks. Modular methods adapt to bodies but not tasks, while imitation methods usually assume one body.

Their few-shot behavior cloning needs only a handful of reward-free demonstrations. A joint-level representation unifies state and action spaces across different robots. A structure-motion encoder captures both shared knowledge and body-specific knowledge. A matching-based policy network then predicts actions for the new setting. Tests in the DeepMind Control suite showed the adaptive policy resisted overfitting while generalizing broadly.

Catalog record

CREATORSeongwoong Cho Donggyun Kim Jinwoo Lee Seunghoon Hong
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE65 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-12147

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Meta-Controller: Few-Shot Imitation of Unseen Embodiments and Tasks in Continuous Control Seongwoong Cho Donggyun Kim Jinwoo Lee Seunghoon Hong Abstract Generalizing across robot embodiments and tasks is crucial for adaptive robotic systems. Modular policy learning approaches adapt to new embodiments but are limited to specific tasks, while few-shot imitation learning (IL) approaches often focus on a single embodiment. In this paper, we introduce a few-shot behavior cloning framework to simultaneously generalize to unseen embodiments and tasks using a few ( e.g., five) reward-free demonstrations. Our framework leverages a joint-level input-output representation to unify the state and action spaces of heterogeneous embodiments and employs a novel structure-motion state encoder that is parameterized to capture both shared knowledge across all embodiments and embodiment-specific knowledge. A matching-based policy network then predicts actions from a few demonstrations, producing an adaptive policy that is robust to over-fitting. Evaluated in the DeepMind Control suite, our framework termed Meta-Controller demonstrates superior few-shot generalization to unseen embodiments and tasks over modular policy learning and few-shot IL approaches. Codes are available at https://github.com/SeongwoongCho/meta-controller . 1 Introduction Generalizing across different robot embodiments and tasks with only a few demonstrations is a fundamental challenge in continuous control [ 6 , 11 , 40 , 29 , 33 , 14 ] . This capability is crucial for developing versatile and adaptive robotic systems that can operate effectively in diverse and dynamic environments. The high diversity of embodiments and tasks, however, makes this particularly challenging. Robot embodiments vary widely in their morphological ( e.g., number and connectivity of joints) and dynamic ( e.g., motor gear ratios, damping coefficients) configurations, complicating the design of a unified architecture capable of handling heterogeneous input states and output actions. Furthermore, the variety of tasks—such as locomotion, object manipulation, and navigation—requires the learning of transferable skills that can efficiently generalize across different tasks. Despite significant advancements in reinforcement learning (RL) and imitation learning (IL), achieving simultaneous generalization across diverse embodiments and tasks with a few demonstrations remains largely underexplored. Modular policy learning approaches [ 34 , 18 , 16 , 12 , 10 ] have shown promise by learning modular policies that can be shared across embodiments with different morphologies. However, these methods primarily focus on specific task types such as locomotion, and lack the flexibility to adapt to a wide range of control tasks, limiting their broader applicability. Conversely, few-shot IL approaches [ 8 , 7 , 39 , 13 , 37 , 36 ] aim to learn novel tasks from a few demonstrations. These techniques excel in scenarios with sparse training data but typically concentrate on a single embodiment with fixed morphological and dynamic structures, restricting their ability to generalize across various embodiments and tasks. As a result, these two fields have developed independently, and their integration remains an open area of research. To address these challenges, we propose a novel framework that flexibly learns arbitrary control tasks on unseen embodiments using a few ( e.g., five) reward-free demonstrations. To handle heterogeneous embodiments within a unified architecture, we tokenize states and actions into joint-level representations, since joints serve as the fundamental building blocks of robots and provide a modular representation for compositional generalization to unseen embodiments [ 12 , 10 ] . Given this unified I/O, we employ a state encoder to capture both embodiment-specific knowledge about morphology and dynamics and shared knowledge about the physics governing the environment. Then we design a matching-based policy network that predicts actions from the encoded states, conditioned on a few demonstrations. Our model is trained using episodic meta-learning on a dataset comprising various embodiments and tasks, followed by few-shot behavior cloning on unseen embodiments and tasks using a few reward-free demonstrations. Our key contributions are as follows: (1) We design a novel structure-motion encoder that operates on joint-level state representations, efficiently disentangling embodiment-specific and task-specific knowledge. (2) We propose a matching-based meta-learning framework that efficiently transfers knowledge of local motions to quickly learn unseen tasks with a few demonstrations. (3) We evaluate our framework in various environments within the DeepMind Control suite, encompassing diverse embodiments and tasks, demonstrating superior few-shot learning performance over existing baselines. 2 Problem Setup A reinforcement learning (RL) problem involves an agent interacting with an environment, typically modeled as a Markov Decision Process (MDP). An MDP is represented by the tuple (\mathcal{S},\mathcal{A},P,R) , where \mathcal{S} is the state space, \mathcal{A} is the action space, P:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\to\mathbb{R} is the transition probability, and R:\mathcal{S}\times\mathcal{A}\xrightarrow{}\mathbb{R} is the reward function. In conventional RL, an agent learns a policy \pi:\mathcal{S}\xrightarrow{}\mathcal{A} that maximizes expected cumulative rewards \mathbb{E}_{\pi}[\gamma^{t}R(s_{t},a_{t})] , where \gamma^{t}\in[0,1] is a discount factor. Training such an RL agent typically requires numerous interactions with the environment and a carefully designed reward function, making it burdensome to learn new tasks. Behavior cloning (BC) addresses these challenges by using supervised learning techniques to imitate an expert policy from offline demonstrations. We focus on the few-shot BC setting, where the training data consists of a few reward-free demonstrations \mathcal{D}=\{\tau_{i}\}_{i\leq N} , with each demonstration \tau_{i}=\{(s_{t}^{i},a_{t}^{i})\}_{t\leq T} being a temporal sequence of states and actions performed by an expert model. In this paper, we consider continuous control problems of multi-joint robots that involve various embodiments and tasks. An embodiment \mathcal{E} refers to the physical configuration of robots, which includes (1) the morphology, i.e., the shape, size, and arrangement of the components such as limbs, joints, and sensors, and (2) the dynamics parameters that affect the robot’s behavior, e.g., motion inertia, mass, gear ratios, and damping. In terms of MDP, different embodiments can have different dimensionality of state and action spaces, e.g. , \dim(\mathcal{S}_{\mathcal{E}_{i}})\neq\dim(\mathcal{S}_{\mathcal{E}_{j}}) and \dim(\mathcal{A}_{\mathcal{E}_{i}})\neq\dim(\mathcal{A}_{\mathcal{E}_{j}}) for \mathcal{E}_{i}\neq\mathcal{E}_{j} , as well as distinct transition probabilities P_{\mathcal{E}} that determine the kinematics of the robot. A task \mathcal{T} is defined by a specific goal or objective that the robot must achieve within its environment, characterized by a reward function R_{\mathcal{T}} . Tasks can vary widely, ranging from locomotion and manipulation to complex interactions with dynamic environments. The combination of different embodiments \mathcal{E} and tasks \mathcal{T} creates a broad class of continuous control problems. Our objective is to achieve simultaneous generalization to unseen embodiments and tasks of continuous control with a few-shot behavior cloning framework. In other words, the model has to learn a policy for a novel continuous control problem from only a few demonstrations \mathcal{D} , where both embodiment \mathcal{E} and task \mathcal{T} can be arbitrary and previously unseen. 2.1 Challenges and Desiderata Despite achieving the simultaneous few-shot generalization to unseen embodiments and tasks is crucial for developing versatile and adaptive robotic systems, this problem remains less explored. We characterize two distinct challenges and desiderata to address each challenge. Handling Heterogeneous Embodiments To generalize to arbitrary embodiments in continuous control, the model must possess an architecture capable of universally handling heterogeneous states and actions of various embodiments. This necessitates a unified input/output (I/O) representation for states and actions, allowing for the sharing of structural characteristics (e.g., dimensionality) and semantics (e.g., input attributes or output control types) across different embodiments. Additionally, the encoder to extract state features should be able to capture transferable knowledge across different embodiments as well as embodiment-specific knowledge to flexibly adapt to distinct morphologies and dynamics of each embodiment. Few-shot Policy Adaptation To achieve robust

Read the complete text