Hierarchical Meta-Reinforcement Learning via Automated Macro-Action Discovery
Minjae Cho, Chuangchuang Sun · 2024
Read Hierarchical Meta-Reinforcement Learning via Automated Macro-Action Discovery by Minjae Cho, Chuangchuang Sun (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-11930.txt.
Introduction
Robots that learn must often master many tasks and adapt fast to new ones, and Minjae Cho and Chuangchuang Sun propose a three-level learning architecture for that challenge. The top level learns representations of tasks; the middle level discovers macro-actions — reusable chunks of behaviour — on its own; the bottom level learns the fine primitive actions. Macro-actions are stripped of task-specific details, so they can be recomposed across tasks.
Guided by macro-actions, the low-level policy reaches goals more efficiently and is less likely to forget old skills while learning conflicting new ones. Separate training schemes for each level keep the deep hierarchy stable. In the MetaWorld benchmark suite, the approach improved both sample efficiency and success rates over prior leading methods. The authors present automated macro-action discovery as a route to agents that carry skills from task to task.
Catalog record
| CREATOR | Minjae Cho, Chuangchuang Sun |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-11930
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Hierarchical Meta-Reinforcement Learning via Automated Macro-Action Discovery Minjae Cho, Chuangchuang Sun Abstract Meta-Reinforcement Learning (Meta-RL) enables fast adaptation to new testing tasks. Despite recent advancements, it is still challenging to learn performant policies across multiple complex and high-dimensional tasks. To address this, we propose a novel architecture with three hierarchical levels for 1) learning task representations, 2) discovering task-agnostic macro-actions in an automated manner, and 3) learning primitive actions. The macro-action can guide the low-level primitive policy learning to more efficiently transition to goal states. This can address the issue that the policy may forget previously learned behavior while learning new, conflicting tasks. Moreover, the task-agnostic nature of the macro-actions is enabled by removing task-specific components from the state space. Hence, this makes them amenable to re-composition across different tasks and leads to promising fast adaptation to new tasks. Also, the prospective instability from the tri-level hierarchies is effectively mitigated by our innovative, independently tailored training schemes. Experiments in the MetaWorld framework demonstrate the improved sample efficiency and success rate of our approach compared to previous state-of-the-art methods. keywords Hierarchical Reinforcement Learning, Meta-Reinforcement Learning, Temporal Abstractions 1 Introduction Reinforcement Learning (RL) has rapidly advanced, enhancing decision-making capabilities in high-dimensional and complex systems. Recent innovations have expanded RL to address diverse challenges, including multi-task RL, safe RL, and offline RL, thus broadening its practical applications. Among these advancements, meta-learning emphasizes training an agent to adapt to new tasks by leveraging previously acquired knowledge. This often involves balancing the performance across multiple training tasks, exposing the learner to diverse task-parameters to facilitate future adaptation to any prospective test task within the same distribution. Hence, due to the inherent challenge in meta-learning [ 1 ] , common methodologies are typically tested on narrow task distributional learning scenarios such as a jumping robot with varying loads (altering transition dynamics) and a running agent with different velocity criteria (changing optimality). Although it faces challenges when extended to more complex meta-learning scenarios, task representation learning [ 2 ] shows its potential in such environments by identifying shared structures across given training tasks. The discovered representations offer fundamental insights into tasks, aiding the decision-maker in adapting and performing effectively on new tasks. Building on task representations, which expand the input space with additional conditioners, we aim to further distribute the policy’s information load by utilizing high-level actions, or macro-actions . Specifically, macro-action provides directional action vectors to the low-level policy. Then, the policy fills with detailed low-level control input along the given direction. This approach is particularly useful for learning multiple tasks and adapting to new ones in hard-parameter sharing. In this setting, effective distribution of information processing can handle challenging task-specific instructions, ensuring that the policy retains its previously acquired optimal behavior in a particular task while learning new tasks. The illustration is given in Figure 1 . Figure 1: This overview outlines our approach. Our higher-level architecture, HiMeta, offers high-level directional action predictions to the policy, on which the policy generates primitive actions based on these directions. Additionally, the directional action prediction is trained using a modified VAE designed to link the current state to the desired goal state. This enables the effective distribution of decision-making tasks, facilitating solutions for complex, high-dimensional multi-task and meta-learning scenarios. In short, beyond the existing bi-level representation learning (task-representation | low-level control), we introduce a tri-level method (task-representation | high-level control | low-level control) that includes an additional layer for constructing macro-actions. For task-representation, we adopt the goal state concept from hierarchical RL literature [ 3 , 4 ] . For high-level control inputs, we propose a modified Variational AutoEncoder (VAE) to identify an effective placeholder for low-level control. This placeholder provides both sign-based and magnitude-level guidance, enabling the policy to refine it into precise low-level controls. In the architecture, the encoder predicts macro-actions, while the decoder uses imputations to bridge the current state with the goal state. Therefore, this approach brings appealing properties in complex, sparse, long-horizon problems. We provide mathematical views of the mechanism of the encoder and decoder. For stacking three levels of layers, we design each to have an independent role in decomposition to avoid the “curse of hierarchy" [ 5 , 6 ] , which refers to the instability of learning higher-level layers caused by frequent dynamical transitions by the low-level decision-maker. While this challenge has previously limited hierarchical learning to bi-level layers, our novel solution of independent training schemes tailored for each hierarchical layer effectively enables three-layer hierarchical learning, demonstrating its effectiveness in broad meta-task learning. Our contributions are as follows: • Proposing a novel architecture to automatically discover macro-actions on which the policy fills the low-level input. This additionally prevents the loss of optimality in a particular task while learning news by the distribution of information load. • Introducing a training scheme and techniques to avoid the “curse of hierarchy,” enabling multi-level layers and collaborative learning across multiple tasks. • Conducting extensive meta-learning experiments on complex and diverse task distributions, achieving state-of-the-art performance compared to baselines. This letter outlines the preliminaries of meta-learning, offering insights into hierarchical decomposition in the following section Section 2 . Additionally, we detail our algorithmic approach and highlight the key insights that drive the state-of-the-art performance of our method in Section 3 , accompanied by extensive experiments in Section 4 . Finally, we provide our concluding remarks and discuss potential improvements and future work in Section 6 . 2 Preliminaries Reinforcement learning operates within the mathematical framework of Markov Decision Processes (MDP) [ 7 ] , characterized by the tuple \langle\mathcal{S},\mathcal{A},T(\cdot),R(\cdot),\gamma\rangle , where \mathcal{S}\in\mathbb{R}^{s} represents the state set (observation), \mathcal{A}\in\mathbb{R}^{a} is the action set, T(\cdot):\mathcal{S}\times\mathcal{A}\rightarrow\mathcal{S} denotes the transition function, and R(\cdot):\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R} is a reward function with \gamma as a discount factor. RL for any arbitrary task, \mathcal{T} , comprises its own transitional and reward dynamics. Formally, the definition of a task, \mathcal{T} , is \langle T(\cdot),R(\cdot)\rangle\in\mathcal{T} . The goal of deep RL is to find a policy parameterized by \theta , denoted as \pi_{\theta}(a|s) , that maps the state to an optimal action, yielding the highest cumulative rewards (optimal trajectory) under task \mathcal{T} , as shown below: \max_{\theta}\mathcal{J}(\pi_{\theta},\mathcal{T})=\mathbb{E}_{\begin{subarray}{c}a\sim\pi_{\theta}(s),\\ s^{\prime}\sim T(s,a)\end{subarray}}\left[\sum_{t=0}^{T}\gamma^{t}R(s_{t},a_{t})\right] (1) Meta-Learning Meta-learning extends multi-task learning by aiming for rapid adaptation to unseen tasks using prior knowledge. Formally, the trained policy aims to maximize the prospective optimality objective of the test task \mathcal{T}_{\text{test}} : \displaystyle\max_{\theta}\mathcal{J}(\pi_{\theta},\mathcal{T}_{\text{test}}),\quad\text{where}\quad\theta=\theta-\nabla\mathcal{J}(\pi_{\theta},\mathcal{T}_{train}) (2) Meta-training is typically achieved by conditioning task representations or finding a parametric space that is close to the new task’s optimal space as above. Task representation aims to discover common or distinct knowledge across tasks to better analyze the test tasks. This is typically done by posterior sampling over the variable of interest x , a latent task representation, using a variational autoencoder with the following objective: \log p(x)=\mathbb{E}_{q(z|x)}[\log p(x|z)]-\text{KL}(q(z|x)\|p(z)) (3) where p(\cdot) denotes the probability, q(\cdot) represents the posterior distribution, and KL is Kullback-Leibler divergence. The representation is a type of informative label, and it is then conditioned to the policy, \pi_{\theta} , to endow better differentiation. This creates useful input spaces that are easily shared across multiple tasks, inducing positive knowledge sharing. The construction of posterior distribution of representations has proven effective in many previous studies involving complex meta-adaptation