Hierarchical Orchestra of Policies
Thomas P Cannon, Özgür Şimşek · 2024
Read Hierarchical Orchestra of Policies by Thomas P Cannon, Özgür Şimşek (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-03008.txt.
Description
Abstract Continual reinforcement learning poses a major challenge due to the tendency of agents to experience catastrophic forgetting when learning sequential tasks. In this paper, we introduce a modularity-based approach, called Hierarchical Orchestra of Policies (HOP), designed to mitigate catastrophic forgetting in lifelong reinforcement learning.
Catalog record
| CREATOR | Thomas P Cannon, Özgür Şimşek |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2411-03008
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Hierarchical Orchestra of Policies Thomas P Cannon, Özgür Şimşek Abstract Continual reinforcement learning poses a major challenge due to the tendency of agents to experience catastrophic forgetting when learning sequential tasks. In this paper, we introduce a modularity-based approach, called Hierarchical Orchestra of Policies (HOP), designed to mitigate catastrophic forgetting in lifelong reinforcement learning. HOP dynamically forms a hierarchy of policies based on a similarity metric between the current observations and previously encountered observations in successful tasks. Unlike other state-of-the-art methods, HOP does not require task labelling, allowing for robust adaptation in environments where boundaries between tasks are ambiguous. Our experiments, conducted across multiple tasks in a procedurally generated suite of environments, demonstrate that HOP significantly outperforms baseline methods in retaining knowledge across tasks and performs comparably to state-of-the-art transfer methods that require task labelling. Moreover, HOP achieves this without compromising performance when tasks remain constant, highlighting its versatility. 1 Introduction Neural networks are typically trained on data drawn independently and identically from a static distribution. While this approach works well in many cases, it becomes challenging in environments that are continuously changing or when new environments are introduced. In dynamic settings, such as reinforcement learning, robotics, or dialogue systems, models must adapt to new information while preserving knowledge from previous tasks ( Parisi et al.,, 2019 ) . However, neural networks often suffer from catastrophic forgetting, where learning new tasks leads to the rapid loss of previously acquired knowledge. The ability to learn new skills while maintaining existing knowledge is referred to as continual learning ( Ring,, 1994 ) . To address the challenge of catastrophic forgetting, researchers have developed three primary categories of methods. The first category, regularization-based , works by constraining updates to network parameters, thereby penalizing deviations from learned weight values that are critical for previous tasks. Notable examples of this approach include Elastic Weight Consolidation (EWC) and Synaptic Intelligence (SI) ( Kirkpatrick et al.,, 2017 ; Zenke et al.,, 2017 ) . The second category, replay-based , mitigates forgetting by periodically rehearsing past experiences, either through actual data or synthetic generations, ensuring that the network continues to perform well on earlier tasks ( Rolnick et al.,, 2019 ; Shin et al.,, 2017 ) . The third category, modularity-based , addresses the issue by structurally separating the network into modules, with each module dedicated to a specific task, thereby minimizing interference between tasks, prominent examples of this method are Progressive Neural Networks (PNN) by ( Rusu et al.,, 2016 ) and adaptive multi-column stacked sparse denoising autoencoder (AMC-SSDA) by Agostinelli et al., (2013) . Finally there are some methods which use a combination of these, for example, Schwarz et al., (2018) uses an active network and a knowledge-base network similar to the modularity-based methods, however it periodically compresses knowledge from the active into the knowledge network using EWC regularisation. Our method, the Hierarchical Orchestra of Policies (HOP), is a modularity-based approach and is most similar to PNN. However, unlike PNN, HOP does not rely on a task identifier during training, which makes it more suitable for domain-incremental learning ( Van de Ven and Tolias,, 2019 ) . Additionally, HOP differs in that it combines network probability outputs directly through hierarchical weightings, rather than using latent connections between networks. Finally, we demonstrate HOP at a significantly larger scale — 18 hierarchical policy levels compared to only 3 in PNN. We show that HOP performs comparably to PNN, even when PNN is provided with task labels while HOP is not. Furthermore, HOP mitigates catastrophic forgetting across several Procgen environments ( Cobbe et al.,, 2020 ) , achieving notable improvements over Proximal Policy Optimization (PPO), a reinforcement learning algorithm with inherent regularization properties ( Schulman et al.,, 2017 ) . 2 Hierarchical Orchestra of Policies Hierarchical Orchestra of Policies (HOP) is a modularity-based deep learning framework designed to mitigate catastrophic forgetting when learning new tasks. In this framework, a task is defined as a specific Markov Decision Process (MDP), where distinct levels within a procedurally generated environment, or levels across different environments, are considered separate tasks ( Puterman,, 2014 ) . Although HOP is task-agnostic, all tasks are treated as episodic. HOP relies on reinforcement learning algorithms that output stochastic policies, represented as \pi(a\mid s) ( Sutton,, 2018 ) . In our work, PPO serves as the base algorithm for HOP. The framework introduces three key mechanisms to form and use a collection of policies: 1. Checkpoints to freeze and store policies at a certain stage of training. 2. Orchestration of policy activation based on state similarity. 3. Hierarchical weightings to balance the influence of previous and new policies. These mechanisms enable the agent to recover and maintain performance across diverse tasks without significant interference, thereby promoting continual learning in complex environments. Checkpoints. The agent initially learns a policy using a base algorithm. After \mathcal{T}_{checkpoint} time-steps, HOP initializes a checkpoint , where the current learning policy \pi is frozen and evaluated in the currently available tasks. During checkpoint evaluation, if the episodic return R surpasses a predefined threshold R_{threshold} , all states encountered during that task episode are stored in a set of trusted states S_{m} which is linked with the policy checkpoint \pi_{m} , where m is the count of the checkpoint. The Orchestra. When the agent resumes learning, it dynamically activates checkpoint policies \pi_{m} determined by the similarity between the current state s_{t} and any s_{m}\in S_{m} . If the current state s_{t} is similar to any state in S_{m} , then the corresponding frozen policy \pi_{m} is activated ( I_{m}=1 ). Similarity is determined by a threshold value \omega , which, in all of our experiments, has been defined as any s_{m}\in S_{m} with a cosine similarity greater than 0.98 with s_{t} . Rather than selecting actions directly from the distribution of a single frozen policy ( a_{t}\sim\pi_{m}(s_{t}) ), which could lead to conflicts when multiple policy checkpoints are activated, HOP combines the distributions from activated policy checkpoints ( I_{m}\pi_{m} ) and the current learning policy \pi_{n} into a joined action policy, denoted as \pi_{n_{a}} (see equation 1 ). Here, n represents the current count of all policies, and the subscript a denotes the combined policy from which the action is sampled. This approach allows the agent to leverage past knowledge while adapting to new tasks, promoting continual learning. To avoid significant and undesired output shifts caused by small changes in the state, frozen policies predict actions based on the most similar state, s_{m}\in S_{m} , to the current state s_{t} ( Szegedy,, 2013 ) . This state is referred to as s^{*}_{m} , and actions are chosen as \pi_{m}(a\mid s^{*}_{m}) rather than directly from s_{t} . This dynamic activation of multiple policies is called the orchestra of policies, a term borrowed from Jonckheere et al., (2023) but applied differently in this work. Hierarchical Weightings. As the agent learns, it is expected to achieve higher task-specific rewards, which suggests that newer policies for the same tasks are likely to outperform older policy checkpoints for the same task. Thus, simply averaging all policies, as represented by \pi_{n_{a}}=\frac{1}{n}\sum_{m=0}^{n}\pi_{m} , is impractical. Moreover, because the agent does not know the identities of tasks, multiple policy checkpoints may activate; therefore,