Beyond the Boundaries of Proximal Policy Optimization
Charlie B. Tan Edan Toledo Benjamin Ellis Jakob N. Foerster Ferenc Huszár · 2024
Read Beyond the Boundaries of Proximal Policy Optimization by Charlie B. Tan Edan Toledo Benjamin Ellis Jakob N. Foerster Ferenc Huszár (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-00666.txt.
Description
Abstract Proximal policy optimization (PPO) is a widely-used algorithm for on-policy reinforcement learning. This work offers an alternative perspective of PPO, in which it is decomposed into the inner-loop estimation of update vectors, and the outer-loop application of updates using gradient ascent with unity learning rate.
Catalog record
| CREATOR | Charlie B. Tan Edan Toledo Benjamin Ellis Jakob N. Foerster Ferenc Huszár |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2411-00666
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Beyond the Boundaries of Proximal Policy Optimization Charlie B. Tan Edan Toledo Benjamin Ellis Jakob N. Foerster Ferenc Huszár Abstract Proximal policy optimization (PPO) is a widely-used algorithm for on-policy reinforcement learning. This work offers an alternative perspective of PPO, in which it is decomposed into the inner-loop estimation of update vectors, and the outer-loop application of updates using gradient ascent with unity learning rate. Using this insight we propose outer proximal policy optimization (outer-PPO); a framework wherein these update vectors are applied using an arbitrary gradient-based optimizer. The decoupling of update estimation and update application enabled by outer-PPO highlights several implicit design choices in PPO that we challenge through empirical investigation. In particular we consider non-unity learning rates and momentum applied to the outer loop, and a momentum-bias applied to the inner estimation loop. Methods are evaluated against an aggressively tuned PPO baseline on Brax, Jumanji and MinAtar environments; non-unity learning rates and momentum both achieve statistically significant improvement on Brax and Jumanji, given the same hyperparameter tuning budget. 1 Introduction Proximal policy optimization (PPO) ( Schulman et al., 2017b ) is ubiquitous within modern reinforcement learning (RL), having found success in domains such as robotics ( Andrychowicz et al., 2020b ) , gameplay ( Berner et al., 2019 ) , and research applications ( Mirhoseini et al., 2021 ) . Given it’s ubiquity, significant research effort has explored the theoretical ( Hsu et al., 2020 ; Kuba et al., 2022 ) and empirical ( Engstrom et al., 2020 ; Andrychowicz et al., 2020a ) properties of PPO. PPO is an on-policy algorithm; at each iteration it collects a dataset using the current (behavior) policy. This dataset is used to construct a surrogate to the true objective, enabling gradient-based optimization while seeking to prevent large changes in policy between iterations, similar to trust region policy optimization ( Schulman et al., 2017a ) . The solution to the surrogate objective is then taken as the behavior parameters for the following iteration, defining the behavior policy with which to collect the following dataset. The behavior policies are therefore exactly coupled with the preceding surrogate objective solution. In this work we instead consider the inner-loop optimization of each surrogate objective to estimate an update vector, which we name the outer gradient . A trivial result follows that the outer loop of PPO can be viewed to update the behavior parameters using unity learning rate \sigma=1 gradient ascent on the outer gradients. Using this insight we propose outer-PPO, a novel variation of PPO that employs an arbitrary gradient-based optimizer in the outer loop of PPO. Outer-PPO decouples the estimation and application of updates in way not possible in standard PPO. An illustration of outer-PPO applying a learning rate greater than unity is provided in figure 1 . The new behaviors enabled by outer-PPO raise several questions related to implicit design choices of PPO: Question 1 . Is the unity learning rate always optimal? Question 2 . Is the independence (lack of prior trajectory information e.g momentum) of each outer update step always optimal? Question 3 . Is initializing the inner loop surrogate objective optimization at the behavior parameters (without exploiting prior trajectory / momentum) always optimal? (i) \bm{\theta}_{k} \bm{\theta}_{k}^{*} (ii) \bm{\theta}_{k} \bm{\theta}_{k}^{*} \bm{g}^{O}_{k} (iii) \bm{\theta}_{k} \bm{\theta}_{k+1} \sigma\bm{g}^{O}_{k} Figure 1: Diagram of outer-PPO estimating and applying the outer gradient as an update. (i) Transitions are collected with policy \pi(\bm{\theta}_{k}) defining a surrogate objective and corresponding ‘trust-region’ (shaded) surrounding \bm{\theta}_{k} ; inner-loop optimization of the surrogate objective (blue dashed) yields \bm{\theta}_{k}^{*} . (ii) Outer-PPO computes outer gradient as \bm{g}^{O}_{k}\leftarrow\bm{\theta}_{k}^{*}-\bm{\theta}_{k} . (iii) Outer-PPO updates behavior parameters using an arbitrary gradient based optimizer applied to the outer gradient to give \bm{\theta}_{k+1} , in this case gradient ascent with a learning rate \sigma>1 . Standard PPO can be understood as directly taking \bm{\theta}_{k+1}\leftarrow\bm{\theta}_{k}^{*} , or as a special case of outer-PPO corresponding to gradient ascent with learning rate \sigma=1 . This work forms an empirical investigation of the aforementioned questions. To motivate this investigation, consider the clipping parameter \epsilon of PPO, controlling the size of the ‘trust region’ within which we seek to restrict our update. If \epsilon is set too low, we restrict ourselves to small policy updates. Conversely, if we set \epsilon too high we decrease the reliability of our update direction. In outer-PPO, introducing an outer learning rate decouples these two effects; we are able to reliably estimate an update vector using a moderate \epsilon , but then take step of large magnitude in this direction. We emphasize that we do not seek to identify the most performant configuration possible but to understand the performance of outer-PPO relative to a well-tuned PPO baseline. To this end we restrict the tuning of outer-PPO to simple grid searches applied to fixed base PPO hyperparameters. Our contributions are as follows: • We propose outer proximal policy optimization (outer-PPO), in which an arbitrary gradient-based optimizer is applied to the ‘outer gradients’ of PPO. By tracking the outer trajectory, outer-PPO further permits a momentum bias to be applied to the inner-loop initialization. • We optimize a PPO baseline through extensive hyperparameter sweeps (total of 38,400 agent trained) on subsets of Brax (6 tasks) ( Freeman et al., 2021 ) , Jumanji (4 tasks) ( Bonnet et al., 2024 ) , and MinAtar (4 tasks) ( Young & Tian, 2019 ) . We open-source the sweep database files to facilitate future research against strongly tuned baselines. • We perform three lightweight outer-PPO grid searches on non-unity outer learning rates, outer Nesterov momentum and biased-initialization, each addressing questions 1, 2 and 3 respectively. • We evaluate the outer-PPO methods against the baseline, using 64 seeds per task over the 14 different tasks. We find non-unity outer learning rates to yield the greatest improvement (5-10%) on both Brax and Jumanji. Outer Nesterov also improves performance on Brax and Jumanji. Biased initialization achieves a moderate improvement on Jumanji alone. No method improves over the baseline on MinAtar. • Given the stated empirical results we conclude the negative for questions 1, 2 and 3. Relaxing each of these PPO design choices can lead to consistent, statistically significant improvement of performance over at least one of the evaluated environment suites. • We propose that practitioners able to experiment may explore non-unity outer learning rates given the simplicity (single hyperparameter) and consistent improvement achieved on Brax and Jumanjji. 2 Background 2.1 Reinforcement Learning We consider the standard reinforcement learning formulation of a Markov decision process \mathcal{M}=\langle\mathcal{S},\mathcal{A},\mathcal{T},r,\gamma\rangle , where \mathcal{S} is the set of states, \mathcal{A} is the set of actions, \mathcal{T}:\mathcal{S}\times\mathcal{A}\rightarrow\Delta(\mathcal{S}) is the state transition probability function, r:\mathcal{S}\times\mathcal{A}\rightarrow\Delta(\mathbb{R}) is the reward function, and \gamma\in[0,1] is the discount factor. We use the notation \Delta(\bm{X}) to denote the probability distribution over a set \bm{X} . The reinforcement learning objective is to maximize the expected return \mathbb{E}_{\pi}[G_{t}]=\mathbb{E}_{\pi}[\sum_{t}\gamma^{t}r_{t}] given a policy \pi:\mathcal{S}\rightarrow\Delta(\mathcal{A}) defining the agent behavior. In actor-critic policy optimization the policy is explicitly represented as a parametric function \pi:\mathcal{S}\times\bm{\theta}^{\pi}\rightarrow\Delta(\mathcal{A}) , and a value function V:\mathcal{S}\times\bm{\theta}^{V}\rightarrow\mathbb{R} is employed to guide optimization. In deep RL ( Mnih et al., 2015 ; Silver et al., 2017 ) neural networks are used for the policy and value functions, for ease of notation we consider