Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbone
Max Sobol Mark, Tian Gao, Georgia Gabriela Sampaio, Mohan Kumar Srirama, Archit Sharma, Chelsea Finn, Aviral Kumar · 2024
Read Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbone by Max Sobol Mark, Tian Gao, Georgia Gabriela Sampaio, Mohan Kumar Srirama, Archit Sharma, Chelsea Finn, Aviral Kumar (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-06685.txt.
Introduction
Max Sobol Mark and co-authors present a new way to train decision making systems in a 2024 paper. Learning by copying experts wastes weaker data, but reinforcement learning can use that data. The catch is that most training tools are built for one kind of policy model. Change the model, and training can fail or turn shaky.
Their method, called policy-agnostic RL, works across policy types and sizes. It samples many actions, ranks and refines them with a value model, and then learns from the best ones. Tests include robot policies, and one large robot model improved in real world trials within forty minutes.
Catalog record
| CREATOR | Max Sobol Mark, Tian Gao, Georgia Gabriela Sampaio, Mohan Kumar Srirama, Archit Sharma, Chelsea Finn, Aviral Kumar |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 76 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-06685
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbone Max Sobol Mark, Tian Gao, Georgia Gabriela Sampaio, Mohan Kumar Srirama, Archit Sharma, Chelsea Finn, Aviral Kumar Abstract Recent advances in learning decision-making policies can largely be attributed to training expressive policy models, largely via imitation learning. While imitation learning discards non-expert data, reinforcement learning (RL) can still learn from suboptimal data. However, instantiating RL training of a new policy class often presents a different challenge: most deep RL machinery is co-developed with assumptions on the policy class and backbone, resulting in poor performance when the policy class changes. For instance, SAC utilizes a low-variance reparameterization policy gradient for Gaussian policies, but this is unstable for diffusion policies [ 52 ] and intractable for autoregressive categorical policies. To address this issue, we develop an offline RL and online fine-tuning approach called policy-agnostic RL ( PA-RL ) that can effectively train multiple policy classes, with varying architectures and sizes. We build off the basic idea that a universal supervised learning loss can replace the policy improvement step in RL, as long as it is applied on “optimized” actions. To obtain these optimized actions, we first sample multiple actions from a base policy, and run global optimization (i.e., re-ranking multiple action samples using the Q-function) and local optimization (i.e., running gradient steps on an action sample) to maximize the critic on these candidates. PA-RL enables fine-tuning diffusion and transformer policies with either autoregressive tokens or continuous action outputs, at different sizes, entirely via actor-critic RL. Moreover, PA-RL improves the performance and sample-efficiency by up to 2 times compared to existing offline RL and online fine-tuning methods. We show the first result that successfully fine-tunes OpenVLA [ 22 ] , a 7B generalist robot policy, autonomously with Cal-QL [ 35 ] , an online RL fine-tuning algorithm, improving from 40% to 70% in the real world in 40 minutes. 1 Introduction Recent successes in training decision-making policies in a number of domains such as robotics and language agents stem largely from the use of expressive models combined with large-scale imitation-style training [ 59 , 6 , 22 , 5 ] , an approach that has been tried and tested in other areas of machine learning. However, training a policy once and freezing it is not good enough for many real-world deployment scenarios, where some adaptation is needed: for example, a robot must adapt its behavior as the surrounding environment or task changes; a language-model powered web navigation agent must attempt to use its own experience to improve behavior as it interacts more with the world [ 2 ] . The hallmark of an adaptation process is in its use of autonomous, non-expert data. In these use cases, imitation alone done once or applied repeatedly is not enough to guarantee the most efficient learning. Reinforcement learning (RL) provides a flexible framework for adaptation and fine-tuning with non-expert data, in offline [ 28 ] , online [ 35 ] , or hybrid [ 3 ] regime. In principle, off-the-shelf RL algorithms could be used to fine-tune any policy. For instance, by running actor-critic RL [ 49 ] , a policy can be trained towards maximizing the Q-function. However, most existing deep RL algorithms entangle the choice of training objectives and algorithm design with the choice of the policy class. For example, soft actor-critic (SAC) [ 15 ] , the base learner for many offline and online fine-tuning algorithms [ 26 , 35 ] , employs reprarameterization which is applicable to and stable for Gaussian (or tanh-Gaussian) policies: swapping the policy for a diffusion policy causes instability [ 52 ] . These instabilities can be severe to the extent that much weaker policy extraction techniques, e.g., critic-based re-ranking [ 17 , 34 ] on top of an imitation policy can outperform the policy gradient Wang et al. , even though theoretically this is not optimal (and indeed, with Gaussian policies performs worse empirically as well [ 12 , 13 ] ). Likewise, in order to extend conservative Q-learning (CQL) [ 26 ] to autoregressive token-based action distributions, Chebotar et al. 2023 had to make many modifications to the loss in the CQL algorithm. Overall, this means that adapting the best policy training methodologies or parameterization from one policy class to another can be challenging, and depending upon the policy itself, practitioners are forced to choose a weaker algorithm or spend cycles modifying other components of their approach. Figure 1 : Policy-agnostic reinforcement learning ( PA-RL ) is a simple approach for training any policy class and backbone via actor-critic RL in both the offline RL and online RL fine-tuning settings. This enables us to benefit from expressive power of different policy classes and priors from pre-training. Our results show that PA-RL is the first method to effectively improve diffusion policies and large generalist pre-trained policies in real-world robotic manipulation tasks. After pre-training with a few task demonstrations or zero-shot language-conditioned trials, it can significantly improve the performance of a base policy in as little as 40 minutes. On simulated benchmarks, we find substantially better results when using PA-RL with diffusion policies, where it sets a new state-of-the-art in both offline RL and online fine-tuning, as well as autoregressive policies. We tackle this challenge by developing a single offline RL and online fine-tuning approach, which we call policy-agnostic RL ( PA-RL ), that effectively fine-tunes any policy class or backbone. To perform policy improvement, the RL algorithm directly optimizes actions (instead of policy parameters). Doing so decouples policy improvement from training the parameteric policy, which can now be done by maximizing the likelihood of “optimized” actions via supervised learning. Concretely, to obtain these optimized actions, we first sample from the base policy several times to get multiple action candidates, and then take gradient steps with respect to the value function to improve those actions in the direction of maximizing Q-values. Then these optimized action samples replace the use of samples from the policy in any value-based RL algorithm, and are used to train the policy themselves. Note that while prior work does use supervised losses for policy training, our main contribution is to show that a single approach of this sort can effectively train multiple policy classes. We evaluate PA-RL empirically on a number of domains including simulated robotic manipulation tasks and real robots, with Gaussian, diffusion, and autoregressive categorical policies based on transformer backbones, on offline RL and online RL fine-tuning problems. Our results show that PA-RL achieves state-of-the-art performance, outperforming the next-best approach by 13% in aggregate over various domains. PA-RL produces the largest gains on long-horizon tasks that present multimodal offline data distributions (e.g., CALVIN [ 32 ] in our experiments), where a more expressive policy class beyond standard tanh-Gaussian is necessary for performance but has been challenging to use thus far. Most notably, PA-RL improves diffusion policies on two manipulation tasks by 80-100% within only 1-2 hours of online RL fine-tuning, on a real WidowX robot, and improves OpenVLA [ 22 ] , a 7B parameter robot VLA foundation model, by 75% after 1 hour of zero-shot trials and 40 minutes of online RL fine-tuning on the real robot.