Entropy Regularized Task Representation Learning for Offline Meta-Reinforcement Learning

Mohammadreza Nakhaeinezhadfard, Aidan Scannell, Joni Pajarinen · 2024

Read Entropy Regularized Task Representation Learning for Offline Meta-Reinforcement Learning by Mohammadreza Nakhaeinezhadfard, Aidan Scannell, Joni Pajarinen (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-14834.txt.

Introduction

Mohammadreza Nakhaei and two colleagues, Aidan Scannell and Joni Pajarinen, wrote a 2024 paper on teaching AI agents to adapt quickly to new tasks. Their field is offline meta-reinforcement learning. The agent trains on stored data from many tasks and must then handle a new task with no fresh practice.

A common approach reads the agent's recent history, called the context, to work out which task it faces. The trouble is that stored history does not match what the agent meets later, so its guesses about the task go wrong. The authors add an entropy regularizer, a term that keeps the task representation from collapsing onto the training data. Better task representations should help agents generalize to tasks they have never seen.

Catalog record

CREATORMohammadreza Nakhaeinezhadfard, Aidan Scannell, Joni Pajarinen
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-14834

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Entropy Regularized Task Representation Learning for Offline Meta-Reinforcement Learning Mohammadreza Nakhaeinezhadfard, Aidan Scannell, Joni Pajarinen Abstract Offline meta-reinforcement learning aims to equip agents with the ability to rapidly adapt to new tasks by training on data from a set of different tasks. Context-based approaches utilize a history of state-action-reward transitions – referred to as the context – to infer representations of the current task, and then condition the agent, i.e., the policy and value function, on the task representations. Intuitively, the better the task representations capture the underlying tasks, the better the agent can generalize to new tasks. Unfortunately, context-based approaches suffer from distribution mismatch, as the context in the offline data does not match the context at test time, limiting their ability to generalize to the test tasks. This leads to the task representations overfitting to the offline training data. Intuitively, the task representations should be independent of the behavior policy used to collect the offline data. To address this issue, we approximately minimize the mutual information between the distribution over the task representations and behavior policy by maximizing the entropy of behavior policy conditioned on the task representations. We validate our approach in MuJoCo environments, showing that compared to baselines, our task representations more faithfully represent the underlying tasks, leading to outperforming prior methods in both in-distribution and out-of-distribution tasks. 1 Department of Electrical Engineering and Automation (EEA), Aalto University, Finland 2 Finnish Center for Artificial Intelligence, Finland {mohammadreza.nakhaei, aidan.scannell, joni.pajarinen}@aalto.fi Code — https://github.com/MohammadrezaNakhaei/ER-TRL 1 Introduction The goal of offline reinforcement learning (RL; Prudencio, Maximo, and Colombini 2023 ; Levine et al. 2020 ) is to leverage offline datasets to learn policies that can accomplish tasks without interacting with the environment. It is a promising approach for real-world applications where online RL may be expensive or dangerous, such as robotics ( Zhou et al. 2023 ; Luo et al. 2023 ) and healthcare ( Emerson, Guy, and McConville 2023 ; Zhu, Li, and Georgiou 2023 ) . Offline meta-RL (OMRL) extends offline RL to the meta-RL setting, where the goal is to solve a previously unseen task, by leveraging behaviors learned from a set of training environments. In contrast to meta-RL, OMRL considers the setting where the agent does not directly interact with the training environments but instead has access to offline datasets from each training environment. A promising approach to OMRL is context encoding ( Li et al. 2020 ; Li, Yang, and Luo 2020 ; Yuan and Lu 2022 ; Zhao, Zhou, and Liu 2023 ; Wang et al. 2023a ; Gao et al. 2024 ) , where a context encoder learns a representation of the task from a history of state-action-reward transitions, referred to as context. The agent is then conditioned on this learned task representation. For example, in off-policy RL, the policy and value functions are conditioned on the task representation, which enables adaptation to different tasks, including previously unseen ones. As highlighted by Gao et al. (2024) , context-based OMRL suffers from a distribution shift in the context encoder. This is because the context encoder is trained with contexts collected by the behavior policy, but at test time the context is collected by a different policy, including the learned policy conditioned on a prior task representation. As such, there is a discrepancy between the contexts at train and test time. This limits the context encoder’s ability to infer the correct task at test time, which consequently limits the agent’s ability to adapt to previously unseen tasks. One approach to alleviating this distribution mismatch is to minimize the mutual information between the task representations and the behavior policy. The intuition is that for environments that require dissimilar policies for each task, the context encoder should not rely on the characteristics of the behavior policy. As such, we would like our learned task representations to be independent of the behavior policy in order to reduce the impact of the behavior policy on the task representation. We sidestep the context shift issue by showing that minimizing the mutual information between the task representations and the behavior policy is equivalent to maximizing the entropy of a meta-behavior policy conditioned on the learned task representation. Our contributions are as follows: • We present Entropy Regularized Task Representation Learning ( er-trl ), an OMRL method that improves the ability of context-based OMRL methods to generalize to previously unseen environments. er-trl reduces the amount of context distribution shift by leveraging a generative adversarial network (GAN; Goodfellow et al. 2014 ) to minimize the mutual information between the task representations and the behavior policy, by estimating the entropy of the meta-behavior policy. • We show that our task representation learning improves performance in in-distribution tasks and also enhances generalization to out-of-distribution tasks. • We then show that er-trl ’s task representation learning outperforms prior methods as it can better predict the true task representation, e.g. the target velocity or direction in locomotion tasks. 2 Background This section formally defines the context-based OMRL framework and subsequently explores the critical challenge of context shift, where the distribution of contexts during training and testing diverges, significantly impacting the performance of context-based OMRL algorithms. Context-based Offline Meta-RL In OMRL, there is a distribution over tasks where each task is represented as a Markov Decision Process (MDP), {\mathcal{M}_{i}=\langle\mathcal{S},\mathcal{A},R_{i},P_{i},\gamma,\rho_{0}\rangle} , consisting of a shared state space \mathcal{S} , action space \mathcal{A} , discount factor \gamma\in[0,1] , and initial state distribution \rho_{0}(s_{0}) , and task-specific reward function R_{i}:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R} and transition dynamics P_{i}(s_{t+1}|s_{t},a_{t}) , which represents the distribution of possible next states conditioned on the current state and action. The objective is to train a meta-policy \pi that can generalize to new tasks, i.e. , maximizing the expected cumulative reward over the distribution of tasks \displaystyle J(\pi)=\mathbb{E}_{\begin{subarray}{c}\mathcal{M}_{i}\sim p(\mathcal{M})\\ s_{0}\sim\rho_{0}(s_{0})\\ s_{t+1}\sim P_{i}(\cdot\mid s_{t},a_{t})\\ a_{t}\sim\pi(a_{t}\mid\cdot)\end{subarray}}\left[\sum_{t=0}^{T}\gamma^{t}R_{i}(s_{t},a_{t})\right]. (1) A common assumption for OMRL is that the reward function and the transition dynamics for all tasks are deterministic. The offline dataset \mathcal{D}_{i}=\{(s_{j},a_{j},r_{j},s_{j}^{\prime})^{i}\}_{j=1}^{n_{i}} for task \mathcal{M}_{i} consists of state, action, reward, next state tuples collected using its corresponding behavior policy \pi_{\beta}^{i}(a|s) . During training, context-based OMRL methods use a mini-batch of transitions from the offline dataset c_{i}\subset\mathcal{D}_{i} – referred to as the context – to infer the task. A context encoder e_{\theta} learns a mapping from the context c_{i} to latent task representations z_{i} . The policy and value functions are then conditioned on the task representations z_{i} and trained to maximize Eq. 1 using the offline datasets. Note that there is no online interaction. Distribution Shift in Context Encoder During training, the context encoder is trained on offline datasets, collected by the behavior policy. The distribution over offline contexts p(c_{i}) depends on the transition dynamics, reward function, and the behavior policy: p(c_{i})=p(s_{1})\prod_{j=1}^{n}\pi_{\beta}^{i}(a_{j}|s_{j})R_{i}(s_{j},a_{j})P_{i}(s_{j+1}|s_{j},a_{j}). However, at test time, the context is collected with an exploration policy \pi_{\text{explore}} , which may be (i) the learned policy conditioned on some prior task representations z_{0} , (ii) a random policy, or (iii) a combination of them. As such, the difference in the behavior and exploration policies leads to the distribution over the context being different at train and test time p(c_{i}^{\text{test}})=p(s_{1})\prod_{j=1}^{n}\pi_{\text{explore}}(a_{j}|.)R_{i}(s_{j},a_{j})P_{i}(s_{j+1}|s_{j},a_{j}). Ideally, the context encoder should compress task-related information in the context and since the transition dynamics and reward

Read the complete text