Learning to Assist Humans without Inferring Rewards
Vivek Myers, Evan Ellis, Sergey Levine, Benjamin Eysenbach, Anca Dragan · 2024
Read Learning to Assist Humans without Inferring Rewards by Vivek Myers, Evan Ellis, Sergey Levine, Benjamin Eysenbach, Anca Dragan (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-02623.txt.
Description
Abstract Assistive agents should make humans’ lives easier. Classically, such assistance is studied through the lens of inverse reinforcement learning, where an assistive agent (e.g., a chatbot, a robot) infers a human’s intention and then selects actions to help the human reach that goal. This approach requires inferring intentions, which can be difficult in high-dimensional settings.
Catalog record
| CREATOR | Vivek Myers, Evan Ellis, Sergey Levine, Benjamin Eysenbach, Anca Dragan |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2411-02623
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Learning to Assist Humans without Inferring Rewards Vivek Myers, Evan Ellis, Sergey Levine, Benjamin Eysenbach, Anca Dragan Abstract Assistive agents should make humans’ lives easier. Classically, such assistance is studied through the lens of inverse reinforcement learning, where an assistive agent (e.g., a chatbot, a robot) infers a human’s intention and then selects actions to help the human reach that goal. This approach requires inferring intentions, which can be difficult in high-dimensional settings. We build upon prior work that studies assistance through the lens of empowerment: an assistive agent aims to maximize the influence of the human’s actions such that they exert a greater control over the environmental outcomes and can solve tasks in fewer steps. We lift the major limitation of prior work in this area — scalability to high-dimensional settings — with contrastive successor representations. We formally prove that these representations estimate a similar notion of empowerment to that studied by prior work and provide a mechanism for optimizing it. Empirically, our proposed method outperforms prior methods on synthetic benchmarks, and scales to Overcooked, a cooperative game setting. Theoretically, our work connects ideas from information theory, neuroscience, and reinforcement learning, and charts a path for representations to play a critical role in solving assistive problems. 1 1 1 Code: https://github.com/vivekmyers/empowerment_successor_representations Website: https://empowering-humans.github.io 1 Introduction Figure 1 : We propose an algorithm for training assistive agents to empower human users — the assistant should take actions that enable human users to visit a wide range of future states, and the human’s actions should exert a high degree of influence over the future outcomes. Our algorithm scales to high-dimensional settings, opening the door to building assistive agents that need not directly reason about human intentions. AI agents deployed in the real world should be helpful to humans. When we know the utility function of the humans an agent could interact with, we can directly train assistive agents through reinforcement learning with the known human objective as the agent’s reward. In practice, agents rarely have direct access to a scalar reward corresponding to human preferences (if such a consistent model even exists) [ 1 ] , and must infer them from human behavior [ 2 , 3 ] . This inference can be challenging, as humans may act suboptimally with respect to their stated goals, not know their goals, or have changing preferences [ 4 ] . Optimizing a misspecified reward function can have poor consequences [ 5 ] . An alternative paradigm for assistance is to train agents that are intrinsically motivated to assist humans, rather than directly optimizing a model of their preferences. An analogy can be drawn to a parent raising a child. A good parent will empower the child to make impactful decisions and flourish, rather than proscribing an “optimal” outcome for the child. Likewise, AI agents might seek to empower the human agents they interact with, maximizing their capacity to change the environment [ 6 ] . In practice, concrete notions of empowerment can be difficult to optimize as an objective, requiring extensive modeling assumptions that don’t scale well to the high-dimensional settings deep reinforcement learning agents are deployed in. What is a good intrinsic objective for assisting humans that doesn’t require these assumptions? We propose a notion of assistance based on maximizing the influence of the human’s actions on the environment. This approach only requires one structural assumption: the AI agent is interacting with an environment where there is a notion of actions taken by the human agent — a more general setting than the case where we model the human actions as the outcome of some optimization procedure, as in inverse RL [ 7 , 8 ] or preference-based RL [ 9 ] . Prior work has studied various objectives for empowerment. For instance, Du et al., [6] approximate human empowerment as the variance in the final states of random rollouts. Despite excellent results in certain settings, this approach can be challenging to scale to higher dimensional settings, and does not necessarily enable human users to achieve the goals the want to achieve. By contrast, our approach exclusively empowers the human with respect to the distribution of (useful) behaviors induced by their current policy (which we term the effective empowerment ), and can be implemented through a simple objective derived from contrastive successor features, which can then be optimized with scalable deep reinforcement learning ( Fig. 1 ). We provide a theoretical framework connecting our objective to prior work on empowerment and goal inference, and empirically show that agents trained with this objective can assist humans in the Overcooked environment [ 10 ] as well as more complex versions of the obstacle gridworld assistance benchmark proposed by Du et al., [6] . Our core contribution is an objective for training agents that are intrinsically motivated to assist humans without requiring a model of the human’s reward function. The approach, Empowerment via Successor Representations (ESR), maximizes the influence of the human’s actions on the environment, and, unlike past approaches for assistance without reward inference, is based on a scalable model-free objective that can be derived from learned successor features that encode which states the human may want to reach given their current action. By maximizing effective empowerment, our objective empowers the human to reach the desired states, not all states, without assuming a human model. We analyze this objective in terms of empowerment and goal inference, drawing novel mathematical connections between time-series representations, decision-making, and assistance. We empirically show that agents trained with our objective can assist humans in two benchmarks proposed by past work: the Overcooked environment [ 10 ] and an obstacle-avoidance gridworld [ 6 ] . 2 Related Work Our approach broadly connects ideas from contrastive representation learning and intrinsic motivation to the problem of assisting humans. Assistive Agents. There are two lines of past work on assistive agents that are most relevant. The first line of work focuses on the setting of an assistance game [ 2 ] , where a robot (AI) agent tries to optimize a human reward of which it is initially unaware. Practically, inverse reinforcement learning (IRL) can be used in such a setting to infer the human’s reward function and assist the human in achieving their goals [ 3 ] . The key challenge with this approach is that it requires modeling the human’s reward function. This can be difficult in practice, especially if the human’s behavior is not well-modeled by the reward architecture. Slightly misspecified reward functions can lead to catastrophic outcomes (i.e., directly harmful behavior in the assistance context) [ 11 , 12 , 13 ] . These concerns have prompted interest, under the umbrella term AI Alignment , in ensuring agents are safe and adhere to human values [ 14 , 15 , 16 , 17 ] . Our approach avoids some of these issues by focusing on empowerment, a more general objective that does not require modeling the human’s reward function. The second line of work focuses on empowerment-like objectives for assistance and shared autonomy. Empowerment generally refers to a measure of an agent’s ability to influence the environment [ 18 , 19 ] . In the context of assistance, Du et