Generalizing Behavior via Inverse Reinforcement Learning with Closed-Form Reward Centroids

Filippo Lazzati, Alberto Maria Metelli · 2025

Read Generalizing Behavior via Inverse Reinforcement Learning with Closed-Form Reward Centroids by Filippo Lazzati, Alberto Maria Metelli (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-12010.txt.

Introduction

Copying an expert into a new setting fails when many reward functions explain the same demonstrations yet disagree on what to do next. Filippo Lazzati and one co-author confront that ambiguity in inverse reinforcement learning in 2025, and propose picking the average policy.

Their criterion averages, in effect, over the policies induced by a bounded subset of the rewards consistent with the expert's behavior. The surprise is practical: that average policy comes from planning with a single reward, the centroid of the subset, which the authors derive in closed form. An efficient algorithm estimates the centroid from an offline pile of demonstrations alone, and simulations illustrate how the resulting behavior relates to the expert's own.

Catalog record

CREATORFilippo Lazzati, Alberto Maria Metelli
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE168 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-12010

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Generalizing Behavior via Inverse Reinforcement Learning with Closed-Form Reward Centroids Filippo Lazzati, Alberto Maria Metelli Abstract We study the problem of generalizing an expert agent’s behavior, provided through demonstrations, to new environments and/or additional constraints. Inverse Reinforcement Learning (IRL) offers a promising solution by seeking to recover the expert’s underlying reward function , which, if used for planning in the new settings, would reproduce the desired behavior. However, IRL is inherently ill-posed : multiple reward functions, forming the so-called feasible set , can explain the same observed behavior. Since these rewards may induce different policies in the new setting, in the absence of additional information, a decision criterion is needed to select which policy to deploy. In this paper, we propose a novel, principled criterion that selects the “average” policy among those induced by the rewards in a certain bounded subset of the feasible set. Remarkably, we show that this policy can be obtained by planning with the reward centroid of that subset, for which we derive a closed-form expression . We then present a provably efficient algorithm for estimating this centroid using an offline dataset of expert demonstrations only. Finally, we conduct numerical simulations that illustrate the relationship between the expert’s behavior and the behavior produced by our method. 1 Introduction Imitation Learning (IL) pomerlau1988alvinn ; abbeel2004apprenticeship addresses the problem of efficiently learning a desired behavior by imitating an expert’s behavior osa2018IL . By avoiding the difficult task of specifying a reward function, IL offers a direct and effective way to construct artificial agents with human-like capabilities using demonstrations of behavior alone ho2016generativeadversarialimitationlearning ; Fu2017LearningRR ; reddy2019sqil ; garg2021IQlearn . Consequently, it has been successfully applied to a variety of real-world applications and control problems, including autonomous driving kuderer2015driving , healthcare yu2019healthcare , motion planning finn2016guided , and path planning ratliff2006maximum ; silver2010terrain . However, in many situations, the environment where the learner must operate has different dynamics or requires additional constraints compared to the environment where the expert’s demonstrations were collected Fu2017LearningRR ; arora2020survey ; schlaginhaufen2023identifiability . A common example are “sim-to-real” scenarios, where demonstrations are easily obtained in a simulator while the target environment is the (possibly more constrained) real world openai2019solvingrubikscuberobot ; matas2018simtoreal ; desai2020IfoTX . In such cases, the expert’s demonstrations provide valuable information about the desired behavior, but this information must be extrapolated, as merely transferring the expert’s observed policy can lead to failure Fu2017LearningRR ; ni2021firl ; arora2020survey . Inverse Reinforcement Learning (IRL) russell1998learning ; ng2000algorithms addresses this generalization problem by assuming the existence of an expert’s reward function that, if used for planning in the new environment, would induce the desired behavior amin2016resolving ; Fu2017LearningRR ; arora2020survey ; rolland2022identifiability ; schlaginhaufen2024transferability . The task is thus reduced to inferring the expert’s reward function from demonstrations of behavior. This inference requires knowledge of the true relationship between the expert’s policy and reward, referred to as the model of behavior skalse2023misspecificationinversereinforcementlearning . Common models of behavior studied in the literature include assuming that the expert’s policy is optimal for the expert’s reward ng2000algorithms , soft-optimal Ziebart2010ModelingPA ; Fu2017LearningRR , \epsilon -optimal poiani2024inversereinforcementlearningsuboptimal , the best policy satisfying certain constraints schlaginhaufen2023identifiability , or derived from a softmax over the optimal Q -function Ramachandran2007birl . Unfortunately, regardless of the chosen model of behavior, even if the expert’s policy is known exactly, the expert’s reward cannot be uniquely identified ng2000algorithms ; cao2021identifiability . Instead, there exists a set of equally plausible reward functions, known as the feasible set , that are consistent with the expert’s behavior metelli2021provably ; metelli2023towards . Crucially, these rewards generalize differently , meaning they induce different optimal policies in the target environment skalse2023invariancepolicyoptimisationpartial , leading to ambiguity about which behavior should be deployed. To address this, many works in the literature aim to reduce the size of the feasible set using additional data or structural assumptions amin2016resolving ; Fu2017LearningRR ; cao2021identifiability ; kim2021rewardidentification ; rolland2022identifiability ; schlaginhaufen2023identifiability ; schlaginhaufen2024transferability . In this work, we consider a more challenging and realistic setting where no additional data or structure is available, and we are requested to make the best possible decision given only the demonstrations. In this context, existing IRL approaches can be broadly grouped into methods that work well in the best-case, worst-case, or average-case. “Best-case” methods ziebart2008maxent ; choi2011mapbirl ; wulfmeier2016maximumentropydeepirl ; finn2016guided arbitrarily select a reward from the feasible set and deploy its corresponding policy. “Worst-case” methods lazzati2025rel deploy the policy that minimizes the worst-case suboptimality over all rewards in the feasible set. “On-average” methods Ramachandran2007birl ; michini2012improving ; bajgar2024walking minimize expected suboptimality under a prior reward distribution. Unsurprisingly, “best-case” methods tend to be overly optimistic and often fail to generalize, as observed in empirical studies such as finn2016guided ; Fu2017LearningRR ; viano2021robust . Conversely, “worst-case” methods can be overly conservative , and the resulting policy can be inefficient to compute lazzati2025rel . Positioned between these extremes, “on-average” methods offer a more reasonable compromise. However, the common choices of priors considered in the literature either rely on domain knowledge or turn out to be biased in the set of policies that the expert can demonstrate (see Section 3.2 ), making them intuitively unsuitable for our specific setting, where no additional information is available. In this paper, we introduce novel priors over rewards that are unbiased with respect to the policy that the expert will demonstrate. Building on them, we develop provably efficient “on-average” algorithms for generalizing the observed expert’s behavior to new environments and constraints. Contributions. We summarize the main contributions of this work below: • We categorize existing IRL-based approaches for generalizing behavior into three groups, and we demonstrate that the most popular reward prior considered in the literature is biased (Section 3 ). • We propose novel unbiased reward priors for three common models of behavior (Section 4.1 ), and we derive closed-form expressions for their centroids restricted to the feasible sets (Section 4.2 ). • We develop provably efficient algorithms to estimate these centroids from offline data (Section 5 ). • Finally, we conduct numerical simulations to illustrate the relationship between the expert’s behavior and the behavior produced by our method (Section 6 ). Proofs are provided in Appendices C - F , and additional related work is discussed in Appendix A . 2 Preliminaries Notation. Given an integer N\in\mathbb{N} , we let \llbracket N\rrbracket\coloneqq\{1,2,\dotsc,N\} . Given two finite sets \mathcal{X},\mathcal{Y} , we denote by \Delta^{\mathcal{X}} and \Delta_{\mathcal{Y}}^{\mathcal{X}} , respectively, the set of probability measures on \mathcal{X} and the set of functions from \mathcal{Y} to \Delta^{\mathcal{X}} . For vectors x\in\mathbb{R}^{n} , we define \|x\|_{\infty}\coloneqq\max_{i}|x_{i}| . Given a matrix W , we let \text{det}(W) be its determinant. We use notation a\,\propto\,b to denote proportionality between two quantities a,b . Given a Riemannian manifold \mathcal{X}\subseteq\mathbb{R}^{n} with dimensionality k\leq n lee2019introduction , we denote the integral of a function f:\mathcal{X}\to\mathbb{R} over \mathcal{X} as \int_{\mathcal{X}}f(x)dx , where dx represents the k -dimensional volume element of the manifold. We denote by \text{vol}(\mathcal{X})\coloneqq\int_{\mathcal{X}}dx the k -dimensional volume of \mathcal{X} . The centroid C(\mathcal{X}) is a generalization of the arithmetic mean: C(\mathcal{X})\coloneqq\int_{\mathcal{X}}xdx/\text{vol}(\mathcal{X})\propto\int_{\mathcal{X}}xdx . Appendix B provides additional notation for the supplementary material. Markov Decision Processes (MDPs). An infinite-horizon discounted Markov Decision Process (MDP) puterman1994markov is defined

Read the complete text