Imitation Learning via Focused Satisficing
Rushit N. Shah, Nikolaos Agadakos, Synthia Sasulski, Ali Farajzadeh, Sanjiban Choudhury, Brian Ziebart · 2025
Read Imitation Learning via Focused Satisficing by Rushit N. Shah, Nikolaos Agadakos, Synthia Sasulski, Ali Farajzadeh, Sanjiban Choudhury, Brian Ziebart (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-14820.txt.
Introduction
Rushit Shah, Nikolaos Agadakos, Synthia Sasulski, and three co-authors rethink how machines learn by copying in this two thousand twenty-five paper. The standard view holds that demos are near best under some hidden cost rule. Satisficing theory says people pick acts that clear their own goal bar. That bar can shift. A moon landing that works, but is slow and jerky, may be fine for a new hand.
Their focused satisficing method uses a margin-based goal to guide deep reinforcement learning. The learned plan aims to beat the goal bar of the person who gave the demos. It does so on new demos, and never spells out that bar in a model. Tests showed it copies the best parts of demos better than past methods. It also won higher rates of acts sure to be fine for the demonstrator. Its true returns held up across test worlds.
Catalog record
| CREATOR | Rushit N. Shah, Nikolaos Agadakos, Synthia Sasulski, Ali Farajzadeh, Sanjiban Choudhury, Brian Ziebart |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 72 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-14820
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Imitation Learning via Focused Satisficing Rushit N. Shah, Nikolaos Agadakos, Synthia Sasulski, Ali Farajzadeh, Sanjiban Choudhury, Brian Ziebart Abstract Imitation learning often assumes that demonstrations are close to optimal according to some fixed, but unknown, cost function. However, according to satisficing theory , humans often choose acceptable behavior based on their personal (and potentially dynamic) levels of aspiration , rather than achieving (near-) optimality. For example, a lunar lander demonstration that successfully lands without crashing might be acceptable to a novice despite being slow or jerky. Using a margin-based objective to guide deep reinforcement learning, our focused satisficing approach to imitation learning seeks a policy that surpasses the demonstrator’s aspiration levels—defined over trajectories or portions of trajectories—on unseen demonstrations without explicitly learning those aspirations . We show experimentally that this focuses the policy to imitate the highest quality (portions of) demonstrations better than existing imitation learning methods, providing much higher rates of guaranteed acceptability to the demonstrator, and competitive true returns on a range of environments. † † Accepted for publication at the 34th International Joint Conference on Artificial Intelligence (IJCAI 2025). 1 Introduction Hand-engineered policies and reinforcement-learned policies from hand-specified cost functions often fail to perform adequately in complicated tasks of interest (e.g., self-driving). Prevalent imitation learning approaches Osa et al. (2018) address this issue either by directly mimicking human demonstrations via behavioral cloning Pomerleau (1991) or by estimating reward functions that rationalize demonstrator behavior Ng and Russell (2000) ; Abbeel and Ng (2004) —both under the assumption that the demonstrator is (near) optimal. The many advantages autonomous systems have over human actors, including faster reaction time Whelan (2008) , more precise control Ladha et al. (2023) , increased rationality, and lossless memory Miller (1956) , can violate this assumption and lead to potential value misalignment Amodei et al. (2016) between demonstrator and imitator. New perspectives are needed to train more capable imitation learners from less capable demonstrators without supplemental annotations Christiano et al. (2017) ; Brown et al. (2019) ; Rafailov et al. (2024) or assuming some expert-level demonstrations being available Tangkaratt et al. (2021) . Figure 1: Left: Pareto-dominating in the cost function bases ( {\textrm{f}}_{1} , {\textrm{f}}_{2} ) of acceptable behavior (purple: imitator acceptable set ) guarantees the imitator is acceptable to the demonstrator (red: demonstrator acceptable set ). Right: The subdominance (orange lines) measures how far imitator trajectory rollouts are from guaranteed acceptance (by a margin). Figure 2: Existing reward-based imitation methods, e.g., TREX Brown et al. (2019) , seek to outperform demonstrations using a pipeline of engineered components (top) to first segment trajectories into “snippets,” and to ultimately estimate a reward function that is then optimized using reinforcement learning. Our approach (bottom) uses the subdominance as the reinforcement learning objective, which is defined by the relative performance of the imitator compared to the demonstrations in each cost feature. This effectively uses feedback from the learned imitator policy to guide additional reinforcement learning without an explicit reward function. When faced with challenging decision tasks, satisficing theory Simon (1956) suggests that demonstrators produce behavior that is acceptable rather than (near) optimal. By viewing imitation learning through this lens, we aim for imitator behavior that is similarly acceptable to the demonstrator, despite never knowing the demonstrator’s precise acceptability criteria (Figure 1 , left)—working instead with an assumed class of cost functions that defines it. To pursue this aim, we develop Min imally Sub dominant F ocused I mitation ( MinSubFI ), which employs the subdominance Ziebart et al. (2022) , a margin-based measure of insufficiency (i.e., the distance from guaranteeing imitator-acceptability by a margin), as a training objective for policy gradient optimization (Figure 1 , right). This produces policies that are maximally acceptable rather than reward-maximizing. Compared to existing inverse reward learning methods Brown et al. (2019) ; Burchfiel et al. (2016) ; Wirth et al. (2017) ; Wu et al. (2019) ; Chen et al. (2020) ; Zhang et al. (2021) , which are highly reliant on an estimated scalar reward function to guide reinforcement learning (e.g., using the pipeline of engineered components in Figure 2 , top), our approach more directly optimizes the imitator’s policy, enabling it to: • Learn context-sensitive policies without learning context-sensitive cost functions; • Ignore less optimal demonstrations without requiring explicit noise modeling; • Automatically select and learn from portions of trajectories (i.e., snippets) of high quality; and • Provide generalization guarantees for changing acceptability (e.g., due to skill improvement or fatigue). Under the MinSubFI objective, many of the same engineered components of existing approaches (Figure 2 , top) are jointly optimized in a unified manner (Figure 2 , bottom). We evaluate the benefits of MinSubFI on imitation learning tasks using human and synthetic demonstrations with both engineered and learned cost features. 2 Satisficing Demonstrations & Policy Gradient Subdominance Minimization We now formally recast imitation learning through the lens of satisficing theory . Under this perspective, policies are learned from demonstrations that are acceptable , according to an unknown acceptability set, rather than near-optimal . We broadly define this notion of acceptability over trajectories and trajectory “snippets” (i.e., portions of trajectories), and develop new imitation learning methods that are designed to be performant with respect to the demonstrators’ unknown acceptability sets in both theory and practice. 2.1 Imitation Learning Problem Setting We consider the imitation learning Osa et al. (2018) task of producing a policy \hat{\pi} based on demonstrated trajectories of states and actions, \tilde{\xi}=(\tilde{s}_{1},\tilde{a}_{1},\tilde{s_{2}},\ldots,\tilde{s}_{T}) . Demonstrations are produced from a task-indexed Markov decision process (MDP), \mathcal{M}=(\mathcal{S},\mathcal{A},\{\tau_{i}\},C) , characterized by states \mathcal{S} , actions \mathcal{A} , state transition probability distributions \tau_{i}:\mathcal{S}\times\mathcal{A}\rightarrow\Delta_{\mathcal{S}} (with \Delta representing a probability simplex), and a cost function C:\mathcal{S}\rightarrow\mathbb{R}_{\geq 0} . The state transition probability distribution is defined for s=a=\varnothing to provide an initial state distribution. Each state transition probability distribution, \tau_{i} , corresponds to a different task i that shares the same state-action space, but may have different initial states, different absorbing (goal) states, or different dynamics more generally. We use \tilde{\xi}_{i,j} to denote the j {}^{\text{th}} demonstration for the i {}^{\text{th}} task, \tilde{\Xi} to denote the set of all demonstrations, and \tilde{\Xi}_{i} to denote the set of demonstrations corresponding to task i . The cost/reward function is unavailable to the imitator (providing at most \mathcal{M}\backslash C ), distinguishing imitation learning from (offline) reinforcement learning Levine et al. (2020) . 2.2 Satisficing Perspective of Demonstrations According to satisficing theory Simon (1956) , when faced with challenging decision tasks, humans tend to prioritize behaviors that are acceptable to them rather than striving for optimality. This implies that demonstrated behavior is selected to be acceptable , according to some aspirational criteria of the demonstrator, rather than being (near) optimal . Definition 1 . Trajectory \xi satisfices (or is acceptable ) for a particular aspiration , defined by ({\bf w},\nu,t,t^{\prime}) if and only if it is less costly than the aspirational threshold \nu evaluated using the cost function parameterized by {\bf w} : \text{cost}_{\bf w}(\xi_{t:t^{\prime}})<\nu . It satisfices the aspiration/acceptability set \Omega=\{({\bf w},\nu,t,t^{\prime})\} , i.e., \xi\in\text{Satisf}_{\Omega} , if and only if \xi satisfices each aspiration in \Omega . Note that the aspiration set can be context-dependent and vary for each demonstration. For example, it may change with the growing experience