Swarm Behavior Cloning
Jonas Nüßlein 1 , Maximilian Zorn 1 , Philipp Altmann 1 and Claudia Linnhoff-Popien 1 1 Institute of Computer Science, LMU Munich, Germany [email protected] · 2024
Read Swarm Behavior Cloning by Jonas Nüßlein 1 , Maximilian Zorn 1 , Philipp Altmann 1 and Claudia Linnhoff-Popien 1 1 Institute of Computer Science, LMU Munich, Germany [email protected] (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-07617.txt.
Introduction
Jonas Nüßlein and co-authors improve imitation learning for groups of agents in a 2024 paper. Behavior cloning trains an agent to copy expert examples. Using several trained copies together, as an ensemble, adds robustness, but the copies can drift apart in rare states and their averaged action can turn poor.
Their method, called swarm behavior cloning, nudges the ensemble members to agree more while keeping their inner workings diverse. Across eight test environments, the approach cut the spread between members' actions a great deal. It also raised overall returns notably. The study matters for robot and game agents that learn from demonstrations, where rare states are common and a single averaged action must stay sensible.
Catalog record
| CREATOR | Jonas Nüßlein 1 , Maximilian Zorn 1 , Philipp Altmann 1 and Claudia Linnhoff-Popien 1 1 Institute of Computer Science, LMU Munich, Germany [email protected] |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-07617
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Swarm Behavior Cloning Jonas Nüßlein 1 , Maximilian Zorn 1 , Philipp Altmann 1 and Claudia Linnhoff-Popien 1 1 Institute of Computer Science, LMU Munich, Germany [email protected] Abstract In sequential decision-making environments, the primary approaches for training agents are Reinforcement Learning (RL) and Imitation Learning (IL). Unlike RL, which relies on modeling a reward function, IL leverages expert demonstrations, where an expert policy π e \pi_{e} (e.g., a human) provides the desired behavior. Formally, a dataset D D of state-action pairs is provided: D = ( s , a = π e ( s ) ) D={(s,a=\pi_{e}(s))} . A common technique within IL is Behavior Cloning (BC), where a policy π ( s ) = a \pi(s)=a is learned through supervised learning on D D . Further improvements can be achieved by using an ensemble of N N individually trained BC policies, denoted as E = { π i ( s ) } 1 ≤ i ≤ N E={\{\pi_{i}(s)\}}_{1\leq i\leq N} . The ensemble’s action a a for a given state s s is the aggregated output of the N N actions: a = 1 N ∑ i π i ( s ) a=\frac{1}{N}\sum_{i}\pi_{i}(s) . This paper addresses the issue of increasing action differences—the observation that discrepancies between the N N predicted actions grow in states that are underrepresented in the training data. Large action differences can result in suboptimal aggregated actions. To address this, we propose a method that fosters greater alignment among the policies while preserving the diversity of their computations. This approach reduces action differences and ensures that the ensemble retains its inherent strengths, such as robustness and varied decision-making. We evaluate our approach across eight diverse environments, demonstrating a notable decrease in action differences and significant improvements in overall performance, as measured by mean episode returns. 1 INTRODUCTION Reinforcement Learning (RL) is a widely recognized approach for training agents to exhibit desired behaviors by interacting with an environment over time. In RL, an agent receives a state s , selects an action a , and receives feedback in the form of a reward, learning which behaviors are beneficial (high reward) and which are detrimental (low reward) based on the reward function provided by the environment [ Sutton and Barto, 2018 ] . While RL has shown great promise, designing an effective reward function can be highly challenging. A well-designed reward function must satisfy several criteria: (1) the optimal behavior should yield the maximum possible return R^{*} (the sum of all rewards in an episode); (2) suboptimal behaviors must be penalized, resulting in a return R<R^{*} , ensuring that shortcuts or unintended strategies are discouraged; (3) the reward function should be dense, providing informative feedback at every step of an episode rather than just at the end; (4) the reward should support gradual improvement, avoiding overly sparse rewards such as those that assign 1 to the optimal trajectory and 0 to all others, which can hinder exploration and learning [ Eschmann, 2021 , Knox et al., 2023 ] . Due to the inherent complexity of crafting such reward functions, the field of Imitation Learning (IL) has emerged as an alternative approach [ Zheng et al., 2021 , Torabi et al., 2019b ] . Instead of relying on an explicitly defined reward function, IL uses expert demonstrations to model the desired behavior. This paradigm has proven effective in various real-world applications, such as autonomous driving [ Bojarski et al., 2016 , Codevilla et al., 2019 ] and robotics [ Giusti et al., 2015 , Finn et al., 2016 ] . A prominent method within IL is Behavior Cloning (BC), where supervised learning is applied to a dataset of state-action pairs D=\{(s,a=\pi_{e}(s))\} , provided by an expert policy \pi_{e} . Compared to other IL methods like Inverse Reinforcement Learning [ Zhifei and Meng Joo, 2012 , Nüßlein et al., 2022 ] or Adversarial Imitation Learning [ Ho and Ermon, 2016 , Torabi et al., 2019a ] , BC has the advantage of not requiring further interactions with the environment, making it particularly suitable for non-simulated, real-world scenarios. A straightforward extension of BC is the use of an ensemble of N individually trained policies. In this approach, the ensemble action is computed by aggregating the N predicted actions as a=\frac{1}{N}\sum_{i}\pi_{i}(s) . Although ensemble methods often improve robustness, they can encounter challenges when states in the training data D are underrepresented. For these states, the N policies may predict actions \{a_{i}\}_{1\leq i\leq N} that diverge significantly, leading to suboptimal aggregated actions. Figure 1: This figure visualizes schematically the predicted actions of three different Behavior Cloning approaches, represented as black dots, in a 2-dim action space for some state s_{t} . The heatmap represents the Q-values Q(a_{t},s_{t}) . (Left) the left plot shows plain Behavior Cloning . A policy \pi was trained using supervised learning on some training data D . The black dot is the predicted action a_{t}=\pi(s_{t}) . (Middle) in Ensemble Behavior Cloning an ensemble of N policies is trained individually on D . The N predicted actions \{a_{i}=\pi_{i}(s_{t})\} (gray dots) are then aggregated to the ensemble action (black dot). (Right) in our approach Swarm Behavior Cloning an ensemble of N policies is trained as well. However, they are not trained individually but using a modified loss function, see formula (2). The effect is a smaller difference between the N predicted actions, resembling a swarm behavior. Similar to Ensemble Behavior Cloning the ensemble action (black dot) is then aggregated from the N predicted actions (gray dots). In this paper, we address the problem of increasing action differences in such underrepresented states. Specifically, we propose a new loss function that encourages greater alignment among the N policies in the ensemble while preserving the diversity of their computations. This approach reduces action differences and ensures that the ensemble retains its inherent strengths, such as robustness and varied decision-making. As illustrated in Figure 1, this approach—termed Swarm Behavior Cloning —leads to more consistent predictions across the ensemble, with the N predicted actions (gray dots) clustered more closely together compared to standard Ensemble Behavior Cloning (middle plot). By minimizing action divergence, our approach improves the quality of the aggregated action and enhances performance in diverse environments. 2 BACKGROUND 2.1 Reinforcement Learning Reinforcement Learning (RL) problems are often modeled as Markov Decision Processes (MDP). An MDP is represented as a tuple M=\langle S,A,T,r,p_{0},\gamma\rangle where S is a set of states, A is a set of actions, and T(s_{t+1}\>|\>s_{t},a_{t}) is the probability density function (pdf) for sampling the next state s_{t+1} after executing action a_{t} in state s_{t} . It fulfills the Markov property since this pdf solely depends on the current state s_{t} and not on a history of past states s_{\tau<t} . r:S\times A\rightarrow\mathbb{R} is the reward function, p_{0} is the start state distribution, and \gamma\in[0;1) is a discount factor which weights later rewards less than earlier rewards [ Phan et al., 2023 ] . A deterministic policy \pi:S\rightarrow A is a mapping from states to actions. Return R=\sum_{t=0}^{\infty}{\gamma}^{\;t}\cdot r(s_{t},a_{t}) is the (discounted) sum of all rewards within an episode. The task of RL is to learn a policy such that the expected cumulative return is maximized: \pi^{*}=\underset{\pi}{\mathrm{argmax}}\>\>J_{p_{0}}(\pi,M)=\underset{\pi}{\mathrm{argmax}}\>\>\mathbb{E}\>\Biggl[\sum_{t=0}^{\infty}{\gamma}^{\;t}\cdot r(s_{t},a_{t})\>|\>\pi\Biggr]