R2BC: Multi-Agent Imitation Learning from Single-Agent Demonstrations

Connor Mattson, Varun Raveendra, Ellen Novoseller, Nicholas Waytowich, Vernon J. Lawhern, Daniel S. Brown · 2025

Read R2BC: Multi-Agent Imitation Learning from Single-Agent Demonstrations by Connor Mattson, Varun Raveendra, Ellen Novoseller, Nicholas Waytowich, Vernon J. Lawhern, Daniel S. Brown (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-18085.txt.

Introduction

Connor Mattson, with five co-authors, teaches a team of robots from lessons given to one robot at a time. Showing a whole team what to do together is hard for a single human teacher. Their round-robin cloning method lets the operator drive one agent, then the next, building team behavior step by step.

No demonstrations in the full joint action space are needed. In tests the method matched, and sometimes beat, a privileged cloning system trained on synced full-team data. Robot trainers get a practical way for one person to coach many machines without staging full-team demonstrations.

Catalog record

CREATORConnor Mattson, Varun Raveendra, Ellen Novoseller, Nicholas Waytowich, Vernon J. Lawhern, Daniel S. Brown
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-18085

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

R2BC: Multi-Agent Imitation Learning from Single-Agent Demonstrations Connor Mattson, Varun Raveendra, Ellen Novoseller, Nicholas Waytowich, Vernon J. Lawhern, Daniel S. Brown Abstract Imitation Learning (IL) is a natural way for humans to teach robots, particularly when high-quality demonstrations are easy to obtain. While IL has been widely applied to single-robot settings, relatively few studies have addressed the extension of these methods to multi-agent systems, especially in settings where a single human must provide demonstrations to a team of collaborating robots. In this paper, we introduce and study Round-Robin Behavior Cloning (R2BC), a method that enables a single human operator to effectively train multi-robot systems through sequential, single-agent demonstrations. Our approach allows the human to teleoperate one agent at a time and incrementally teach multi-agent behavior to the entire system, without requiring demonstrations in the joint multi-agent action space. We show that R2BC methods match—and in some cases surpass—the performance of an oracle behavior cloning approach trained on privileged synchronized demonstrations across four multi-agent simulated tasks. Finally, we deploy R2BC on two physical robot tasks trained using real human demonstrations. Anonymized videos, code, and supplemental materials can be found at https://sites.google.com/view/r2bc/home . I Introduction Imitation Learning (IL) has become a cornerstone of robot learning, enabling robot agents to mimic human demonstrations without explicitly defined reward functions. In particular, multi-agent IL holds the potential to improve robot learning in a variety of applications involving multi-robot teaming and path planning, such as search and rescue [ 26 ] , disaster response [ 11 ] , surveillance [ 30 ] , swarm robotics [ 18 ] , and warehouse automation [ 5 ] . However, while single-agent IL has seen widespread success, the same cannot yet be said for multi-agent systems, where cooperation, coordination, and partial observability introduce unique challenges that are not easily addressed by simply scaling existing techniques. Several studies have examined cases where multi-agent imitation learning could greatly improve solutions to real-world problems [ 2 , 10 , 4 ] . However, these works make strong assumptions about the type of demonstrations that can be provided to a group of agents, relying on coordinated and synchronous demonstrations in which all agents take correct actions simultaneously. Such synchronized demonstrations reduce the multi-agent IL problem to a single-agent problem in the joint action space of the agents. These methods are unrealistic, and usually implausible, for real world deployment, as human operators cannot reliably teleoperate multiple robots at the same time to accomplish complex tasks. In this paper, we address the following research question: How can we extend imitation learning to multi-agent systems when humans can only feasibly provide demonstrations to one agent at a time? Our work introduces a novel technique, R ound- R obin B ehavior C loning ( R2BC ) (Fig. ), which enables a human demonstrator to provide single-agent demonstrations by cycling through the agents, while at any given time, all non-teleoperated agents execute their current learned policies. The key idea is that demonstrations can be individually provided to agents online to iteratively improve a multi-agent policy, achieving parity with, or outperforming, the performance of traditional IL methods that require centralized demonstrations. The contributions of our work are as follows. • We introduce a novel problem setting, multi-agent imitation learning from single-agent demonstrations , with the objective of training a collective policy using only individual agent demonstrations. • We extend single-agent behavior cloning to this multi-agent problem setting by proposing Round-Robin Behavior Cloning (R2BC), which trains a policy that imitates the demonstrations while removing the unrealistic assumption in prior work of access to coordinated demonstrations. • We show that R2BC is able to match or exceed the performance of behavior cloning trained on joint action coordinated demonstrations in 4 simulated multi-agent domains using a synthetic demonstrator. • We deploy R2BC in navigation and block pushing physical robot tasks using real human demonstrations and demonstrate that R2BC outperforms centralized BC by 3.25x and 5.9x on the two tasks, respectively. To the best of our knowledge, we are the first to propose and deploy a behavior cloning method for multi-agent systems that learns solely from online single-agent demonstrations. II Related Work II-A Multi-Robot Teleoperation Providing demonstrations to a team of coordinating robots is a challenging research problem, as the number of degrees of freedom (DoF) required to control the entire system often exceeds what one human can control at once. Existing methods for teleoperating multi-agent systems include having a single human operator manually switch between robots [ 9 ] —e.g. at any given time, all but one robot receive no-op actions—or having a team of human operators who each control a different individual agent [ 24 ] . Recent studies have shown that learned models can utilize low-dimensional control inputs to control high-DoF systems such as swarms [ 36 ] and manipulators [ 20 ] . While these models can effectively allow humans to control complex systems (and therefore provide demonstrations), they are often trained on task-specific data and are unlikely to generalize to unseen tasks, making them a costly approach to providing demonstrations in deployed settings. Instead, we consider the problem setting in which a human can control a subset of the control parameters, corresponding to the control of an individual agent, for each demonstration. Similarly, HiTAB (Hierarchical Training of Agent Behaviors) [ 33 , 21 ] provides an approach to multi-agent demonstration collection that learns a set of atomic “skills” that a human can use as a discrete action set to control a group of heterogeneous agents. In this work, we assume that no preconceived skills library is available, and we provide demonstrations directly in the agent’s action space. II-B Multi-Agent Imitation Learning (MAIL) The use of imitation learning (IL) to train a policy that mimics an expert task demonstrator has been widely studied in the single-agent literature [ 29 , 12 , 28 , 7 ] . Extensions to multi-agent systems, namely Multi-Agent Imitation Learning (MAIL) , have shown that policies can be learned from demonstrations to solve problems in grid energy management [ 10 ] , autonomous vehicle control [ 4 , 14 ] , and multi-agent path finding [ 2 ] . Current approaches successfully extend IL to the multi-agent setting, e.g. by modeling the spatial-temporal relationship between agents in a Graph Neural Network [ 38 , 19 ] , modeling latent structures of cooperation [ 17 ] , modeling individual agent deviations from an optimized coordinated policy [ 35 ] , combining LfD with RL [ 14 , 25 ] , leveraging large transformers [ 2 ] , and extending single-agent IL methods such as adversarial IL [ 4 , 32 , 8 ] and inverse soft Q-learning [ 22 ] to the multi-agent regime. Importantly, each of these proposed prior solutions involves training a policy from joint demonstrations , where expert trajectories involve all N agents working together simultaneously in noisily-optimal demonstrations. However, to obtain a dataset of joint demonstrations [ 2 ] , one must assume unrealistic access to either synchronized human demonstrations for all agents or a pretrained RL demonstrator policy [ 38 ] , planner [ 18 , 6 , 34 ] , or model for simulating expert behavior [ 38 ]

Read the complete text