Can Optimal Transport Improve Federated Inverse Reinforcement Learning?

David Millard, Ali Baheri · 2026

Read Can Optimal Transport Improve Federated Inverse Reinforcement Learning? by David Millard, Ali Baheri (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-00309.txt.

Introduction

David Millard and coauthor Ali Baheri ask whether optimal transport can fuse reward lessons from a fleet of robots. Each agent works in a slightly different setting. All chase one shared high level goal. Pooling raw data is often blocked in practice. Dynamics differ, privacy binds, and bandwidth runs thin. Inverse reinforcement learning must therefore stay federated.

Each client runs a light maximum entropy learner at home. It infers a local reward function from behavior. Those functions then merge through a Wasserstein barycenter. The merge respects the geometry of the reward space. The authors prove it beats plain parameter averaging. Their estimate stays more faithful to the true shared reward. Communication stays light throughout the process. The frame suits fleets that cannot share raw logs. Heterogeneous agents gain one reward that travels across settings.

Catalog record

CREATORDavid Millard, Ali Baheri
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-00309

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Can Optimal Transport Improve Federated Inverse Reinforcement Learning? David Millard, Ali Baheri Abstract In robotics and multi-agent systems, fleets of autonomous agents often operate in subtly different environments while pursuing a common high-level objective. Directly pooling their data to learn a shared reward function is typically impractical due to differences in dynamics, privacy constraints, and limited communication bandwidth. This paper introduces an optimal transport–based approach to federated inverse reinforcement learning (IRL). Each client first performs lightweight Maximum Entropy IRL locally, adhering to its computational and privacy limitations. The resulting reward functions are then fused via a Wasserstein barycenter, which considers their underlying geometric structure. We further prove that this barycentric fusion yields a more faithful global reward estimate than conventional parameter averaging methods in federated learning. Overall, this work provides a principled and communication-efficient framework for deriving a shared reward that generalizes across heterogeneous agents and environments. \coltauthor Department of Mechanical Engineering, Rochester Institute of Technology, Rochester, NY, USA keywords Inverse Reinforcement Learning, Federated Learning, Optimal Transport 1 Introduction With the growing demand for personalized robotics, it has become increasingly important to enable experience sharing that accelerates learning while preserving privacy. These robots often operate under expert policies in environments whose dynamics evolve over time—for instance, a household vacuum robot adapting to a new layout or a self-driving car navigating an unfamiliar city. Such shifts alter the underlying transition dynamics, yet the robot’s performance should remain robust to these changes. Moreover, within a network of robots, it is likely that another agent has already adapted to a similar environment. However, because each robot operates in a private domain, it is essential to prevent any information leakage about an individual client’s environment. Finally, many robotic platforms are designed primarily for inference or lightweight fine-tuning and therefore lack the computational resources to perform full-scale IRL to recover optimal reward parameters for their specific operating conditions. This setting motivates the problem of federated IRL, the ability to share and aggregate learned reward functions across multiple agents in a privacy-preserving manner. Many recent works have begun to explore federated variants of inverse reinforcement learning. In many approaches, model fusion is achieved through direct parameter averaging 15 , without accounting for the geometric structure of the underlying reward landscape. While these averaging schemes are simple and communication-efficient, they often yield suboptimal or unstable global rewards when local parameters are poorly fitted, an expected outcome for clients with limited computational resources or insufficient expert demonstrations. In highly heterogeneous settings, such instability can propagate through aggregation, leading to inconsistent or even unsafe reward estimates that degrade overall system performance. Moreover, the resulting global reward may fail to preserve consistent behavioral semantics across clients operating under distinct dynamics. These limitations motivate the development of a geometry-aware fusion mechanism that produces faithful global rewards even under noisy, biased, or undertrained local estimates. To address these challenges, we treat each locally learned reward in the discrete setting as what it parameterizes: a function over a shared state–action support. We explicitly evaluate this function on the common lattice, apply a shift-and-normalize step to obtain a probability distribution of reward mass, and then fuse clients by computing an entropically regularized Wasserstein barycenter over these normalized measures. This geometry-aware fusion preserves spatial structure, dampens underfit artifacts from weak clients, and yields a stable, semantically consistent global reward under heterogeneity. Finally, we project the fused measure back to parameters via least squares in the shared feature basis, enabling lightweight clients to contribute without sharing trajectories or destabilizing the model. Our contributions are as follows: 1. We introduce a federated inverse reinforcement learning framework that fuses learned reward functions using an entropically regularized Wasserstein barycenter. 2. We provide a theoretical analysis establishing stability and parameter-error bounds for the proposed barycentric aggregation, showing that it contracts toward the true reward function under bounded local estimation error. 3. We demonstrate through empirical studies on discrete and continuous-control benchmarks that barycentric reward fusion consistently yields more stable and transferable global rewards than parameter averaging. 2 Related Work Learning reward functions across multiple agents with heterogeneous dynamics has been explored from several perspectives. In federated and distributed IRL, prior work has developed decentralized optimization schemes and convergence guarantees for recovering rewards without centralizing expert trajectories 9 ; 5 ; 10 . However, these approaches typically retain the standard parameter-averaging server update from conventional federated learning, focusing primarily on modifying local optimization steps rather than the aggregation mechanism itself. Beyond purely federated settings, multi-task and transfer-style IRL methods link related tasks to share statistical strength, for example, through maximum causal entropy formulations, teacher–student architectures, and group learning 8 ; 18 ; 13 . In contrast, our formulation preserves the privacy and communication constraints central to federated learning while explicitly addressing heterogeneity in transition dynamics and demonstration distributions, as well as hardware limitations that can lead to underfitted local reward parameters. Classical IRL methods include early formulations of the inverse problem 17 , apprenticeship learning by feature expectation matching 1 , maximum entropy (MaxEnt) IRL 25 , and adversarial variants such as GAIL 11 . We adopt a MaxEnt local learner for its robustness and well-studied gradients 23 . Moreover, MaxEnt IRL integrates with the linear function approximation, which we employ during experimentation to maintain meaningful support while keeping computational complexity manageable 24 . Although our framework is designed to be lightweight, we explicitly assume a limited computational budget on the client side, consistent with prior work highlighting the resource constraints of embedded and mobile robotic platforms 16 ; 12 . In the broader federated learning literature, FedAvg remains the canonical baseline 15 , and, to the best of our knowledge, no prior work has proposed an alternate aggregation technique when federating IRL. Optimal transport provides a geometry-aware framework for comparing and aggregating distributions. Entropic regularization enables efficient computation through the Sinkhorn algorithm 6 , and the Wasserstein barycenter defines a principled notion of averaging over probability measures 2 . Recent work has applied optimal transport to reinforcement learning, using it to analyze reward ambiguity in IRL 3 and to develop consensus or policy update mechanisms in multi-agent RL based on Wasserstein distance 19 ; 4 ; 22 ; 21 . In this work, we compute a Wasserstein barycenter over locally learned reward functions to obtain a fused reward that remains interpretable, privacy-preserving, and robust to training inconsistencies. 3 Preliminaries & Problem Formulation This section presents the components of our method and defines the problem domain it addresses. 3.1 Inverse Reinforcement Learning IRL seeks to recover the underlying reward function that explains observed expert behavior in a Markov Decision Process (MDP). Given an MDP \mathcal{M}=(\mathcal{S},\mathcal{A},P,\gamma) with an unknown reward function R^{*}:\mathcal{S}\times\mathcal{A}\to\mathbb{R} and expert demonstrations \mathcal{D}^{E}=\{\tau_{j}\}_{j=1}^{N} where each trajectory \tau_{j}=(s_{0},a_{0},\dots,s_{T},a_{T}) represents a sequence of state–action pairs, the goal is to infer a reward function \hat{R} such that the optimal policy under \hat{R} reproduces the expert’s behavior. Formally, IRL seeks \hat{R} satisfying \displaystyle\pi_{\hat{R}}^{*}=\argmax_{\pi}\;\mathbb{E}_{\pi,P}\!\left[\sum_{t=0}^{T}\gamma^{t}\hat{R}(s_{t},a_{t})\right],\quad\text{with}\quad\pi_{\hat{R}}^{*}\approx\pi_{R^{*}}^{*}. (1) In our work, we seek to find a linear function approximation such that \hat{R}_{\theta}(s_{t},a_{t})=\theta^{\top}\phi(s_{t},a_{t}), where \phi(s,a) denotes the feature vector and \theta the reward parameters. MaxEnt IRL models the trajectory distribution as \displaystyle p(\tau\,|\,\theta)=\frac{1}{Z(\theta)}\exp\!\left(\sum_{t=0}^{T}\gamma^{t}\theta^{\top}\phi(s_{t},a_{t})\right), (2) where Z(\theta) represents the partition function ensuring normalization. The objective maximizes the likelihood of

Read the complete text