Is Optimal Transport Necessary for Inverse Reinforcement Learning?

Zixuan Dong, Yumi Omori, Keith Ross · 2025

Read Is Optimal Transport Necessary for Inverse Reinforcement Learning? by Zixuan Dong, Yumi Omori, Keith Ross (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-06793.txt.

Introduction

Zixuan Dong, Yumi Omori, and Keith Ross ask, in a 2025 paper, what a reward signal really needs to teach an agent from a few expert demos. Many methods score a learner's path by how close it stays to the expert's path. The parts of that idea that do the real work, though, had not been tested in a clean way.

The team varied two design choices: how closeness to the expert is estimated, and how well steps line up in time. Across thirty-two benchmarks and three learning methods, closeness alone was enough for offline learning. A light touch of time matching helped a bit offline and mattered much more online or with several demos. Their advice is to keep reward design simple before adding complex machinery.

Catalog record

CREATORZixuan Dong, Yumi Omori, Keith Ross
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-06793

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Is Optimal Transport Necessary for Inverse Reinforcement Learning? Zixuan Dong, Yumi Omori, Keith Ross Abstract Inverse Reinforcement Learning (IRL) aims to recover a reward function from expert demonstrations. Recently, Optimal Transport (OT) methods have been successfully deployed to align trajectories and infer rewards. While OT-based methods have shown strong empirical results, they introduce algorithmic complexity, hyperparameter sensitivity, and require solving the OT optimization problems. In this work, we challenge the necessity of OT in IRL by proposing two simple, heuristic alternatives: (1) Minimum-Distance Reward, which assigns rewards based on the nearest expert state regardless of temporal order; and (2) Segment-Matching Reward, which incorporates lightweight temporal alignment by matching agent states to corresponding segments in the expert trajectory. These methods avoid optimization, exhibit linear-time complexity, and are easy to implement. Through extensive evaluations across 32 online and offline benchmarks with three reinforcement learning algorithms, we show that our simple rewards match or outperform recent OT-based approaches. Our findings suggest that the core benefits of OT may arise from basic proximity alignment rather than its optimal coupling formulation, advocating for reevaluation of complexity in future IRL design. 1 Introduction Inverse Reinforcement Learning (IRL) addresses the problem of recovering a reward function from expert demonstrations, such that the expert behavior is (approximately) optimal under the learned rewards ( Ng and Russell 2000 ) . This reward function can then be used to train agents that imitate the expert, even when explicit task rewards are unavailable. IRL is especially useful in applications where reward engineering is challenging or infeasible, such as robotics, healthcare, and human behavior modeling. A recent line of work proposes to cast the IRL reward labeling problem as one of trajectory alignment via Optimal Transport (OT) ( Luo et al. 2023 ; Fu et al. 2024 ) . In the offline version of the problem setting, the dataset consists of expert and non-expert trajectories, none of which are labeled with rewards. Recent OT-based methods estimate the underlying rewards for all the states in the dataset, and then apply an offline RL algorithm to find a policy. To estimate the rewards, the approach defines a cost between states, computes a coupling between expert and non-expert trajectories that minimizes the transport cost, and uses the resulting alignment to define proxy rewards. This strategy has demonstrated success across various offline and online RL benchmarks, and has motivated further refinements such as temporal masking, context-aware distances, and alignment constraints ( Wang et al. 2024a ; Bobrin et al. 2024 ) . Despite their empirical promise, OT-based methods introduce several layers of complexity. They require solving the OT optimization problem (e.g., via Sinkhorn iterations), tuning additional hyperparameters, and defining custom distance metrics. These concerns raise a natural but underexplored question: Is optimal transport actually necessary for IRL? Or, put differently, can we match its performance using simpler heuristic alternatives that avoid solving any optimization problem? In this paper, we take a minimalist view of reward design for IRL. Specifically, we propose two simple and interpretable reward functions: Minimum-Distance Reward , which assigns each non-expert state a reward based on its closest distance to the expert trajectory, ignoring any notion of time or ordering; and Segment-matching Reward , which introduces a lightweight temporal structure by dividing the expert trajectory into segments and assigning rewards based on proximity to temporally aligned expert segments. These two reward functions are conceptually simple, require no learned models or optimization steps, and can be computed with linear or near-linear time complexity. We perform a large-scale empirical study comparing these simple methods to both the original OT formulation and recent temporal extensions. Across 32 benchmarks and three downstream RL algorithms, along with both online and offline settings, we find that our simple methods consistently match or exceed the performance of OT-based approaches. In particular, the Segment-matching method achieves best results in several environments, and the Minimum-Distance method performs surprisingly well despite its complete lack of temporal modeling. Our results have two main implications. First, they question the perceived necessity of optimal transport in IRL, suggesting that much of its benefit can be attributed to proximity-based alignment rather than its mathematical structure. Second, they underscore the importance of evaluating new methods against simple but well-motivated baselines. 2 Preliminary A standard episodic finite-horizon Markov Decision Process (MDP) can be represented as a tuple: \mathcal{M}=(\mathcal{S},\mathcal{A},R,P,\rho_{0},\gamma,T), where \mathcal{S} and \mathcal{A} are the state and action spaces, R:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R} is the reward function, P:\mathcal{S}\times\mathcal{A}\rightarrow\Delta(\mathcal{S}) is the transition function, \rho_{0} is the initial state distribution, \gamma\in(0,1] is the discount factor, and T is the maximal episode length. The objective of an RL algorithm is to learn an optimal policy \pi^{*}(a|s) that maximizes the expected cumulative discounted reward: \pi^{*}=\argmax_{\pi\in\Pi}\mathbb{E}_{\pi}\left[\sum_{t=0}^{T-1}\gamma^{t}R(s_{t},a_{t})\right],s_{0}\sim\rho_{0},\;a_{t}\sim\pi(\cdot|s_{t}),\;s_{t+1}\sim P(\cdot|s_{t},a_{t}) where \Pi is the feasible policy set. Online and Offline RL In online RL, the agent interacts with the environment during training to collect trajectories and update its policy. At each step, the agent selects an action from the current policy, receives a reward, and observes the next state. This interaction allows the agent to continuously improve its policy towards the RL objective via trial and error. However, online interactions can be inefficient, costly and risky, making online RL learning not always feasible. Offline RL then aims to learn purely from the pre-collected trajectory dataset labeled by task rewards, and determine a policy which can achieve the largest possible cumulative reward when it is deployed to interact with the MDP ( Levine et al. 2020 ) . The methodology that we develop in this paper applies to both online and offline setups, and we will be providing experimental results for both. Inverse RL In many real-world scenarios, task-specific rewards are often inaccessible, nontrivial to define, or difficult to engineer. Different from offline RL problems, in Imitation Learning (IL), we learn from a dataset containing expert state-action pairs without reward signals. Specifically, in IL we have at our disposal a dataset \mathcal{D}_{e}=\{\tau_{i}^{e}\}_{i=1}^{K}=\{(s_{0}^{e},a_{0}^{e},s_{1}^{e},a_{1}^{e},\dots s_{T_{i}}^{e},a_{T_{i}}^{e})_{i}\}_{i=1}^{K} , which comprises K expert demonstrations generated by some unknown expert policy \pi_{e} . Approaches to IL problems can mainly be classified into two categories: (i) Behavior Cloning (BC) determines the policy by directly cloning the expert actions in the dataset; (ii) Inverse RL (IRL) infers a reward function R^{*} under which the underlying expert policy \pi_{e} generating the dataset is optimal: \mathbb{E}_{\pi_{e}}\left[\sum_{t=0}^{T-1}\gamma^{t}R^{*}(s_{t},a_{t})\right]\geq\mathbb{E}_{\pi}\left[\sum_{t=0}^{T-1}\gamma^{t}R^{*}(s_{t},a_{t})\right],\quad\forall\pi\in\Pi After learning a reward R^{*} , we can employ standard RL algorithms to find a policy that mimics the expert. IRL avoids manual reward design while enabling the agent to acquire high-performing behavior through demonstration-driven learning. In this paper, we will be using the IRL approach for imitation learning. 3 IRL via Optimal Transport We will first describe the offline version of the problem. Following the setup in Luo et al. 2023 , we assume we have access to a dataset D consisting of both expert and non-expert trajectories. The trajectories do not include a reward signal, but we are informed which of the trajectories are expert trajectories. 3.1 The Optimal Transport Reward Algorithm The Optimal Transport Reward (OTR) heuristic first applies an optimal transport formulation to assign rewards to all trajectories in D , and subsequently uses an offline reinforcement learning algorithm to derive a high-performing policy ( Luo et al. 2023 ) . For

Read the complete text