An Optimal Discriminator Weighted Imitation Perspective for Reinforcement Learning

Haoran Xu, Shuozhe Li, Harshit Sikchi, Scott Niekum, Amy Zhang · 2025

Read An Optimal Discriminator Weighted Imitation Perspective for Reinforcement Learning by Haoran Xu, Shuozhe Li, Harshit Sikchi, Scott Niekum, Amy Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-13368.txt.

Introduction

Haoran Xu and four co-authors introduce, in a 2025 paper, a method for offline reinforcement learning called Iterative Dual Reinforcement Learning, or IDRL. In offline learning, an agent must learn from a fixed dataset rather than from new experience. The method grew out of an experiment in which a discriminator trained on offline data plus expert data gave strong results when its scores weighted behavior cloning.

The authors argue that existing Dual-RL methods estimate the key visitation distribution ratio incorrectly, so their weights differ from the optimal discriminator weights. IDRL corrects this step by step. Each iteration removes transitions with zero weight, as judged by the previous ratio, and reruns Dual-RL on the remaining data. Across D4RL datasets and corrupted human demonstrations, IDRL outperformed strong Primal-RL and Dual-RL baselines in both performance and stability.

Catalog record

CREATORHaoran Xu, Shuozhe Li, Harshit Sikchi, Scott Niekum, Amy Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-13368

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

An Optimal Discriminator Weighted Imitation Perspective for Reinforcement Learning Haoran Xu, Shuozhe Li, Harshit Sikchi, Scott Niekum, Amy Zhang Abstract We introduce Iterative Dual Reinforcement Learning (IDRL), a new method that takes an optimal discriminator-weighted imitation view of solving RL. Our method is motivated by a simple experiment in which we find training a discriminator using the offline dataset plus an additional expert dataset and then performing discriminator-weighted behavior cloning gives strong results on various types of datasets. That optimal discriminator weight is quite similar to the learned visitation distribution ratio in Dual-RL, however, we find that current Dual-RL methods do not correctly estimate that ratio. In IDRL, we propose a correction method to iteratively approach the optimal visitation distribution ratio in the offline dataset given no addtional expert dataset. During each iteration, IDRL removes zero-weight suboptimal transitions using the learned ratio from the previous iteration and runs Dual-RL on the remaining subdataset. This can be seen as replacing the behavior visitation distribution with the optimized visitation distribution from the previous iteration, which theoretically gives a curriculum of improved visitation distribution ratios that are closer to the optimal discriminator weight. We verify the effectiveness of IDRL on various kinds of offline datasets, including D4RL datasets and more realistic corrupted demonstrations. IDRL beats strong Primal-RL and Dual-RL baselines in terms of both performance and stability, on all datasets. Project page at https://ryanxhr.github.io/IDRL/ . 1 Introduction Offline reinforcement learning (RL) has attracted attention as it enables learning policies by utilizing only pre-collected data. It is a promising area for bringing RL into real-world domains, such as industrial control ( Zhan et al., 2022 ) and robotics ( Kalashnikov et al., 2021 ) . In such scenarios, arbitrary exploration with untrained policies is costly or dangerous, but sufficient prior data is available. The major challenge of offline RL is the distributional shift issue ( Levine et al., 2020 ) , i.e., the optimized policy’s distribution is different from the offline data distribution, which means the value function may have extrapolation errors to unseen actions produced by the optimized policy ( Fujimoto et al., 2018 ) . To deal with this challenge, most offline RL algorithms build on the primal form of RL (i.e., maximizing a value function respect to the policy, which needs to alternate between policy evaluation and policy improvement), and impose an additional behavior constraint either by policy constraint, which constrains the policy to be close to the behavior policy using some distance function ( Wu et al., 2019 ; Fujimoto & Gu, 2021 ) ; by value regularization, which directly modifies the value function to be pessimistic ( Kumar et al., 2020 ; Kostrikov et al., 2021a ) ; or by uncentainty estimation, which guides policy optimization to low-uncertainty regions ( An et al., 2021 ; Yu et al., 2020 ) . However, in Primal-RL based methods, the policy may still output out-of-distribution (OOD) actions if the policy constraint weight is not set properly, which will cause either inaccurate or over-constrained value estimation that results in suboptimal behavior. Notably, recently there’s also another line of in-sample based offline RL algorithms ( Xu et al., 2023 ; Mao et al., 2024a ; Mao et al., 2024b ) , which can be derived from the dual formulation of RL (i.e., maximizing the reward respect to the policy’s visitation distribution) ( Sikchi et al., 2023b ) . Different from the primal counterpart, Dual-RL based methods isolate the learning of value function and policy; the value function can be learned using only dataset actions, which brings minimal estimation errors. To extract the implicit policy contained in the value function, one often uses the visitation distribution ratio between the optimized and behavior policy to do weighted behavior cloning (BC) ( Peng et al., 2019 ) . Although Dual-RL methods suffer minimal OOD errors by using in-sample learning, they often lag behind in performance of primal-RL methods. One conventional opinion is using weighted-BC style policy extraction causes poor generalization combined with its mode-covering nature ( Park et al., 2024 ) . In this paper, we aim to investigate pitfalls and push the limit of Dual-RL methods. We start from a surprising finding: learning the optimal visitation distribution ratio using a "gold" discriminator on the offline dataset and an additional expert dataset, and using this ratio as a filtering weight in weighted-BC outperforms all classical offline RL methods on various datasets, shown in Figure 1 . Although this discriminator-weighted BC is an oracle baseline as the expert datasets are unavailable, it points to the fact that Dual-RL methods, as a virtue of learning visitation distribution ratios, could faciliate this kind of dataset filtering and weighting. Given this potential, what is missing in current Dual-RL methods? 1) We show that current Dual-RL methods are unable to accurately estimate the state-action visitation ratio as a result of gradient update strategies they use for minimizing the Bellman residual. We find using a semi-gradient update in Dual-RL changes the fixed-point of optimization, learning an action distribution ratio rather than a visitation distribution ratio. Using the action distribution ratio to learn policies forces learning potentially suboptimal actions at states never visited by an expert, leading to poor generalization during weighted-BC policy extraction. 2) Irrespective of the gradient update strategy used, Dual-RL methods learn a regularized optimal visitation distribution w.r.t offline data distribution (i.e., visitation that maximizes performance while staying close to offline data), as opposed to learning the optimal visitation distribution ratio. Our key contribution in this paper is to address both the limitations above and present a new offline RL algorithm that is able to break the regularization barrier. First, we propose a new objective for accurately learning the visitation distribution ratio using a two-stage procedure. After getting the action distribution ratio using semi-gradient Dual-RL , we adopt an off-policy evaluation approach that recovers the correct state-action visitation distribution ratio using the action distribution ratio. This allows decomposing the original tasks of obtaining visitation distribution ratio into two distinct, complementary, and easy to solve tasks. Second, we show that the regularized optimal visitation ratio has a sparse form (some weights are zero) ( Xu et al., 2023 ) . This property can be used to refine the offline dataset iteratively and gradually learn an optimal visitation ratio that moves closer to the optimal discriminator weight and can be used for learning a better policy by weighted behavior cloning. Our theoretical results justify our method by deriving a smaller optimality lower bound due to this curriculum-refined dataset filtering. (a) Framework of Iterative Dual-RL (b) Results of Optimal-DWBC Figure 1: (a) Illustration of our proposed IDRL framework. IDRL breaks the regularization barrier by performing imitation learning on a iteratively-refined dataset, it can solve hard tasks where previous behavior-regularized offline RL can not do. For example, previous methods will fail at finding the shortest path from blue point to red point while crossing yellow point, due to non-uniform data coverage at different state. (b) Mean scores of optimal discriminator-weighted behavior cloning (Optimal-DWBC) on D4RL Mujoco-{m,m-r,m-e}, Antmaze-{all} and Kitchen-{all} datasets. We train a discriminator d on offline dataset \mathcal{D} and an additional expert dataset \mathcal{D}_{E} , we then use w_{d}(s,a)=\frac{d^{E}(s,a)}{d^{\mathcal{D}}(s,a)}=\frac{d(s,a)}{1-d(s,a)} to do weighted-BC on \mathcal{D}

Read the complete text