A Lecture Note on Offline RL and IRL Part II: Foundations of Inverse Reinforcement Learning and Dynamic Discrete Choice Models

Enoch Hyunwook Kang · 2026

Read A Lecture Note on Offline RL and IRL Part II: Foundations of Inverse Reinforcement Learning and Dynamic Discrete Choice Models by Enoch Hyunwook Kang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-30843.txt.

Introduction

Kang turns the usual reinforcement-learning question around. Instead of a known reward and a search for a good policy, he asks whether offline expert data can reveal the reward the expert was chasing. He first proves that structural econometricians studying dynamic discrete choice and machine learners studying entropy-regularized inverse learning work on the same model under different names. He then walks through classic identification and computation routes, with their limits, and modern adversarial, occupancy, and offline methods, closing with a gradient-based estimator for offline inverse learning.

Catalog record

CREATOREnoch Hyunwook Kang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE133840
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.30843

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Lecture Note on Offline RL and IRL Part II: Foundations of Inverse Reinforcement Learning and Dynamic Discrete Choice Models Enoch Hyunwook Kang Abstract In the forward reinforcement-learning problem, the reward is fixed and known; the learner is asked to find a good policy or value function. Here we turn the question around. Given offline data generated by an expert, can we recover the reward the expert was optimizing? This is the inverse reinforcement learning problem, and remarkably, two communities, structural econometricians studying dynamic discrete choice (DDC) and machine learners studying entropy-regularized IRL , have been working on exactly the same probabilistic model under different names. We begin by proving their equivalence. We then develop the classical identification result of 54 and the classical computational paradigms that grew out of it: Rust’s nested fixed-point algorithm ( 67 ) , the conditional-choice-probability approach of 34 , and the two temporal-difference approaches of 1 : linear semi-gradient TD and approximate value iteration. Each route has its limits: dimensionality, transition-kernel estimation, the deadly triad, or projected fixed-point bias. We then walk through the modern ML/IRL strand: adversarial IRL ( 27 ) , occupancy matching ( 33 ) , IQ-Learn ( 28 ) , and offline ML-IRL ( 88 ) , deriving each method’s actual objective and stating precisely what it does and does not identify. We close with the empirical-risk-minimization framework of 41 , which yields a gradient-based estimator for offline IRL/DDC. \DeclareDelimFormat nameyeardelim, 1 Why IRL and DDC? In the forward reinforcement-learning problem, the reward function is a fixed, known input. The world handed us r and asked us to learn Q^{\star} . In a great many real-world problems, however, r is exactly what we want to learn, and the closest thing we have to it is a log of decisions made by someone who knew what they were doing. • A labor economist watches workers choose between job offers and wants to recover the utility function that rationalizes their choices ( 55 ) . • A marketing analyst sees customers’ coupon redemptions and purchase histories, and wants to recover preferences in order to predict response to a new offer. • A health-economics researcher observes physicians’ treatment decisions and wants to quantify the implicit risk-benefit tradeoffs they are making. • A robotics or autonomous-driving engineer watches expert pilots fly a helicopter or expert drivers navigate traffic, and wants to extract a reward function that, when fed to a downstream RL algorithm, reproduces the expert’s behavior ( 10 ) . In each case the data is offline: a fixed log of state-action-(next-state) tuples from one or more experts, and the reward itself is never directly observed ( 49 ) . Two literatures have been chipping away at this problem for decades, under almost identical probabilistic structure but completely different vocabularies: • Dynamic discrete choice (DDC) models from structural econometrics ( 67 ; 68 ; 8 ) : the agent’s reward is observed plus a private idiosyncratic shock, generally Gumbel-distributed, which the econometrician integrates out. • Inverse reinforcement learning (IRL) with the entropy-regularized (maximum-entropy) variant of 24 ; 27 : the agent maximizes expected return plus an entropy bonus on its policy, producing the same softmax choice rule. Our first order of business (Section 2 ) is to prove these are the same model. Sections 3 – 5 then survey the methods each tradition developed, and Section 6 presents the unified ERM treatment of 41 that fixes the issues both traditions inherited. 2 Preliminaries This section sets up the model, defines the soft Bellman operator, proves its contraction, and, most importantly, proves the equivalence between the DDC and entropy-regularized IRL formulations. Readers familiar with standard MDP notation can skim the MDP setup, but should not skip Section 2.6 (the Gumbel-max identity) or Section 2.7 (the equivalence theorem). 2.1 Markov Decision Processes We consider an infinite-horizon discounted MDP \mathcal{M}=(\mathcal{S},\mathcal{A},P,\nu_{0},r,\beta) where \mathcal{S} is a (possibly continuous) state space, \mathcal{A} is a finite action space, P:\mathcal{S}\times\mathcal{A}\to\Delta(\mathcal{S}) is the transition kernel, \nu_{0}\in\Delta(\mathcal{S}) is the initial distribution, r:\mathcal{S}\times\mathcal{A}\to[-R_{\max},R_{\max}] is the (unknown) reward, and \beta\in(0,1) is the discount. A stationary Markov policy is a measurable map \pi:\mathcal{S}\to\Delta(\mathcal{A}) with \pi(a\mid s) the probability of a at s . We write \mathbb{P}_{\nu_{0},\pi} and \mathbb{E}_{\nu_{0},\pi} for the distribution and expectation over the resulting trajectory \tau=(s_{0},a_{0},s_{1},a_{1},\ldots) . The discounted state-action occupancy of \pi is d^{\pi}(s,a)\;:=\;(1-\beta)\sum_{t=0}^{\infty}\beta^{t}\,\mathbb{P}_{\nu_{0},\pi}(s_{t}=s,\,a_{t}=a), a probability measure on \mathcal{S}\times\mathcal{A} . The offline dataset is \mathcal{D}\;=\;\bigl\{(s_{i},a_{i},s_{i}^{\prime})\bigr\}_{i=1}^{N}, sampled iid from d^{\pi^{\star}} for a single expert policy \pi^{\star} . Note what is missing: we do not observe rewards in \mathcal{D} : only states, actions, and next states. That omission is precisely what makes the problem “inverse,” and it is the canonical setup of both DDC and IRL. 2.2 Function Spaces, Norms, Projections Let \mathcal{Q}:=\{Q:\mathcal{S}\times\mathcal{A}\to\mathbb{R}\;\text{measurable, bounded}\} . The sup-norm is \|Q\|_{\infty}:=\sup_{s,a}|Q(s,a)| . For \rho\in\Delta(\mathcal{S}\times\mathcal{A}) we write \|Q\|_{\rho}^{2}:=\mathbb{E}_{(s,a)\sim\rho}[Q(s,a)^{2}] and \langle Q_{1},Q_{2}\rangle_{\rho}:=\mathbb{E}_{\rho}[Q_{1}Q_{2}] , so L^{2}(\rho) is a Hilbert space. For a closed linear subspace \mathcal{F}\subseteq L^{2}(\rho) , the L^{2}(\rho) -projection is the unique element \Pi_{\mathcal{F},\rho}Q\in\mathcal{F} satisfying \|Q-\Pi_{\mathcal{F},\rho}Q\|_{\rho}=\inf_{f\in\mathcal{F}}\|Q-f\|_{\rho} . This restriction to closed linear subspaces matters: projections onto arbitrary closed nonconvex sets may fail to be unique. For closed linear subspaces, \Pi_{\mathcal{F},\rho} is a non-expansion: \|\Pi Q_{1}-\Pi Q_{2}\|_{\rho}\leq\|Q_{1}-Q_{2}\|_{\rho} for all Q_{1},Q_{2} . We will need this fact, both the projection itself and its non-expansion property, when analyzing temporal-difference methods in Section 4.4 . 2.3 The Dynamic Discrete Choice Formulation The DDC setup follows the Type 1 Extreme Value (T1EV) derivation in 41 . At each period the agent observes the deterministic flow payoff r(s_{t},a_{t}) plus an idiosyncratic alternative-specific shock \epsilon_{t,a_{t}} . We use the unit-scale Gumbel distribution with location parameter \delta : \displaystyle F_{\delta}(x) \displaystyle=\exp\!\bigl(-\exp(-(x-\delta))\bigr), (1) \displaystyle f_{\delta}(x) \displaystyle=\exp\!\bigl(-((x-\delta)+\exp(-(x-\delta)))\bigr), (2) \displaystyle\mathbb{E}[\epsilon] \displaystyle=\delta+\gamma_{E}=:\mu_{\epsilon}, (3) where \gamma_{E} is the Euler–Mascheroni constant. The mean-zero T1EV normalization used for the baseline DDC–MaxEnt equivalence is \delta=-\gamma_{E},\qquad\mu_{\epsilon}=0. The DDC agent maximizes \pi^{\star}\in\operatorname*{arg\,max}_{\pi}\;\mathbb{E}_{\nu_{0},\pi,G}\!\left[\sum_{t=0}^{\infty}\beta^{t}\bigl(r(s_{t},a_{t})+\epsilon_{t,a_{t}}\bigr)\right], (4) where the expectation averages over the Markov chain and the idiosyncratic shocks. The policy is induced by choosing the alternative with the largest shock-augmented choice-specific value. The derivation in Section 2.6 shows that, under the mean-zero normalization \delta=-\gamma_{E} , this DDC problem implies \displaystyle\pi^{\star}(a\mid s) \displaystyle=\frac{\exp(Q^{\star}(s,a))}{\sum_{a^{\prime}\in\mathcal{A}}\exp(Q^{\star}(s,a^{\prime}))}, (5) \displaystyle Q^{\star}(s,a) \displaystyle=r(s,a)+\beta\,\mathbb{E}_{s^{\prime}\sim P(\cdot\mid s,a)}\!\left[\log\sum_{a^{\prime}\in\mathcal{A}}\exp(Q^{\star}(s^{\prime},a^{\prime}))\right]. (6) If \delta\neq-\gamma_{E} , the same derivation gives the same softmax rule but adds the constant \beta(\delta+\gamma_{E}) to the right side of the Q Bellman equation. 2.4 The Entropy-Regularized IRL Formulation The MaxEnt-IRL setup in 41 instead treats the agent as choosing a stochastic stationary policy and explicitly rewards entropy. With regularization coefficient \lambda , the forward problem is \pi^{\star}\in\operatorname*{arg\,max}_{\pi}\;\mathbb{E}_{\nu_{0},\pi}\!\left[\sum_{t=0}^{\infty}\beta^{t}\bigl(r(s_{t},a_{t})+\lambda\operatorname{H}(\pi(\cdot\mid s_{t}))\bigr)\right], (7) where \operatorname{H}(\pi(\cdot\mid s)):=-\sum_{a}\pi(a\mid s)\log\pi(a\mid s) . Throughout the main text, and in the equivalence theorem below, we set \lambda=1, which is the entropy scale corresponding to unit-scale T1EV shocks. Section 2.6 derives the same softmax rule ( 5 ) and soft Bellman equation ( 6 ) from this entropy-regularized problem. 2.5 The Soft Bellman Operator Definition 2.1 (Soft Bellman operator) . For Q\in\mathcal{Q} , define (\mathcal{T}_{\mathrm{soft}}Q)(s,a)\;:=\;r(s,a)+\beta\,\mathbb{E}_{s^{\prime}\sim P(\cdot\mid s,a)}\!\Bigl[\log\sum_{a^{\prime}\in\mathcal{A}}\exp(Q(s^{\prime},a^{\prime}))\Bigr]. We write \operatorname{LSE}(Q(s^{\prime},\cdot)):=\log\sum_{a^{\prime}\in\mathcal{A}}\exp(Q(s^{\prime},a^{\prime})) for the unit-temperature log-sum-exp. If the DDC shock mean is \mu_{\epsilon}\neq 0 , the corresponding uncentered DDC operator is (\mathcal{T}_{\mathrm{DDC},\mu_{\epsilon}}Q)(s,a)\;:=\;(\mathcal{T}_{\mathrm{soft}}Q)(s,a)+\beta\mu_{\epsilon}. The log-sum-exp is a smooth surrogate for \max , sandwiched between it and a \log|\mathcal{A}| slack: \max_{a^{\prime}}Q(s^{\prime},a^{\prime})\leq\operatorname{LSE}(Q(s^{\prime},\cdot))\leq\max_{a^{\prime}}Q(s^{\prime},a^{\prime})+\log|\mathcal{A}| . The two facts below are what we need to treat

Read the complete text