DO-IQS: Dynamics-Aware Offline Inverse Q-Learning for Optimal Stopping with Unknown Gain Functions
Anna Kuchko · 2025
Read DO-IQS: Dynamics-Aware Offline Inverse Q-Learning for Optimal Stopping with Unknown Gain Functions by Anna Kuchko (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-03515.txt.
Introduction
Anna Kuchko addresses this 2025 learning problem: recover the hidden reward rule behind an expert's stopping decisions. In optimal stopping, an agent watches a process and chooses when to act, and the payoff rule is often unknown. Copying the policy alone fails when the setting shifts, so the payoff itself must be inferred. Existing inverse learning methods stumble here for several reasons.
Data thin out near the stopping boundary, the gains are not memoryless, and risky uses demand offline, stable learning. Kuchko's method folds in the system's dynamics, approximates the cumulative gain and value together, and oversamples near the boundary where data are scarce. Tests on real and synthetic data, including a critical-events intervention case, show the approach working where earlier ones falter.
Catalog record
| CREATOR | Anna Kuchko |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-03515
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DO-IQS: Dynamics-Aware Offline Inverse Q-Learning for Optimal Stopping with Unknown Gain Functions Anna Kuchko Abstract We consider the Inverse Optimal Stopping (IOS) problem where, based on stopped expert trajectories, one aims to recover the optimal stopping region through the continuation and stopping gain functions approximation. The uniqueness of the stopping region allows the use of IOS in real-world applications with safety concerns. Although current state-of-the-art inverse reinforcement learning methods recover both a Q-function and the corresponding optimal policy, they fail to account for specific challenges posed by optimal stopping problems. These include data sparsity near the stopping region, the non-Markovian nature of the continuation gain, a proper treatment of boundary conditions, the need for a stable offline approach for risk-sensitive applications, and a lack of a quality evaluation metric. These challenges are addressed with the proposed D ynamics-Aware O ffline I nverse Q -Learning for Optimal S topping ( DO-IQS ), which incorporates temporal information by approximating the cumulative continuation gain together with the world dynamics and the Q-function without querying to the environment. In addition, a confidence-based oversampling approach is proposed to treat the data sparsity problem. We demonstrate the performance of our models on real and artificial data including an optimal intervention for the critical events problem. 1 Introduction Rapidly advancing autonomous systems require efficient and precise detection and anticipation of hazardous situations. The theory of optimal stopping (OS) can be applied to problems like stopping of an autonomous vehicle, identification of a shutdown time for a production system, change point detection, etc. Designing a reward function to solve the OS problems requires a high degree of involvement and an extensive knowledge of the system’s dynamics. Recent advances in the area of Inverse Reinforcement Learning (IRL) allow one to learn both a Q-function and the optimal policy directly from demonstrations [ 8 , 12 ] . For safety-sensitive applications, the world dynamics model can also be learnt offline for greater risk awareness [ 29 , 30 ] . While useful for general optimal control problems, these methods often fail while applied to OS problems, mainly due to sparsity of the expert data near the stopping region, non-Markovian nature of the continuation gain and/or the state-space dynamics. We address these challenges with our proposed Dynamics-Aware Offline Inverse Q-Learning model for OS problems with unknown gain functions (DO-IQS). Figure 1: POMDP structure of the OS problem with cumulative continuation gain (left) and DO-IQS model structure (right). 1.1 Outline of contributions The literature on the (inverse) OS problems is mostly limited to financial applications and considers very specific examples that overlook a more general setup and do not fully address the main challenges of a practical application. In this paper, we make the following contributions to the theory and practice of IOS problems: • We formally define a reinforcement learning (RL) formulation of the OS problem accounting for both stopping and continuation gain functions, allowing us to use IRL methods (Definition 1 ). • An Inverse Optimal Stopping problem definition is presented (Definition 3 ). Fitting a stopping surface can be challenging in higher dimensions. Estimating gain functions instead provides higher precision and greater flexibility ( Insight 1 ). • We discuss the main challenges that arise in IOS problems, including the scarcity of data near the stopping boundary and the sparsity of stopping decisions caused by a natural class imbalance between continuation and stopping actions (Section 3 ). We propose a solution to the problem via oversampling strategies ( Insight 2 ). • To allow offline inference (while also correcting the Bellman error of the IQ-Learning algorithm), we estimate the environment model ( Insight 3 ). We further develop an approach, which we call a Dynamics-Aware Offline Inverse Q-Learning for Optimal Stopping (DO-IQS, Figure 1 ), to incorporate non-Markovian continuation reward learning to (Section 4.3 ) and to fight the sparsity problem. We demonstrate the performance of our approach on OS problems, including an OS of a two-dimensional Brownian motion and learning optimal intervention policies for critical events (Section 5 ). 1.2 Related work The literature on IOS and related problems is scarce and often lacks generality. Kruse and Strack [16] considers a problem of finding a way to change (unknown) gain function in a way that makes a given stopping rule optimal and show how such a problem can be analytically solved for a particular class of one-dimensional diffusion processes. We note that this approach lacks generality and is hard to use in higher dimensions. Qiao and Beling [22] use Bayesian modification of apprenticeship learning first introduced by Abbeel and Ng [1] to solve a standard secretary problem. Al-Hafez et al. [2] modify the IQ-Learn algorithm of Garg et al. [8] to account for problems with absorbing states by introducing a fixed target reward function and regressing it towards the absorbing states. Although this approach has some similarities with OS problems, it does not make use of the natural structure of the OS rewards and cannot be applied to more challenging OS examples. An analytical solution to a class of mean-field stopping games is also presented by Huang and Xie [13] . Pattanayak and Krishnamurthy [21] address an inverse optimal Bayesian stopping problem using a modification of the Max-Entropy IRL ( Ziebart et al. [32] ) with application to inverse sequential hypothesis testing and inverse Bayesian search problems over multiple environments. In our approach, we take on a more general set-up that does not require learning in different environments or a manual engineering of feature vectors. This allows us to conduct inference in higher dimensions and for a wider range of OS problems. Moreover, the authors evaluate the results of their inference by comparing the true environment costs to the IRL estimates. Instead, we use classification metrics to evaluate the quality of the resulting stopping region which, unlike the gain functions, is unique. Another recent work by Damera Venkata and Bhattacharyya [5] studies an OS approach to optimal time to intervention (OTI) problems, where the stopping gain function takes the form of a hazard rate process and is recovered using deep recurrent survival analysis. We show that our approach can also be applied to inverse OTI problems without requiring threshold tuning, imposing constraints on the shape of the hazard rate, and avoiding an error made by the survival model. 2 Background and problem setup 2.1 Optimal stopping: general formulation Let S=\{s_{t}\}_{t\geq 0}\in\mathbb{R}^{d} be a time-homogeneous Markov state-space process defined on a probability space (\Omega,\mathcal{F},\mathbb{P}_{s},s\in S) equipped with filtration (\mathcal{F}_{t})_{t\geq 0} generated by the history of observed states up to time t . A stopping time \tau:\Omega\rightarrow\{0,1,...,\infty\} is a random variable \tau\in\{0,1,2,...,\infty\} . Given a continuation gain function g:\mathbb{R}^{d}\rightarrow\mathbb{R} and a stopping gain function G:\mathbb{R}^{d}\rightarrow\mathbb{R} a value function defines the expected cumulative future payoff, if the state process is stopped at the time \tau : V^{\tau}(s)=\mathbb{E}_{s}\left[\sum_{t=0}^{\tau-1}\gamma^{t}g(S_{t})\mathds{1}_{\{\tau\geq 1\}}+\gamma^{\tau}G(S_{\tau})\right], (1) where \gamma\in\left[0,1\right] is a discount factor. The corresponding optimal stopping problem consists in finding an optimal stopping time \tau^{\star} maximising the value function, i.e. V^{\tau^{\star}}(s)=\sup_{\tau}V^{\tau}(s) . By the general theory of OS [ 24 ] , V^{\tau^{\star}}=V^{\star} , which is a fixed point of the dynamic programming operator T : V^{\star}=TV^{\star} , where TV=max\{G,g+\gamma PV\} , and P is a transition operator