Quantifying Potential Observation Missingness in Inverse Reinforcement Learning

Leo Benac, Abhishek Sharma, Alihan Hüyük, Finale Doshi-Velez · 2026

Read Quantifying Potential Observation Missingness in Inverse Reinforcement Learning by Leo Benac, Abhishek Sharma, Alihan Hüyük, Finale Doshi-Velez (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-12831.txt.

Introduction

Benac and three colleagues measure missing notes behind expert actions. Inverse reinforcement learning infers rewards from demos, but data may omit the decider's notes. In clinics, that can make expert actions look poor even when near-best given info. Usual methods learn wrong rewards. The team finds smallest tweaks to notes making expert actions appear best, and builds a method. Synthetic routing, a cancer sim, and ICU data show its use for gauging gaps.

Catalog record

CREATORLeo Benac, Abhishek Sharma, Alihan Hüyük, Finale Doshi-Velez
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE79024
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.12831

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Quantifying Potential Observation Missingness in Inverse Reinforcement Learning Leo Benac, Abhishek Sharma, Alihan Hüyük, Finale Doshi-Velez Abstract Inverse reinforcement learning (IRL), which infers reward functions from demonstrations, is a valuable tool for modeling and understanding decision-making behavior. Many variants of IRL have been developed to capture complexities of human decision-making, such as subjective beliefs, imperfect planning, and dynamic goals. However, an often-overlooked issue in real-world behavioral datasets is that the recorded data may be missing observations that were available to the original decision-maker. In use-inspired settings such as healthcare, this can make expert actions appear suboptimal, even when they were near-optimal given the information available at the time. As a result, the rewards learned by standard IRL may be misleading. In this paper, we identify the minimal perturbations to the recorded observations needed for the expert’s actions to appear optimal. We develop a practical algorithm for this problem and demonstrate its utility for quantifying the possible extent of missing observations in behavioral datasets through extensive experiments on synthetic navigation tasks, a cancer treatment simulator, and ICU treatment data. 1 Introduction A common use of inverse reinforcement learning (IRL) in healthcare is to analyze retrospective treatment data: given trajectories of patient vital signs (states) and clinician treatments (actions), can we infer the implicit objectives guiding those decisions? More generally, IRL aims to determine what reward function a decision-making agent might be optimizing given observations of their past actions ( 20 ) . Although IRL is typically performed as an intermediary step in imitation learning (mimicking the decision-making policy of some demonstrator) ( 5 ; 26 , e.g.) or apprenticeship learning (matching the demonstrator’s performance in terms of some notion of ground-truth rewards) ( 1 , e.g.) , it has also been an invaluable tool for modeling and understanding human decision-making behavior—providing a means to efficiently infer the likely goals of expert decision-makers and concisely describe them as reward functions. In line with this descriptive purpose , the literature on IRL has focused on accounting for various complexities of human decision-making that are not typically encountered in reinforcement learning—creating reward-based models of experts when they might have subjective beliefs about their environment that are different from the objective truth ( 25 ; 11 ) , when their ability to plan future actions is not perfect ( 13 ; 22 ) , or when their goals might be shifting over time ( 18 ; 12 ) . Additionally, recent descriptive models have leveraged this assumption of expert near-optimality to directly infer the environment’s unknown transition dynamics rather than a reward function ( 4 ) . However, one complication of analyzing observed behavior that is often overlooked in IRL is the possibility of missing observations---that is some of the information observed by the original decision-maker might not have been recorded for us to observe. 1 1 1 This is not to be confused with partial observations, where both the original decision-maker and us as researchers do not observe parts of the environment state. Figure 1 : Suppose states consist of shapes and colors, and actions are determined solely by colors. The behavior in two episodes begins to differ after different color changes at t=4 . If the recorded data omits colors, conventional IRL cannot accurately predict actions from that time onward. Our approach provides an alternative perspective: some unobserved change at t=4 is needed for the actions to be perfectly predictable. Missing observations can have significant implications, especially from a modeling perspective. For instance, consider a healthcare scenario where either Treatment A or Treatment B is most beneficial for a patient depending on some biomarker, say whether their blood pressure is low or high (suppose it is low 25% of the time). Accordingly, doctors always assign Treatment A to patients with low blood pressure and Treatment B to patients with high blood pressure. Now suppose blood pressure is not recorded in a dataset of treatment decisions, despite being available to doctors at decision time. If we use conventional IRL with this dataset, we might conclude that Treatment B has higher reward but that 25% of the doctors act sub-optimally. However, an equally plausible explanation given our data is that the doctors act near-optimally, and that the factor making Treatment A preferable is simply missing from the recorded observations. We might prefer one explanation over the other depending on our prior belief about the situation: the former if we trust the data collection process, or the latter if we trust the doctors’ expertise, which is our assumption in this work. This type of retrospective analysis is representative of existing healthcare uses of IRL, including work on sepsis management, ICU hypotension management, and ICU ventilation and sedation decisions ( 17 ; 27 ; 31 ) . We provide a toy example in Figure 1 to convey the intuition of our setup. Motivated by this gap in decision modeling with IRL, we ask the following question: What is the smallest perturbation that needs to be made to observations so that the corresponding actions appear to be optimal? Of course, making this question meaningful requires a careful notion of what counts as a small perturbation, which is one of the key technical issues we address. While answering this question does not allow us to recover the missing information itself, it can help us gauge how much information may be missing—more precisely, the minimal extent of missingness required if the decision-makers are assumed to be experts acting optimally. Determining whether this potential missingness is substantial can inform downstream choices about how the dataset should be used. For instance, if we intend to perform policy evaluation or regular reinforcement learning using a behavioral dataset, we might prefer algorithms designed to mitigate missing observations ( 15 ; 28 , e.g.) . Contributions. Our contributions are three-fold. Conceptually, we introduce the idea of quantifying missingness in terms of minimal perturbations needed to align the actions of an expert with those of an optimal policy. We formulate this as a new optimization problem in Section 3 . Technically, we develop an algorithm that can solve this optimization problem effectively (Section 4 ). Empirically, we demonstrate that we can recover perturbations with magnitude proportional to missingness, in a way that is robust to missingness in unimportant features, and that the learned perturbations capture meaningful structure in behavior by separating trajectories according to the underlying hidden decision context. 2 Related Work Missingness in Imitation Learning. As we briefly mentioned in the introduction, IRL is often performed as an intermediary step in imitation learning, which aims to copy the decision-making policy of a demonstrator. This is achieved by first inferring the demonstrator’s reward function via IRL, and then optimizing that reward function via regular reinforcement learning to obtain their policy. Besides this strategy, other methods have also been developed for imitation learning that avoid inferring reward functions all together ( 8 , e.g.) . For reward-free imitation, 32 ; 16 study missing observations from a causal perspective. They characterize when the demonstrator’s policy can still be imitated despite missing variables in the underlying causal structure, i.e. despite hidden confounding. More recently, 26 extended this line of work to reward-based imitation. Our goal is complementary. Rather than

Read the complete text