Towards Physiologically Sensible Predictions via the Rule-based Reinforcement Learning Layer
Lingwei Zhu, Zheng Chen, Yukie Nagai, Jimeng Sun · 2025
Read Towards Physiologically Sensible Predictions via the Rule-based Reinforcement Learning Layer by Lingwei Zhu, Zheng Chen, Yukie Nagai, Jimeng Sun (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-19055.txt.
Introduction
Zhu, Chen, Nagai, and Sun add a rule-based reinforcement learning layer to correct impossible medical predictions, in a paper from twenty twenty-five. Predictors in healthcare can output physiologically impossible label sequences. Prior fixes need heavy expert knowledge to specify all valid transitions. Their layer needs only the much smaller set of impossible transitions.
The layer takes predicted labels as state and outputs corrected labels as actions. Rewards come from general rules that penalize impossible moves. The module is lightweight, general, and attaches to any predictor. Tests across several healthcare classification problems show significant gains. Impossible predictions drop sharply while overall accuracy rises.
Catalog record
| CREATOR | Lingwei Zhu, Zheng Chen, Yukie Nagai, Jimeng Sun |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-19055
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Towards Physiologically Sensible Predictions via the Rule-based Reinforcement Learning Layer Lingwei Zhu, Zheng Chen, Yukie Nagai, Jimeng Sun Abstract This paper adds to the growing literature of reinforcement learning (RL) for healthcare by proposing a novel paradigm: augmenting any predictor with Rule-based RL Layer (RRLL) that corrects the model’s physiologically impossible predictions. Specifically, RRLL takes as input states predicted labels and outputs corrected labels as actions. The reward of the state-action pair is evaluated by a set of general rules. RRLL is efficient, general and lightweight: it does not require heavy expert knowledge like prior work but only a set of impossible transitions. This set is much smaller than all possible transitions; yet it can effectively reduce physiologically impossible mistakes made by the state-of-the-art predictor models. We verify the utility of RRLL on a variety of important healthcare classification problems and observe significant improvements using the same setup, with only the domain-specific set of impossibility changed. In-depth analysis shows that RRLL indeed improves accuracy by effectively reducing the presence of physiologically impossible predictions. † † affiliationnotice: Equal contribution 1 Introduction The healthcare community has witnessed a surging interest in reinforcement learning (RL) as a solution to a wide variety of clinical decision problems ( Gottesman et al. 2019 ; Liu et al. 2020 ; Yu et al. 2021 ) . Unlike supervised learning models that take as input abundance of IID data and outputs a mapping from data to true labels, RL operates under the sequential decision making framework that learns from trial-and-error interaction with the environment ( Sutton & Barto 2018 ) . As noted by Yu et al. 2021 , RL is desirable as it is capable of finding optimal policies using only previous experiences of treatment and outcomes. This makes RL more attractive than existing pharmacodynamics- or simulation-based studies where an accurate model is intractable due to heterogeneity of biological systems and their individual responses. So far, successful RL examples include dynamic adaptive treatment ( Guez et al. 2008 ; Pineau et al. 2009 ; Vincent 2014 ; Fatemi et al. 2021 ) to automatic medical diagnosis ( Kao et al. 2018 ; Peng et al. 2018 ; Xu et al. 2019 ; Xia et al. 2020 ) . These applications often take as input clinical observations and assessments of patients, with success heavily dwelling on expert knowledge such as clinical decision rules ( Heerink et al. 2023 ; Yu et al. 2023 ) or dynamics of disease progression ( Saboo et al. 2021 ; Hu et al. 2023 ) . The expert knowledge is required to design an adequate Markov Decision Process ( Puterman 1994 ) such that the states/rewards/dynamics are faithful and informative; the actions are reflective of intended changes and thus the optimal policy can represent meaningful solutions. Figure 1: Overview of the proposed rule-based reinforcement learning layer. It takes as input state predictions from the predictor and outputs physiological sensible reassigned labels as actions. Reward function is jointly defined by the mismatch to true labels and the set of impossible transitions. This paper complements the growing body of RL for healthcare literature by focusing on another important but less studied problem: supporting physiologically sensible predictions. This problem naturally emerges from data-driven learning models that achieve high prediction accuracy but can make mistakes that violate the physiological rules. Take sleep for example, a human goes through five sleep stages, from wake to rapid eye movement (REM). While a sudden transition from REM to wake is possible, the reverse direction violates the physiological rules and is therefore impossible ( Berry et al. 2013 ) . However, these rules are implicit underlying training data, and the established techniques or data transformation can cause a discontinuity to the rules. As a result, the data-driven models on one hand can achieve high accuracy; on the other hand, they inevitably suffer from physiologically impossible predictions resulting from not respecting the rules during training, preventing them from deployment to real circumstances ( Burrello et al. 2020 ; Boonyakitanont et al. 2021 ; Theodorou et al. 2024 ) . We propose to tackle this challenge with a new framework that augments the data-driven models with an additional Rule-based RL Layer (RRLL). Specifically, RRLL is simple and lightweight: it does not require heavy expert knowledge such as the knowledge of disease progression to characterize possible states, dynamics and reward functions ( Fatemi et al. 2021 ; Hu et al. 2023 ) . By contrast, it is sufficient for RRLL to function knowing only the rules of impossibility. This often creates a much smaller set compared to the rules of possibility, e.g. the wake stage can transition to many other stages that eventually lead to REM. With the rules of impossibility, the RL layer takes in predictions output by a classifier as states, and outputs as actions physiologically sensible, reassigned labels, evaluated by the rules as rewards. After training, the entire model can work in an end-to-end manner with RRLL simply acting as the last layer. The contributions of the paper are threefold. Firstly, we present a new framework that augments any data-driven model with an additional rule-based RL layer (RRLL) to support clinically sensible decisions. We are unaware of any published results that address a similar idea. In Section 2 we discuss physiological considerations and existing methods in detail. Secondly, we show the efficacy of the proposed RL layer by extensive experiments with real physiological data for sleep stage prediction and seizure detection. We comprehensively discuss the MDP for the RL problem in Section 3 . Lastly, we make public the datasets and to facilitate future research such as utilizing more sophisticated RL algorithms for the layer or developing other healthcare datasets into RL environments. Figure 2: Overview of RRLL workflow. (Left) Given a trained base predictor with frozen weights, we loop through the dataset to obtain its tentative predicted labels \hat{y} and features \boldsymbol{z} . The predicted labels and feature vectors, together with the agent’s last action constitute states. (Middle) The agent draws an action which is a reassigned label to the input instance. The state-action pair is then evaluated by the set of impossibility \zeta . (Right) After RRLL learning, the entire model can be used in an end-to-end manner. 2 Related Works RL for Healthcare. The literature of RL for healthcare has mainly evolved around dynamic adaptive treatment and automatic diagnosis ( Yu et al. 2021 ) . The former being more intuitive, often leverage disease assessments as states and dosage as actions, and the goal is to learn an optimal policy that can adaptively recommend dosage along disease progression ( Guez et al. 2008 ; Pineau et al. 2009 ; Fatemi et al. 2021 ; Li et al. 2023 ) . The latter typically abstracts symptoms as a binary state vector, with actions corresponding to queries for a certain symptoms in the state vector. The optimal policy is one that can identify the user’s disease as soon as possible by making as few as possible queries ( Peng et al. 2018 ; Xu et al. 2019 ; Zhong et al. 2022 ; Yu et al. 2023 ) . Our work differs from them in that RRLL aims at