Stable CDE Autoencoders with Acuity Regularization for Offline Reinforcement Learning in Sepsis Treatment

Yue Gao · 2025

Read Stable CDE Autoencoders with Acuity Regularization for Offline Reinforcement Learning in Sepsis Treatment by Yue Gao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-15019.txt.

Introduction

Yue Gao makes a representation learner stable for sepsis treatment policies in a paper from 2025. Intensive care data arrive at irregular times, which controlled differential equations handle. Gao shows their training can be unstable, ruining the learned states and the policy built on them. Two fixes matter: keeping training stable or stopping it, and tying the states to clinical severity scores. Experiments used a well-known intensive care cohort.

Stable training gave states that tracked those scores and separated those who lived from those who did not. Policies built on them scored near the top of the scale. Unstable training collapsed both the states and the policy. Visual maps confirmed the contrast. The paper offers practical checks for clinical reinforcement learning.

Catalog record

CREATORYue Gao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-15019

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Stable CDE Autoencoders with Acuity Regularization for Offline Reinforcement Learning in Sepsis Treatment Yue Gao Abstract Effective reinforcement learning (RL) for sepsis treatment depends on learning stable, clinically meaningful state representations from irregular ICU time series. While previous works have explored representation learning for this task, the critical challenge of training instability in sequential representations and its detrimental impact on policy performance has been overlooked. This work demonstrates that Controlled Differential Equations (CDE) state representation can achieve strong RL policies when two key factors are met: (1) ensuring training stability through early stopping or stabilization methods, and (2) enforcing acuity-aware representations by correlation regularization with clinical scores (SOFA, SAPS-II, OASIS). Experiments on the MIMIC-III sepsis cohort reveal that stable CDE autoencoder produces representations strongly correlated with acuity scores and enables RL policies with superior performance (WIS return > 0.9 >0.9 ). In contrast, unstable CDE representation leads to degraded representations and policy failure (WIS return ∼ \sim 0). Visualizations of the latent space show that stable CDEs not only separate survivor and non-survivor trajectories but also reveal clear acuity score gradients, whereas unstable training fails to capture either pattern. These findings highlight practical guidelines for using CDEs to encode irregular medical time series in clinical RL, emphasizing the need for training stability in sequential representation learning. 1 Introduction Sequential decision-making is a cornerstone of modern healthcare, particularly in dynamic clinical scenarios, where timely diagnosis and adaptive treatment strategies are essential for patient survival Shashikumar et al. 2021 ; Solís-García et al. 2023 . A representative case is sepsis management, which requires timely diagnosis and appropriate treatment strategies. Reinforcement learning (RL) offers a promising framework for modeling sequential decision-making in clinical settings, where treatment strategies must adapt over time to a patient’s evolving condition. In the context of sepsis management, RL has been employed to derive policies that optimize interventions including fluid resuscitation and vasopressor administration. For instance, the AI Clinician Komorowski et al. 2018 , demonstrated RL’s potential by training on the Medical Information Mart for Intensive Care III (MIMIC-III) dataset to recommend treatment strategies that outperformed clinician baselines. Subsequent advances in deep and distributional RL further refined policy robustness and flexibility, underscoring RL’s role in sepsis management Killian et al. 2020 ; Jayaraman et al. 2024 ; Böck et al. 2022 . These advancements underscore the potential of RL to generate effective sequential treatment decisions in critical care environments. In this work, we use MIMIC-III Johnson et al. 2016a ; Johnson et al. 2016b ; Goldberger et al. 2000 as experimental data, which containts clinical data from 40,000+ ICU patients, with vital signs, lab results, and treatments recorded at irregular time intervals. It provides a rich foundation for developing data-driven sepsis management tool. However, a key challenge lies in constructing informative state representations from this noisy, irregularly sampled data. Prior work has explored recurrent neural networks (RNNs) and autoencoders for this task, but their instability during training often leads to suboptimal representations and degraded policy performance Böck et al. 2022 ; Solís-García et al. 2023 ; Killian et al. 2020 . Neural Controlled Differential Equations (Neural CDEs) are state-of-the-art models for irregular time series data due to their ability to model continuous-time dynamics and handle irregular sampling Killian et al. 2020 ; Morrill et al. 2021 . Unlike discrete-step RNNs, CDEs use differential equations to propagate hidden states, capturing latent physiological trends more accurately. This makes them particularly suited for sepsis management, where patient states evolve smoothly between sparse observations. However, CDEs are prone to training instability such as gradient explosion or collapse when unregularized or trained for excessive epochs Agarwala et al. 2022 . This instability stems fundamentally from numerical challenges in solving the underlying differential equations Shoosmith 2003 , particularly when backpropagating through adaptive ODE solvers Baker et al. 2022 . Figure 1: Distinguishing training failure modes: (Left) Classical overfitting; (Right) Numerical instability. Figure 2: Overall architecture of the proposed framework. A key challenge in applying Neural CDEs to clinical time series is distinguishing between numerical instability and model overfitting, which are two fundamentally different failure modes. Overfitting appears when the training loss continues to decrease while the validation loss starts going upward; In contrast, CDEs exhibit unique instability patterns where both losses increase simultaneously (Figure 1 ). This instability occurs when numerical solvers fail to handle stiffness in the learned vector fields Shoosmith 2003 . For clinical time series modeling, this instability is particularly crucial, as trajectory smoothness and sudden acuity transitions must both be captured. This limitation in CDE state representation was overlooked in prior studies. For instance, Killian et al. 2020 compared CDEs against other autoencoders for sepsis state representation and showed its superiority, but did not address their instability, leading to inconsistent results and an erroneous dismissal of clinical acuity score regularization. In this work 1 1 1 Code available at https://github.com/GAOYUEtianc/RL_mimic_CDE_stable , we use the MIMIC-III database to train, evaluate, and compare CDE representations for sequential state encoding, with the goal of improving discrete Batch-Constrained Q-learning (dBCQ) models for offline sepsis treatment decision-making, with the following key contributions: 1. Identify and address the critical challenge in clinical RL: training instability in Neural CDEs for time-series state representation. Proposed a rigorous early stopping strategy that reliably produces robust representations. 2. Demonstrate that clinical acuity scores (SOFA/SAPS-II/OASIS) effectively regularize CDE autoencoder training when properly stabilized, contradicting prior claims about their ineffectiveness in RL representations. 3. Establish the consistency in MSE loss trend and acuity correlation loss trend during CDE autoencoder training process, validating acuity scores as meaningful proxies for patient state severity in CDE state representations. 4. Compare three stabilization methods (gradient clipping, implicit solvers, stiffness regularization) for clinical time series representation training, and provide practical guidance for selecting methods based on impact on downstream RL policy performance. Our findings revisit and clarify inconsistencies reported in prior works such as Killian et al. 2020 , which concluded that acuity-based regularization fails to improve CDE representations. We show this claim likely arose from unstable training regimes that obscure representation quality. By addressing CDE instability, we show that acuity regularization improves both state representation quality and RL performance. 2 Background 2.1 MIMIC‑III Data Irregular Properties The MIMIC‑III repository contains ICU patient data including physiological signals, laboratory results, medications, and interventions as multivariate time series Johnson et al. 2016a . However, these measurements were recorded at irregular intervals based on clinical needs, resulting in uneven sampling patterns and substantial missing data Johnson et al. 2016b ; Rubanova et al. 2019 . The dataset contains inherent measurement noise and informative missing patterns, where missing observations could provide clinical insights. This creates a low signal-to-noise ratio that makes direct trajectory modeling challenging Che et al. 2018 ; Yoon et al. 2018 . Different variables are recorded at varying frequencies, causing covariate shift and temporal aliasing that reduce model generalization Zhang et al. 2019 . Simple imputation techniques like forward‑filling or mean substitution fail to capture complex temporal dependencies and often introduce bias Yoon et al. 2018 . To mitigate these challenges, researchers have developed several approaches. Time‑aware recurrent models use masking and time interval embeddings to explicitly handle missingness and irregularity Che et al. 2018 . Continuous‑time methods such

Read the complete text