An optimal dynamic treatment regime estimator for indefinite-horizon survival outcomes
Jane She, Matthew Egberg, Michael R. Kosorok · 2025
Read An optimal dynamic treatment regime estimator for indefinite-horizon survival outcomes by Jane She, Matthew Egberg, Michael R. Kosorok (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-18070.txt.
Introduction
Jane She, Matthew Egberg, and Michael Kosorok build a method for choosing treatment sequences when the goal is the longest survival and the road has no fixed length. Chronic care unfolds in stages, and patients differ in how many decision points they face. Classic methods assume a set number of stages. Their estimator instead learns one policy for all patients and all stages at once.
The engine is a survival version of random forests, tuned to maximize mean survival time. Patient history is summarized so the data do not balloon as stages accumulate. Refitting the model repeatedly polishes the policy, and the theory pins down how many rounds suffice and that the estimator settles on the truth. Simulations with flexible stage counts look good. The team finishes by applying the method to insurance claims for children with Crohn's disease.
Catalog record
| CREATOR | Jane She, Matthew Egberg, Michael R. Kosorok |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 111 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-18070
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
An optimal dynamic treatment regime estimator for indefinite-horizon survival outcomes Jane She, Matthew Egberg, Michael R. Kosorok Abstract We propose a new method in indefinite-horizon settings for estimating optimal dynamic treatment regimes for time-to-event outcomes. This method allows patients to have different numbers of treatment stages and is constructed using generalized survival random forests to maximize mean survival time. We use summarized history and data pooling, preventing data from growing in dimension as a patient’s decision points increase. The algorithm operates through model re-fitting, resulting in a single model optimized for all patients and all stages. We derive theoretical properties of the estimator such as consistency of the estimator and value function and characterize the number of refitting iterations needed. We also conduct a simulation study of patients with a flexible number of treatment stages to examine finite-sample performance of the estimator. Finally, we illustrate use of the algorithm using administrative insurance claims data for pediatric Crohn’s disease patients. Some keywords: Reinforcement Learning, Infinite Horizon, Indefinite Horizon, Survival Analysis, Random Forests, Machine Learning, Q-learning, Precision Medicine, Dynamic Treatment Regimes 1 Introduction In this paper, we propose a dynamic treatment regime (DTR) estimator in the indefinite-horizon setting for survival outcomes through non-parametric modeling. Our method is flexible, allowing patients to have differing number of treatment visits as well as differing visit lengths. The estimator operates by maximizing the truncated mean survival time. The estimator also has robust censoring assumptions, including the handling of dependent censoring. For an introduction and overview of DTRs, please refer to [ Kosorok and Laber, 2019 ] . On a high level, the algorithm uses recursive fitting to iteratively optimize outcomes over increasingly longer time trajectories. The process continues until reaching baseline, thereby optimizing outcomes across a patient’s entire treatment trajectory. The primary outcome of interest is the truncated mean survival time. The recursive framework integrates nonparametric model fitting through generalized survival random forests to estimate remaining life at each visit. Rather than fitting multiple separate models which may compound in error, the final result is a single-model DTR estimator. This novel feature is particularly advantageous in the indefinite horizon setting and evaluates all patients across their entire trajectories using a single policy. [ Clifton and Laber, 2020 ] provides a general introduction to the recursive framework mentioned, which is applied to survival data in this context. Estimating a DTR is often of interest for chronic conditions such as diabetes, arthritis, mental illness, inflammatory bowel disease, etc., which require multiple treatments over time [ Chakraborty and Murphy, 2014 ] . As such, DTRs use data-driven approaches to maximize long-term outcomes for patients, often employing machine learning methodologies. For example, in diabetes care, it may be of interest to estimate timing and dosage of insulin treatments, which are given frequently, with the overall goal of controlling a patient’s glucose levels in the long-term, leading to reduced morbidities including kidney disease, vision deficits, and/or nerve damage. Often, machine learning and reinforcement learning are employed in estimating DTRs, with a variety of established methods such as Q-learning [ Watkins and Dayan, 1992 ] [ Clifton and Laber, 2020 ] , V-learning [ Luckett et al., 2019 ] , backwards and sequential outcome weighted learning [ Zhao et al., 2015 ] , and A-learning [ Blatt et al., 2004 , Schulte et al., 2014 ] . There are three common multi-visit decision settings considered in precision medicine, defined by the number of decision points and their characteristics: finite horizon, infinite horizon, and indefinite horizon. The methods presented in this paper are targeted towards the indefinite horizon setting where the number of decision points is large, but finite. Here, the challenge in estimating DTRs comes from the need for a flexible number of treatment visits for patients, as the number of decision points is not fixed and patients may receive differing numbers of treatment sequences. Q-learning has been employed in a variety of different reinforcement learning problems related to precision medicine such as in [ Zhao et al., 2011 ] [ Moodie et al., 2012 ] , [ Song et al., 2015 ] [ Zhang et al., 2015 ] , and has been utilized in all three time horizon settings. The algorithm relies on backwards recursion to maximize expected cumulative utility, given that future actions will follow the optimal decision strategy. However, in the case where there is patient drop out, leading to censored data, and a varying number of decision points per patient, standard backwards recursion becomes challenging as it is not clear what the final decision point is. To address both censoring with survival outcomes as well as flexibility in the number of treatment visits, [ Goldberg and Kosorok, 2012 ] employed a novel Q-learning algorithm to estimate the optimal DTR in finite horizon settings to maximize the mean survival time. The authors construct an auxiliary Q-learning model by modifying the data so observations have the same number of visits, then use backwards recursion along with inverse-probability-of-censoring weighting (IPCW) to account for censored observations. However, the algorithm assumes that censoring is independent of event times and covariates, which may be restrictive. In a finite horizon setting with survival outcomes, [ Simoneau et al., 2020 ] propose a DTR estimator by solving weighted ordinary least squares using backwards recursion and IPCW to account for right-censoring. The resulting estimator is doubly robust, and uses the mean survival time for the outcome. While this method allows for a flexible number of treatment visits and censoring, the resulting estimator is limited to a binary treatment class. Additionally, the method uses the accelerated failure time (AFT) model, which may lead to model misspecification. More recently, [ Cho et al., 2023 ] proposed a generalized survival random forest utilizing a Q-learning-based algorithm along with survival random forests by [ Ishwaran et al., 2008 ] to estimate the optimal DTR in finite horizon settings. This method considers both censoring in survival outcomes and flexibility in number of treatment visits while relaxing the censoring assumptions to allow for dependent censoring. The method allows for censoring to be conditionally independent given the covariates, prior history, and prior treatments, and employs non-parametric modeling. The survival random forests have desirable theoretical properties including consistency of estimated regime values as well as desirable convergence rates. However, such a method is designed for the finite-horizon setting and cannot be reliably used for a larger number of visits, as the method fits separate models for each visit, compounding in error if applied to a large number of visits. In the indefinite horizon setting, [ Ertefaie, 2014 ] develops a DTR estimator when there is no fixed last visit which operates through estimating equations and does not require backwards induction. However, this method does not consider survival outcomes or censoring. An additional limitation is that the method is only valid in cases where decision points have fixed spacing, limiting its applicability when treatment frequency may depend on patient health. While estimation of multi-visit regimes for survival data is a relatively new area, there have been a variety of methods developed to estimate DTRs for survival outcomes, such as [ Jiang et al., 2017 ] , [ Huang et al., 2014 ] and [ Goldberg and