A tutorial on optimal dynamic treatment regimes

Chunyu Wang · 2025

Read A tutorial on optimal dynamic treatment regimes by Chunyu Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-16988.txt.

Introduction

Chunyu Wang and Brian Tom wrote this tutorial in two thousand twenty-five, per its author line, which adds a second name to the catalog's. It teaches optimal dynamic treatment regimes, plans that set each care step by how a patient has fared so far.

The aim of such a plan is the best average outcome if all were to follow it. The tutorial lays out the formal frame in causal terms, and the terms that must hold to link the target to real data. It then walks through the main models and ways to learn the best plan from data. Worked cases on made-up and real data show the methods in use. The piece is meant as a clear door into a fast field.

Catalog record

CREATORChunyu Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE88 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-16988

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A tutorial on optimal dynamic treatment regimes Chunyu Wang Abstract Abstract A dynamic treatment regime is a sequence of treatment decision rules tailored to an individual’s evolving status over time. In precision medicine, much focus has been placed on finding an optimal dynamic treatment regime which, if followed by everyone in the population, would yield the best outcome on average; and extensive investigation has been conducted from both methodological and applications standpoints. The aim of this tutorial is to provide readers who are interested in optimal dynamic treatment regimes with a systematic, detailed but accessible introduction, including the formal definition and formulation of this topic within the framework of causal inference, identification assumptions required to link the causal quantity of interest to the observed data, existing statistical models and estimation methods to learn the optimal regime from data, and application of these methods to both simulated and real data. Keywords: causal inference, potential outcomes, precision medicine, model misspecification 1 Introduction Motivated by precision medicine which attempts to target the right treatment to the right patient at the right time, optimal dynamic treatment regimes, pioneered by Murphy, 2003 and Robins, 2004 , have gained more and more attention in recent years. Dynamic treatment regimes, in contrast to the population average treatment effect (ATE), highlight (i) the heterogeneity across individuals in treatment effect based on the fact that a treatment may have a negligible effect when averaged over all patients but could be beneficial for certain patient subgroups ( Powers et al., 2018 ) and (ii) the heterogeneity in treatment effect across time within an individual in response to his evolving characteristics, which are especially important in chronic diseases. An optimal dynamic treatment regime is the one leading to the best benefit (defined on the basis of specific outcomes) on average if followed by the entire population of interest. Methods for estimating optimal dynamic treatment regimes can be classified into indirect and direct approaches based on data from either randomized experiments or observational studies ( Laber et al., 2014b ; Barto and Dietterich, 2004 ) . Basically, an indirect method models in a parametric, semiparametric, or nonparametric manner the conditional distribution of the outcome (or updated outcome) from which the optimal treatment rule can be straightforwardly derived for each decision point. Q-learning ( Schulte et al., 2014 ; Laber et al., 2014a ) , A-learning ( Robins, 2004 ; Murphy, 2003 ) and causal tree ( Athey and Imbens, 2016 ) are standard and commonly used indirect methods based on different model assumptions. A direct method directly works on the defined average benefit and searches for the optimal regime which maximizes the objective over a pre-specified class of candidate regimes. Augmented inverse probability weighted estimator ( Zhang et al., 2012 ; Zhang et al., 2013 ) and marginal structural mean models ( Robins et al., 2008 ; Orellana et al., 2010 ) are examples of direct methods. In addition to the statistical models mentioned above, there are extensive development of machine learning methods in estimating optimal treatment regimes, which are more flexible and powerful in capturing highly nonlinear profile and usually perform better in prediction. However, they lead to complicated treatment decision rules, which are difficult to interpret and implement in practice, therefore will not be of great value in informing subsequent clinical designs and supporting clinical decision making ( Zhang et al., 2018 ) . We restrict our tutorial to parsimonious and interpretable statistical models in learning optimal dynamic treatment regimes from observed data. This tutorial aims to provide a self-contained and accessible introduction to optimal dynamic treatment regimes. Section 2 presents the definition and formulation of this topic in the casual inference framework, and also identification assumptions required to link the causal quantity to the observed data. Section 3 introduces standard statistical models developed to estimate the optimal regime from data. In Section 4, we conduct simulation studies under two settings to illustrate the implementation and performance of different methods. A real data application is given in Section 5, followed by a discussion in Section 6. R code is provided in the supplementary R markdown file. 2 Notations and framework Suppose we have observed data collected from n individuals in a longitudinal follow-up study. For each individual i , the corresponding records are collected in time order as O_{i}=(L_{1i},A_{1i},\dots,L_{Ki},A_{Ki},Y_{i}),i=1,\dots,n , where A_{ji} is the observed treatment assignment, taking values in \mathcal{A}_{j} , a finite set of all possible treatment options with elements a_{j} at the j th decision time; L_{ji} is the covariate information (a vector of intermediate outcomes) collected subsequent to A_{j-1,i} but prior to A_{ji} ; and Y_{i} is the observed final outcome of which, without loss of generality, larger values indicate better reward. We assume that the decision time points, indexed by j , for j=1,\dots,K are common across individuals and that no death or dropout occurs during the follow-up period. Moreover, \{O_{i}\}_{i=1}^{n} are assumed to be independent and identically distributed (i.i.d.) and the subscript i may be omitted subsequently for simplicity and ease of exposition. To formalize an optimal dynamic treatment regime, we introduce the counterfactual framework. For a given treatment history \bar{a}:=\bar{a}_{K}=(a_{1},\dots,a_{K}) , we define Y^{\ast}(\bar{a}) as the potential final outcome that would have been observed if, possibly contrary to fact, the individual had followed treatment history \bar{a} . Similarly, L^{\ast}_{j}(\bar{a}_{j-1}) , is the vector of potential intermediate outcomes that would have occurred if previously the individual had received treatment history \bar{a}_{j-1}=(a_{1},\dots,a_{j-1}) for j=2,\dots,K . For the baseline covariates, L_{1}^{\ast} is exactly the same as L_{1} since no prior treatments are included. We then define O^{\ast} to be the collection of all potential outcomes, i.e., O^{\ast}=\{L_{1}^{\ast},\dots,L^{\ast}_{j}(\bar{a}_{j-1}),\dots,L^{\ast}_{K}(\bar{a}_{K-1}),Y^{\ast}(\bar{a})\text{ for all }\bar{a}\in\bar{\mathcal{A}}\} , where \bar{\mathcal{A}}:=\bar{\mathcal{A}}_{K}=\mathcal{A}_{1}\times\dots\times\mathcal{A}_{K} is the set of all possible treatment histories \bar{a} across all K decisions. In general, for any given vector (random or not) v , we use the overbar to denote the prior (including the current) elements and use the underbar to denote the future (including the current) elements, that is, \bar{v}_{j}=(v_{1},\dots,v_{j}) and \underline{v}_{j}=(v_{j},\dots,v_{K}) . Before defining optimal dynamic treatment regimes, we firstly define dynamic treatment regimes. A dynamic treatment regime d=(d_{1},\dots,d_{K}) is a sequence of decision rules, one per decision point, indicating how treatment actions will be tailored over time to an individual’s changing status ( Murphy, 2003 ) . Specifically, the j th rule d_{j}:=d_{j}(\bar{l}_{j},\bar{a}_{j-1}) , is a deterministic function, taking the covariate history \bar{l}_{j}=(l_{1},\dots,l_{j}) as well as the treatment history \bar{a}_{j-1} prior to the j th decision time as inputs and outputting a specific treatment value a_{j}\in\mathcal{A}_{j} , j=1,\dots,K . In fact, the feasible set of output values usually depends on the input and is a subset of \mathcal{A}_{j} . For example, consider the case that once individuals start a particular treatment, they should remain on it thereafter. In this setting, the feasible output is always 1 if any element of input treatment history is 1. See Section 2 in Schulte et al., 2014 for a more rigorous statement on this issue. Let \mathcal{D} denote the class of well-defined dynamic treatment regimes. The optimal dynamic treatment regime is defined as the one that maximizes the expected potential final outcome over all eligible treatment regimes. That is, \displaystyle d^{\text{opt}} \displaystyle=\arg\max_{d\ \in\mathcal{D}}\mathbb{E}[Y^{\ast}(d)] (1)

Read the complete text