Modified treatment policies that depend on the natural history of treatment. Iván Díaz 1,∗ , Nicholas T. Williams 2 , Paweł Morzywołek 3 , and Kara E. Rudolph 4 1 Division of Biostatistics, Department of Population Health, New York University Grossman School of Medicine 2 Division of Biostatistics, School of Public Health, University of California at Berkeley 3 Section of Biostatistics, Department of Public Health, University of Copenhagen 4 Department of Epidemiology, Mailman School of Public Health, Columbia University ∗ Corresponding author: ivan.diaz@nyu.edu Abstract Longitudinal modified treatment policies (LMTP) are a class of interventions that allow the definition, identification, and estimation of causal effects in general settings, such as with continuous or multivariate exposures, treatment regimens that require grace periods. Targeted machine learning estimators (i.e., double/debiased) have been formulated for LMTPs that assign the exposure at time t t as a function of the natural value of treatment at time t t . However, important applications such as estimating the effect of a delay in the start of a treatment require formulating LMTPs that depend not only on the natural value of treatment at time t t but also on the history of the natural value of treatment prior to time t t . This paper develops targeted learning estimators for this general case. We discuss the definition of the effects, and propose estimators that use an augmented-data version of the sequential regression form of the longitudinal g-computation formula. Our estimators are based on the efficient influence function and provide n \sqrt{n} inference under standard doubly robust rate assumptions on the convergence of the outcome and treatment regressions. We apply the new estimators to assess the effect of delaying a risky pain treatment by one month on 12-month incidence of opioid use disorder. Key words: Modified treatment policies, natural value of treatment, longitudinal g-computation formula. 1 Introduction Modified treatment policies (MTPs), also known as dynamic interventions that depend on the natural value of treatment ( 6 ; 10 ; 18 ; 32 ; 8 ; 11 ) , provide a flexible way to define, identify, and estimate causal effects in real-world situations such as continuous or multivariate exposures ( 12 ) , limited overlap or practical positivity violations ( 31 ) , grace periods or delays in treatment initiation ( 30 ) , longitudinal studies with time-varying treatments, censoring, and competing risks ( 5 ) , resource-constrained treatment policies ( 25 ) , and policy interventions with spillovers or network interference ( 1 ) . 24 provide a thorough review of the literature on methods involving the natural value of treatment. Longitudinal modified treatment policies were originally formulated broadly enough to allow the intervention at time t to depend on the history of natural treatment values up to time t ( 18 ) . The natural treatment value at time t is the treatment that would have been observed had the intervention been stopped immediately before treatment at time t was assigned. Thus, in the original formulation, the policy at time t may depend both on the contemporaneous natural treatment value and on earlier natural treatment values. In contrast, subsequent work on inverse probability weighted ( 32 ) and targeted learning estimators ( 8 ) focused on the narrower class of policies in which the intervention at time t depends only on the contemporaneous natural treatment value. This restriction rules out policies whose treatment rule depends on the natural values of treatment at previous timepoints, such as when the intervention modifies the timing of a naturally occurring treatment process rather than only the contemporaneous treatment value. For example, in longitudinal policy evaluation, one may wish to estimate the effect of delaying the natural date of policy adoption, such as shifting the enactment time of a naloxone access law ( 22 ) . In this case, treatment at time t depends on the natural policy status at an earlier time, or equivalently on the natural adoption time. For longitudinal modified treatment policies that depend only on the contemporaneous natural value of treatment, 8 developed targeted machine learning estimators based on efficient influence function that achieve \sqrt{n} -consistency under slow convergence rates for the nuisance regressions, including the outcome regressions and treatment mechanisms. These estimators rely on the fact that, for this class of interventions, the generalized g-computation formula of 18 admits a standard iterated conditional expectation representation ( 2 ; 15 ) . We show that this representation fails for longitudinal modified treatment policies that depend on the natural history of treatment. We therefore introduce a modified iterated conditional expectation formula on an augmented data structure and prove that it equals the corresponding g-computation functional. We then derive the efficient influence function of the resulting nonparametric functional and characterize conditions for \sqrt{n} -consistent estimation. These results yield two extensions of the estimators of 8 : a targeted minimum loss-based estimator and a sequentially doubly robust estimator. Like their original counterparts, our estimators are asymptotically efficient under a product-rate condition requiring certain regression errors to converge to zero at \sqrt{n} -rate in L_{2} norm. We assess finite-sample performance in simulations and illustrate the methods in an application estimating the effect of delaying a risky pain treatment by one month on 12-month incidence of opioid use disorder or overdose diagnosis among 12,745 Medicaid-insured patients with lumbar spinal stenosis. 2 Notation and definition of causal effects Let Z_{1},\ldots,Z_{n} denote i.i.d. observations with Z=(L_{1},A_{1},L_{2},A_{2},\ldots,L_{\tau},A_{\tau},Y)\sim\mathsf{P} , where L_{t} denotes time-varying covariates, A_{t} denotes a discrete exposure variable, and Y denotes an outcome of interest measured at the end of study follow-up. We let \mathsf{P}_{n} to denote the empirical distribution of Z_{1},\ldots\,Z_{n} , and assume \mathsf{P} is an element of the non-parametric statistical model defined as all continuous densities on Z with respect to a dominating measure \nu . We let \mathsf{P}f=\int f(z)\mathrm{d}\mathsf{P}(z) for a given function f(z) , and let \mathsf{E} denote the expectation with respect to \mathsf{P} , i.e., \mathsf{E}\{f(Z)\}=\int f(z)\mathrm{d}\mathsf{P}(z) . We use ||f||^{2} to denote the L_{2}(\mathsf{P}) norm \int f^{2}(z)\mathrm{d}\mathsf{P}(z) . \bar{X}_{t}=(X_{1},\ldots,X_{t}) will be used to denote the history of a variable, \underline{X}_{t}=(X_{t},\ldots,X_{\tau}) to denote the future of a variable, and H_{t}=(\bar{A}_{t-1},\bar{L}_{t}) to denote the history of all variables up until just before A_{t} . When useful, we will also use notation \bar{X}_{t:k}=(X_{t},\ldots X_{k}) . We let \mathsf{g}_{t}(a_{t}\mid h_{t}) denote the probability mass function of A_{t} conditional on H_{t}=h_{t} , and use calligraphic font to denote the support of a random variable, e.g., \mathcal{A}_{t} denotes the support of A_{t} . Causal effects are defined relying on counterfactual logic. Counterfactual random variables will be formally defined using a non-parametric structural equation model ( 17 ) . Specifically, for each timepoint t , we assume the existence of deterministic functions f_{L_{t}} , f_{A_{t}} , and f_{Y} such that L_{t}=f_{L_{t}}(A_{t-1},H_{t-1},U_{L,t}) , A_{t}=f_{A_{t}}(H_{t},U_{A,t}) , and Y=f_{Y}(A_{\tau},H_{\tau},U_{Y}) , where U=(U_{L,t},U_{A,t},U_{Y}:t\in\{1,\ldots,\tau\}) is a vector of exogenous variables with a distribution satisfying assumptions that will be given in § 3 . Unless defined otherwise, sums of the type \sum_{k=s}^{t}a_{k} and products of the type \prod_{k=s}^{t}a_{k} for s>t are defined as zero and one. 2.1 Longitudinal modified treatment policies that only depend on the contemporaneous natural value of treatment We first revisit the definition of LMTPs used in 8 to contrast it with the more general definition we use in this manuscript (§ 2.2 ). 8 defined LMTP effects in terms of hypothetical interventions where the equation A_{t}=f_{A_{t}}(H_{t},U_{A,t}) is removed from the structural model, and the exposure is assigned as follows. For a user-defined function d_{1} , at the first timepoint, t=1 , the post-intervention treatment is defined as a transformation A_{1}^{d}=d_{1}(A_{1},L_{1}) of the natural value of treatment at that time, A_{1} , and the history of data collected up until A_{1} , L_{1} . At time t=1 , the natural value of treatment is equal to the observed value of treatment, but this will not be the case for subsequent timepoints. The hypothetical intervention, d_{1} , generates an intervened-on treatment at t=1 , which we have denoted A_{1}^{d} , and counterfactual L_{2}(A_{1}^{d}) . These are then part of the counterfactual history and the natural value of treatment at time t=2 , denoted with H_{2}(A_{1}^{d})=(L_{1},A_{1}^{d},L_{2}(A_{1}^{d})) and A_{2}(A_{1}^{d}) . At time t=2 , the post-intervention treatment is defined as A_{2}^{d}=d_{2}(A_{2}(A_{1}^{d}),H_{2}(A_{1}^{d})) . In particular, note that this intervention depends only on the contemporaneous natural value of treatment at time t=2 , A_{2}(A_{1}^{d}) , and not on the natural value of treatment at t=1 , A_{1} . Instead, the intervention depends on the intervened-on value of treatment at t=1 , namely A_{1}^{d}=d_{1}(A_{1},L_{1}) . This intervention at time t=2 generates an intervened-on A_{2}^{d} and updated counterfactual history, which are then used to assign the post-intervention exposure at time t=3 as A_{3}^{d}=d_{3}(A_{3}(\bar{A}_{2}^{d}),H_{3}(\bar{A}_{2}^{d})) . This process is iterated forward in time until interventions at all times t\in[\tau] have been performed, at which point we obtain the counterfactual outcome Y(\bar{A}^{d})=f_{Y}(A_{\tau}^{d},H_{\tau}(\bar{A}_{\tau-1}^{d}),U_{Y}) . The general definition of LMTPs that depend on the current natural value of treatment is as follows: Definition 1 . An intervention variable A_{t}^{d} is said to be a longitudinal modified treatment that depends on the contemporaneous natural value of treatment if it is defined as A^{d}_{t}=d_{t}(A_{t}(\bar{A}_{t-1}^{d}),H_{t}(\bar{A}_{t-1}^{d})) for a user-given function d_{t} . The data generating process that leads to the above counterfactual outcome is illustrated in Figure 1(a) with \tau=2 and omitting L_{1} . 11 provide a large list of interventions of practical interest that can be described using this framework. Importantly, delay interventions do not fit this definition. We elaborate on this point in § 2.3 . 2.2 Longitudinal modified treatment policies that depend on the natural history of treatment In this paper, we generalize the above definition to allow dependence of the intervention at time t on the history of the natural value of treatment—instead of just on the contemporaneous natural value of treatment. Specifically, counterfactual outcomes are generated as follows. The intervention at t=1 is defined identical to § 2.1 . At time t=1 , the post-intervention variable is defined as a transformation A_{1}^{d}=d_{1}(A_{1},L_{1}) of the natural value (which, again, is also the observed value) of treatment A_{1} and L_{1} . This intervention generates a counterfactual history and treatment at time t=2 , denoted with H_{2}(A_{1}^{d})=(L_{1},A_{1}^{d},L_{2}(A_{1}^{d})) and A_{2}(A_{1}^{d}) , respectively. At time t=2 , the two definitions diverge. At this timepoint the post-intervention exposure is generated as A_{2}^{d}=d_{2}(\bar{A}_{2}(A_{1}^{d}),H_{2}(A_{1}^{d})) . The difference between this and the definition of A_{2}^{d} in § 2.1 is the overbar for the history of the natural value \bar{A}_{2}(A_{1}^{d}) , which is defined as \bar{A}_{2}(A_{1}^{d})=(A_{1},A_{2}(A_{1}^{d})) . In particular, note that this intervention can depend on two values of treatment at t=1 , the natural value of treatment, A_{1} , and the intervened value of treatment, A_{1}^{d}=d_{1}(A_{1},L_{1}) . This intervention at time t=2 generates a counterfactual history and natural value of treatment at time t=3 , H_{3}(\bar{A}_{2}^{d})=(L_{1},A_{1}^{d},L_{2}(A_{1}^{d}),A_{2}^{d},L_{3}(\bar{A}_{2}^{d})) and A_{3}(\bar{A}_{2}^{d}) , similar to § 2.1 . These variables are then used to assign the post-intervention exposure at time t=3 as A_{3}^{d}=d_{3}(\bar{A}_{3}(\bar{A}_{2}^{d}),H_{3}(\bar{A}_{2}^{d})) , where \bar{A}_{3}(\bar{A}_{2}^{d})=(A_{1},A_{2}(A_{1}^{d}),A_{3}(\bar{A}_{2}^{d})) . This process is iterated forward in time until interventions at all times t\in[\tau] have been performed, at which point we obtain the counterfactual outcome Y(\bar{A}^{d})=f_{Y}(A_{\tau}^{d},H_{\tau}(\bar{A}_{\tau-1}^{d}),U_{Y}) . Below is the formal definition, which coincides with the general definition given in §5.2 of 18 . Figure 2 gives a graphical illustration of the differences between this definition and the definition of § 2.1 . Definition 2 . An intervention A_{t}^{d} is a generalized longitudinal modified treatment policy that depends on the history of the natural value of treatment (G-LMTP) if it is defined as A^{d}_{t}=d_{t}(\bar{A}_{t}(\bar{A}_{t-1}^{d}),H_{t}(\bar{A}_{t-1}^{d})) for a user-given function d_{t} , where the natural history of treatment \bar{A}_{t}(\bar{A}_{t-1}^{d}) is defined as (A_{1}, A_{2}(A_{1}^{d}),A_{3}(\bar{A}_{2}^{d}),\ldots,A_{t}(\bar{A}_{t-1}^{d})) . A_{1} A_{1}^{d} L_{2}(A_{1}^{d}) A_{2}(A_{1}^{d}) A_{2}^{d} Y(\bar{A}^{d}) (a) LMTP that depends on the natural contemporaneous value of treatment A_{1} A_{1}^{d} L_{2}(A_{1}^{d}) A_{2}(A_{1}^{d}) A_{2}^{d} Y(d) (b) LMTP that depends on the natural history of treatment. The dashed line is used to highlight the additional arrow used here, as compared to the figure above. Figure 1 : Single world intervention graphs (SWIGs) for a modified treatment policy with two timepoints. The dashed arrow corresponds to the generalization addressed in this work. 2.3 Interventions that depend on the natural history of treatment In this section we discuss two scientifically and clinically relevant interventions that require the G-LMTP framework, highlighting issues that arise when more narrowly defined LMTPs do not have access to the history of natural treatment values once a modification has occurred. Specificially, the G-LMTP framework is required for policies that depend on aspects of the natural treatment trajectory that may be overwritten by the intervention. 2.3.1 Delay interventions In vaccine studies, one may be interested in assessing the effect of a delay in receiving a vaccine booster. Likewise, one may be interested in estimating the effect of delaying a potentially harmful intervention such as invasive mechanical ventilation in acutely ill patients. Here we briefly show that such delay policies cannot be expressed within the LMTP framework described by 8 , but can be expressed by the G-LMTP framework described in this paper. Consider a simple setting with only two time points and a binary intervention at each timepoint. A delay in exposure of one timepoint would correspond to an intervention that assigns treatment at time t=1 as A_{1}^{d}=d_{1}(A_{1},L_{1})=0 . At time t=2 , the desired delay intervention should assign the treatment as A_{2}^{d}=A_{1} . This however, is not possible in the setting depicted by Figure 1(a) , as there is no dependence of the MTP on the prior natural value of treatment under that setup. In other words, A_{2}^{d}=d_{2}(A_{2}(A_{1}^{d}),L_{2}(A_{1}^{d}),A_{1}^{d},L_{1}) must be defined solely in terms of the natural value of treatment at time 2 and the intervened history, i.e., d_{1}(A_{1},L_{1}) . If d_{1} modifies A_{1} (e.g., from 1 to 0), the function d_{2} no longer has access to the original A_{1} , and thus cannot shift it forward in time. In contrast, under an G-LMTP, A_{2}^{d} may be defined as a function of A_{1} even if A_{1}^{d}\neq A_{1} , and the policy can be explicitly constructed to reassign the original treatment at time 1 to occur at time 2 instead. To further elaborate, consider a long-standing question in critical care regarding the effects of delaying invasive mechanical ventilation. Let A_{t}\in\{0,1,2\} denote the level of oxygen support at time t , where A_{t}=0 indicates no support, A_{t}=1 indicates non-invasive support, and A_{t}=2 indicates invasive mechanical ventilation. For an arbitrary vector \bar{a}_{t} , define the function d_{t}(a_{t},\bar{a}_{t-1})=\begin{cases}1&\text{if }a_{t}=2\text{ and }a_{s}\leq 1\text{ for all }s0 denote the maximum allowed increase. For arbitrary treatment values a_{t} and a_{t-1} , define, for t\geq 2 , d_{t}(a_{t},a_{t-1})=a_{t-1}+\delta if a_{t}-a_{t-1}>\delta , and d_{t}(a_{t},a_{t-1})=a_{t} otherwise. The rule at t=1 may be defined as d_{1}(a_{1},a_{0})=a_{1} . The corresponding G-LMTP is A_{t}^{d}=d_{t}(A_{t}(\bar{A}_{t-1}^{d}),A_{t-1}(\bar{A}_{t-2}^{d})) . This policy compares the natural dose at time t with the natural dose at time t-1 , both evaluated under the counterfactual history induced by the previous interventions. It therefore modifies only large natural escalations, while leaving smaller natural increases, dose decreases, and dose stability unchanged. This intervention cannot generally be represented as an LMTP that depends only on the contemporaneous natural value of treatment. Under such an LMTP, the treatment at time t may be written as A_{t}^{d}=d_{t}(A_{t}(\bar{A}_{t-1}^{d}),A_{t-1}^{d}). The intervened previous dose A_{t-1}^{d} , does not generally correspond to the previous natural dose A_{t-1}(\bar{A}_{t-2}^{d}) . Once a large escalation has been capped at time t-1 , the contemporaneous LMTP no longer has access to the dose the clinician would naturally have assigned at that time. As a result, it cannot determine whether the clinician’s next natural dose represents a large increase relative to the previous natural dose. To illustrate, take \delta=10 and suppose A_{1}=10 . Since the policy does not modify the dose at time 1 , both the G-LMTP and the corresponding current-time LMTP assign A_{1}^{d}=10 . Suppose that, under this intervened history, the clinician would naturally assign A_{2}(A_{1}^{d})=40 . Both policies would then cap the increase at 10 , yielding A_{2}^{d}=20 . Now consider the next timepoint. Suppose that, under the common intervened history through time 2 , the clinician would naturally assign A_{3}(A_{1}^{d},A_{2}^{d})=45. At this point the two policies differ. The G-LMTP compares this current natural dose with the previous natural dose, A_{2}(A_{1}^{d})=40 , and therefore leaves the dose unchanged because 45-40\leq\delta . Thus the G-LMTP assigns A_{3}^{d}=45 . In contrast, a current-time LMTP that only has access to the intervened history would compare the current natural dose with the previously administered intervened dose A_{2}^{d}=20 , and would assign A_{3}^{d}=30 . This is a different intervention: it caps increases relative to the previously administered intervened dose, rather than capping the clinician’s natural dose-escalation decision. 3 Identification of causal effects For fixed values \bar{a}_{t} and \bar{l}_{t} , we recursively define a_{t}^{d}=d_{t}(\bar{a}_{t},h^{d}_{t}) , where h^{d}_{t}=(\bar{a}_{t-1}^{d},\bar{l}_{t}) . G-LMTPs will be identified under the following assumptions. Alternative assumptions under a different causal model were given by 18 and 32 . Assumption 1 (Positivity) . If (a_{t},l_{t})\in\mathop{\mathrm{supp}}\{A_{t},L_{t}\mid A_{t-1}=a^{d}_{t-1},H_{t-1}=h^{d}_{t-1}\} then (a_{t}^{d},h^{d}_{t})\in\mathop{\mathrm{supp}}\{A_{t},H_{t}\} for t\in\{1,\ldots,\tau\} . Assumption 2 (Strong sequential randomization) . U_{A,t}\mbox{$\perp\!\!\!\perp$}(\underline{U}_{L,{t+1}},\underline{U}_{A,t+1})\mid H_{t} for all t\in\{1,\ldots,\tau\} . We have the following identification theorem, which allows us to compute the parameters \theta_{\mbox{\scriptsize lmtp}} as a function of the observed data distribution. Theorem 1 (Identification of the effect of H-LMTPs) . Under Assumptions 1 and 2 , \theta_{\mbox{\scriptsize lmtp}} is identified as \theta=\int_{\mathcal{\bar{A}}_{\tau},\mathcal{\bar{L}}_{\tau}}\mathsf{E}[Y\mid A_{\tau}=a_{\tau}^{d},H_{\tau}=h_{\tau}^{d}]\prod_{k=1}^{\tau}\mathrm{d}\mathsf{P}(a_{k},l_{k}\mid A_{k-1}=a_{k-1}^{d},H_{k-1}=h_{k-1}^{d}) The proof of this theorem is identical to the first part of the proof of Theorem 1 in 8 and is given in the Supplementary Materials. Theorem 1 in 8 additionally shows that current-time LMTPs admit a sequential regression representation of the identification formula, which greatly facilitates their estimation because it allows for the straightforward generalization of estimators developed for static interventions ( 15 ; 20 ) . In contrast, constructing a sequential regression representation for G-LMTPs requires more care. Consider a simple example with only two timepoints, no baseline covariates, and an intervention that only depends on the history of the natural value of treatment, i.e., A_{1}^{d}=d_{1}(A_{1}) and A_{2}^{d}=d_{2}(A_{1},A_{2}(A_{1}^{d})) . The outcome expectation under this intervention is identified as \int E[Y\mid A_{2}=d_{2}(a_{2},a_{1}),L_{2}=l_{2},A_{1}=d_{1}(a_{1})]\mathrm{d}\mathsf{P}(a_{2},l_{2}\mid A_{1}=d_{1}(a_{1}))\mathrm{d}\mathsf{P}(a_{1}). The integral with respect to (a_{2},l_{2}) cannot be written as a conditional expectation because the outcome regression involves setting A_{1} to two different values a_{1} and d_{1}(a_{1}) . However, progress can be made by decoupling the value of a_{1} used in the conditioning set \mathsf{P}(a_{2},l_{2}\mid A_{1}=d_{1}(a_{1})) from that used in d_{2}(a_{2},a_{1}) (see § 5 and Figure 2 for more discussion). Specifically, define m_{2}(a_{2},h_{2})=E[Y\mid A_{2}=a_{2},L_{2}=l_{2},A_{1}=a_{1}] . Then, for each s_{1} in the support of A_{1} , define m_{1}(s_{1},a_{1})=E[m_{2}(d_{2}(s_{1},A_{2}),H_{2})\mid A_{1}=a_{1}] , and note that \theta=E[m_{1}(A_{1},d_{1}(A_{1}))] . This can be generalized as in the following proposition. Proposition 1 (Sequential regression representation of the identification formula) . Let m_{\tau}(a_{\tau},h_{\tau})=E[Y\mid A_{\tau}=a_{\tau},H_{\tau}=h_{\tau}] , and q_{\tau}(\bar{s}_{\tau-1},A_{\tau},H_{\tau})=m_{\tau}(d_{\tau}((\bar{s}_{\tau-1},A_{\tau}),H_{\tau}),H_{\tau}) . For each \bar{s}_{t}\in{\bar{\mathcal{A}}}_{t} , recursively for t=\tau-1,\ldots,1 define \displaystyle m_{t}(\bar{s}_{t},a_{t},h_{t}) \displaystyle=E[q_{t+1}(\bar{s}_{t},A_{t+1},H_{t+1})\mid A_{t}=a_{t},H_{t}=h_{t}] \displaystyle q_{t}(\bar{s}_{t-1},a_{t},h_{t}) \displaystyle=m_{t}((\bar{s}_{t-1},a_{t}),d_{t}((\bar{s}_{t-1},a_{t}),h_{t}),h_{t}), where we define s_{0}=\emptyset . Then we have \theta=E[q_{1}(A_{1},H_{1})] . To illustrate this proposition, consider an example with \tau=3 timepoints, no baseline covariates, and an intervention that only depends on the history of the natural value of treatment but not on covariates, i.e., A_{1}^{d}=d_{1}(A_{1}) , A_{2}^{d}=d_{2}(A_{1},A_{2}(A_{1}^{d})) , and A_{3}^{d}=d_{3}(A_{1},A_{2}(A_{1}^{d}),A_{3}(\bar{A}_{2}^{d})) . According to Theorem 1 , the outcome expectation under this intervention would be identified as \int E[Y\mid d_{3}(\bar{a}_{3}),l_{3},d_{2}(\bar{a}_{2}),l_{2},d_{1}(a_{1})]\mathrm{d}\mathsf{P}(a_{3},l_{3}\mid d_{2}(\bar{a}_{2}),l_{2},d_{1}(a_{1}))\mathrm{d}\mathsf{P}(a_{2},l_{2}\mid d_{1}(a_{1}))\mathrm{d}\mathsf{P}(a_{1}). The sequential regression algorithm in Proposition 1 would proceed as follows. The recursion is initialized as m_{3}(a_{3},h_{3})=E[Y\mid A_{3}=a_{3},H_{3}=h_{3}] and q_{3}(\bar{s}_{2},a_{3},h_{3})=m_{3}(d(\bar{s}_{2},a_{3}),h_{3}) , which is equal to E[Y\mid d_{3}(\bar{s}_{2},a_{3}),l_{3},a_{2},l_{2},a_{1}] . Then, we start the iteration by computing m_{2}(\bar{s}_{2},a_{2},h_{2})=E[q_{3}(\bar{s}_{2},A_{3},H_{3})\mid A_{2}=a_{2},H_{2}=h_{2}] , which according to our definitions is equal to \int E[Y\mid d_{3}(\bar{s}_{2},a_{3}),l_{3},a_{2},l_{2},a_{1}]\mathrm{d}\mathsf{P}(a_{3},l_{3}\mid a_{2},l_{2},a_{1}). The iteration then proceeds by computing \displaystyle q_{2}(s_{1},a_{2},h_{2}) \displaystyle=m_{2}((s_{1},a_{2}),d_{2}(s_{1},a_{2}),h_{2}) \displaystyle=\int E[Y\mid d_{3}(s_{1},a_{2},a_{3}),l_{3},d_{2}(s_{1},a_{2}),l_{2},a_{1}]\mathrm{d}\mathsf{P}(a_{3},l_{3}\mid d_{2}(s_{1},a_{2}),l_{2},a_{1}) and the regression m_{1}(s_{1},a_{1})=E[q_{2}(s_{1},A_{2},H_{2})\mid A_{1}=a_{1}] which is equal to \int E[Y\mid d_{3}(s_{1},a_{2},a_{3}),l_{3},d_{2}(s_{1},a_{2}),l_{2},a_{1}]\mathrm{d}\mathsf{P}(a_{3},l_{3}\mid d_{2}(s_{1},a_{2}),l_{2},a_{1})\mathrm{d}\mathsf{P}(a_{2},l_{2}\mid a_{1}). The prediction q_{1}(A_{1})=m_{1}(A_{1},d(A_{1})) is averaged across values of A_{1} to obtain the identification formula. In other words, we augment the data using s_{t} to track the possible values that A_{t} can take in d_{t+1} , while reserving the original notation a_{t} to compute integrals with respect to P(a_{t+1},l_{t+1}\mid\ldots) (see Figure 2 ). 4 Optimality theory Efficiency theory in this paper focuses on functions d that do not depend on \mathsf{P} (recall that the function is deterministic but allowed to take a randomizer as argument). For each tt\\ \bar{s}_{(t+1):k}\end{subarray}}\mathsf{E}\left[C_{t,k}^{\prime}\left\{\frac{g_{k}^{\prime}(s_{k}\mid H_{k})}{g_{k}^{\prime}(A_{k}\mid H_{k})}-\frac{g_{k}(s_{k}\mid H_{k})}{g_{k}(A_{k}\mid H_{k})}\right\}\{m_{k}^{\prime}(\bar{s}_{k},A_{k},H_{k})-m_{k}(\bar{s}_{k},A_{k},H_{k})\}\,\,\bigg|\,\,A_{t}=a_{t},H_{t}=h_{t}\right], (2) where for t=0 the conditioning event is the null set, and for t=\tau we let \mathsf{Rem}_{\tau}(\bar{s}_{\tau},a_{\tau},h_{\tau};\eta^{\prime})=0 . The error of m_{t}^{\prime} as an estimator of m_{t} is characterized by the following first-order expansion: m_{t}^{\prime}(\bar{s}_{t},a_{t},h_{t})-m_{t}(\bar{s}_{t},a_{t},h_{t})=\mathsf{E}\big[\bar{\varphi}_{t+1}(\bar{s}_{t},Z;\eta^{\prime})\mid A_{t}=a_{t},H_{t}=h_{t}\big]+\mathsf{Rem}_{t}(\bar{s}_{t},a_{t},h_{t};\eta^{\prime}). (3) This lemma shares important connections to the von Mises-type expansions used in some of the semi-parametric inference literature ( 29 ; 28 ; 19 , e.g.,) . Inspection of this lemma teaches us that an estimator of m_{t} constructed by regressing \varphi_{t+1}(\bar{s}_{t},Z_{i};\hat{\eta}) for each s_{t} , has a doubly robust point-wise error equal to \mathsf{Rem}_{t}(\bar{s}_{t},a_{t},h_{t};\hat{\eta}) . Furthermore, this implies the following corollary: Corollary 1 (Efficient Influence Function) . Assume that d does not depend on \mathsf{P} . The efficient influence function for estimating \theta=\mathsf{E}[q_{1}(A_{1},H_{1})] in the non-parametric model is given by \bar{\varphi}_{1}(Z) . If the modified treatment policy depends only on the contemporaneous natural value of treatment, i.e., if d_{t}(\bar{a}_{t},h_{t}^{d})=d_{t}(a_{t},h_{t}^{d}) , then all the above results reduce to the results presented by 8 . If the intervention only depends on some of the history of natural values of treatment, then the argument \bar{s}_{t-1} in the function q_{t} can be modified to only include values on which the intervention depends. We elaborate on this point in the Supplementary Materials § 4 . 5 Estimation and statistical inference In this section we assume that preliminary estimators \hat{g}_{t} , and \hat{m}_{t} are available. These estimators may be obtained from flexible regression methods from the machine learning literature or ensembles thereof ( 27 ) . As implied by Theorem 2 , the consistency of these estimators will determine the consistency of our estimators of the parameter \theta . We will use Super Learner ( 27 ) , an ensemble regression algorithm known that builds a convex combination of regression algorithms in a user-given library, with weights chosen to minimize the cross-validated prediction error. A naïve plug-in estimator for m_{t}:t=1,\ldots,\tau may be obtained using pooled regressions in an augmented dataset. Let K_{t}=|\bar{\mathcal{A}}_{t}| . Define the augmented dataset \mathcal{D}_{\tau}=\left\{(i,Z_{i},\bar{s}_{\tau}):i\in\{1,\dots,n\},\;\bar{s}_{\tau}\in\bar{\mathcal{A}}_{\tau}\right\} . That is, \mathcal{D}_{\tau} is a dataset with nK_{\tau} rows and \tau additional columns corresponding to possible exposure sequences \bar{s}_{\tau} . Start by running a regression of Y_{i} on (A_{\tau,i},H_{\tau,i}) to obtain an estimator of m_{\tau}(a_{\tau},h_{\tau})=E[Y\mid A_{\tau}=a_{\tau},H_{\tau}=h_{\tau}] among the original data. Generate predictions q_{\tau}(\bar{s}_{\tau-1},A_{\tau},H_{\tau})=m_{\tau}(d_{\tau}((\bar{s}_{\tau-1},A_{\tau}),H_{\tau}),H_{\tau})=E[Y\mid A_{\tau}=d_{\tau}((\bar{s}_{\tau-1},A_{\tau}),H_{\tau}),H_{\tau}] using the augmented data \mathcal{D}_{\tau} . Because predictions are generated using the augmented data, q_{\tau}(\bar{s}_{\tau-1},A_{\tau},H_{\tau}) is a nK_{\tau} length vector. Recursively for t=\tau-1,\ldots,1 : (1) Add q_{t+1}(\bar{s}_{t},A_{t+1},H_{t+1}) as a new column to \mathcal{D}_{t+1} . (2) Construct \mathcal{D}_{t} by subsetting \mathcal{D}_{t+1} to unique combinations of (i,\bar{s}_{t}) , such that \mathcal{D}_{t} has nK_{t} rows. (3) Obtain an estimator m_{t}(\bar{s}_{t},a_{t},h_{t})=\mathsf{E}[q_{t+1}(\bar{s}_{t},A_{t+1},H_{t+1})\mid\bar{s}_{t},A_{t}=a_{t},H_{t}=h_{t}] by regressing q_{t+1}(\bar{s}_{t},A_{t+1},H_{t+1}) on (\bar{s}_{t},A_{t},H_{t}) among \mathcal{D}_{t} . (4) Generate predictions q_{t}(\bar{s}_{t-1},A_{t},H_{t})=m_{t}((\bar{s}_{t-1},A_{t}),d_{t}((\bar{s}_{t-1},A_{t}),H_{t}),H_{t}). An example of this approach is shown in Figure 2 . Note that this approach exchanges \tau K_{\tau} regressions in a size- n dataset for \tau regressions in a sequence of n,nK_{\tau-1},\dots,nK_{1} sized datasets which may be more computationally efficient. A similar pooling strategy has also previously been used for estimating so-called time-smooth counterfactual curves ( 26 ) . Here, we give the algorithm in full generality for interventions that may depend on the entire history \bar{s}_{t} . When the intervention depends only on selected components of \bar{s}_{t} , the augmented data need only carry those components. For example, delay policies require only a rolling lagged window of natural treatment values with length is determined by the magnitude of the delay. We give concrete examples in the Supplementary Materials § 4 . Step 1. Estimate m_{3}(a_{3},a_{2},a_{1}) by regressing Y on (A_{3},A_{2},A_{1}) and create the augmented dataset. \footnotesize\begin{array}[]{cccccc}i&A_{1}&A_{2}&A_{3}&Y\\ \hline\cr 1&a_{1,1}&a_{1,2}&a_{1,3}&y_{1}\\ \vdots&\vdots&\vdots&\vdots&\vdots\\ n&a_{n,1}&a_{n,2}&a_{n,3}&y_{n}\\ \end{array}\mapsto\begin{array}[]{ccccccccc}i&A_{1}&A_{2}&A_{3}&s_{1}&s_{2}&Y\\ \hline\cr 1&a_{1,1}&a_{1,2}&a_{1,3}&0&1&y_{1}\\ 1&a_{1,1}&a_{1,2}&a_{1,3}&0&0&y_{1}\\ 1&a_{1,1}&a_{1,2}&a_{1,3}&1&1&y_{1}\\ 1&a_{1,1}&a_{1,2}&a_{1,3}&1&0&y_{1}\\ \vdots&\vdots&\vdots&\vdots&\vdots&\vdots&\vdots&\vdots\\ n&a_{n,1}&a_{n,2}&a_{n,3}&1&0&y_{n}\\ \end{array} Step 2. Generate predictions q_{3}(\bar{s}_{2},A_{3})=m_{3}(d_{3}(\bar{s}_{2},A_{3}),A_{2},A_{1}) in the augmented dataset and subset to the unique rows of (i,s_{1},s_{2}) . Estimate m_{2}(s_{2},s_{1},A_{2},A_{1}) by regressing q_{3}(\bar{s}_{2},A_{3}) on (s_{2},s_{1},A_{2},A_{1}) . \footnotesize\begin{array}[]{ccccccc}i&A_{1}&A_{2}&s_{1}&s_{2}&q_{3}(\bar{s}_{2},A_{3})\\ \hline\cr 1&a_{1,1}&a_{1,2}&0&1&q_{3}((0,0),a_{1,3})\\ 1&a_{1,1}&a_{1,2}&0&0&q_{3}((0,1),a_{1,3})\\ 1&a_{1,1}&a_{1,2}&1&1&q_{3}((1,1),a_{1,3})\\ 1&a_{1,1}&a_{1,2}&1&0&q_{3}((1,0),a_{1,3})\\ \vdots&\vdots&\vdots&\vdots&\vdots&\vdots\\ n&a_{n,1}&a_{n,2}&1&0&q_{3}((1,0),a_{n,3})\end{array} Step 3. Generate predictions q_{2}(s_{1},A_{2})=m_{2}(A_{2},s_{1},d_{2}(s_{1},A_{2}),A_{1}) in the augmented dataset and subset to the unique rows of (i,s_{1}) . Estimate m_{1}(s_{1},A_{1}) by regressing q_{2}(s_{1},A_{2}) on (s_{1},A_{1}) . \footnotesize\begin{array}[]{cccccccc}i&A_{1}&A_{2}&s_{1}&s_{2}&q_{3}(\bar{s}_{2},A_{3})&q_{2}(s_{1},A_{2})\\ \hline\cr 1&a_{1,1}&a_{1,2}&0&1&q_{3}((0,0),a_{1,3})&q_{2}(0,a_{1,2})\\ 1&a_{1,1}&a_{1,2}&0&0&q_{3}((0,1),a_{1,3})&q_{2}(0,a_{1,2})\\ 1&a_{1,1}&a_{1,2}&1&1&q_{3}((1,1),a_{1,3})&q_{2}(1,a_{1,2})\\ 1&a_{1,1}&a_{1,2}&1&0&q_{3}((1,0),a_{1,3})&q_{2}(1,a_{1,2})\\ \vdots&\vdots&\vdots&\vdots&\vdots&\vdots&\vdots\\ n&a_{n,1}&a_{n,2}&1&0&q_{3}((1,0),a_{n,3})&q_{2}(1,a_{n,2})\\ \end{array}\mapsto\begin{array}[]{cccccccc}i&A_{1}&s_{1}&q_{2}(s_{1},A_{2})\\ \hline\cr 1&a_{1,1}&0&q_{2}(0,a_{1,2})\\ 1&a_{1,1}&1&q_{2}(1,a_{1,2})\\ \vdots&\vdots&\vdots&\vdots\\ n&a_{n,1}&1&q_{2}(1,a_{n,2})\\ \end{array} Step 4. Generate predictions q_{1}(A_{1})=m_{1}(A_{1},d_{1}(A_{1})) in the augmented dataset and subset to the unique rows of i . \footnotesize\begin{array}[]{cccccccc}i&A_{1}&s_{1}&q_{2}(s_{1},A_{2})&q_{1}(A_{1})\\ \hline\cr 1&a_{1,1}&0&q_{2}(0,a_{1,2})&q_{1}(a_{1,1})\\ 1&a_{1,1}&1&q_{2}(1,a_{1,2})&q_{1}(a_{1,1})\\ \vdots&\vdots&\vdots&\vdots&\vdots\\ n&a_{n,1}&1&q_{2}(1,a_{n,2})&q_{1}(a_{n,1})\\ \end{array}\mapsto\begin{array}[]{cccccccc}i&q_{1}(A_{1})\\ \hline\cr 1&q_{1}(a_{1,1})\\ \vdots&\vdots\\ n&q_{1}(a_{n,1})\\ \end{array} Figure 2 : Schematic of the pooled-regression approach to Proposition 1 with \tau=3 , A_{t}\in\{0,1\} , and no covariates. Although this naïve estimator is simple to implement, its error is first order in the error of each regression estimate at time t . This means that it is not possible to use this algorithm to construct an estimator that is generally \sqrt{n} -consistent under data-adaptive regression. To achieve this, we will leverage two approaches that rely on the canonical gradient of Proposition 1 : targeted minimum loss-based estimation (TMLE), and sequential doubly robust (SDR) regression. Both of these estimators are \sqrt{n} -consistent, efficient, and asymptotically normal under assumptions, but the SDR has the advantage over TMLE that it has a simpler error structure therefore leading to potential gains in finite sample bias and consistency, whereas the advantage of the TMLE over SDR is that it is a substitution estimator therefore offering protection in cases where the parameter space is bounded and the true value of the parameter is close to the boundary. The discussion about the difference between TMLE and SDR estimators is given elsewhere ( 16 ; 15 ; 8 , e.g.,) and is therefore omitted here. 5.1 Targeted minimum loss-based estimator We use sample splitting to avoid imposing Donsker conditions on the regression estimators ( 14 ; 33 ; 4 ) . Let {\cal V}_{1},\ldots,{\cal V}_{J} denote a random partition of the index set \{1,\ldots,n\} into J prediction sets of approximately the same size. For each j , the associated training sample is given by {\cal T}_{j}=\{1,\ldots,n\}\setminus{\cal V}_{j} . We let \hat{\eta}_{j} denote the estimator of \eta obtained by training the corresponding prediction algorithm using only data in the sample {\cal T}_{j} . Further, we let j(i) denote the index of the validation set which contains observation i . The targeted minimum loss-based estimator \hat{\theta}_{\mbox{\scriptsize tmle}} is computed as a substitution estimator that uses an estimate \tilde{m}_{1,j(i)} constructed to solve the cross-validated efficient influence function estimating equation \mathsf{P}_{n}\{\varphi_{1}(\cdot,\tilde{\eta}_{j(\cdot)})-\hat{\theta}_{\mbox{\scriptsize tmle}}\}=0 . The construction of \tilde{m}_{1,j(i)} is motivated by the observation that, for an appropriately defined variable \omega(\bar{s}_{t},a_{t},h_{t}) the efficient influence function of \theta can be expressed as a sum of terms of the form \omega(\bar{s}_{t},a_{t},h_{t})\{q_{t+1}(\bar{s}_{t},a_{t+1},h_{t+1})-m_{t}(\bar{s}_{t},a_{t},h_{t})\} , which, for fixed \bar{s}_{t} , takes the form of score functions \omega(W)\{M-\mathsf{E}(M\mid W)\} for appropriately defined variables M and W and some weight function \omega . If \mathsf{E}(M\mid W) is estimated within a weighted generalized linear model with canonical link that includes an intercept, then the weighted MLE estimate solves the score equation \sum_{i}\omega(W_{i})\{M_{i}-\hat{\mathsf{E}}(M_{i}\mid W_{i})\}=0 . TMLE uses this observation to iteratively tilt preliminary estimates of m_{t} towards a solution of the efficient influence function estimating equation. The algorithm is defined as follows: Step 1. Initialize \tilde{\eta}=\hat{\eta} and \tilde{m}_{\tau+1,j(i)}=\tilde{q}_{\tau+1,j(i)}=Y_{i} . Step 2. For k=1,\ldots,\tau , and for each \bar{s}_{k} , compute the weights \omega_{k,i}(\bar{s}_{k})=D_{0,k}(\bar{s}_{k})\left(\prod_{u=1}^{k}\frac{g_{u}(s_{u}\mid H_{u})}{g_{u}(A_{u}\mid H_{u})}\right) . Step 3. For t=\tau,\ldots,1 , for each \bar{s}_{t} : • Fit the generalized linear tilting model \link\tilde{m}_{t}^{\epsilon}(\bar{s}_{t},A_{t,i},H_{t,i})=\epsilon+\link\tilde{m}_{t,j(i)}(\bar{s}_{t},A_{t,1},H_{t,i}) , where \link(\cdot) is the canonical link. The parameter \epsilon may be estimated by running a generalized linear model of the pseudo-outcome \tilde{q}_{t+1,j(i)}(\bar{s}_{t},A_{t+1,i},H_{t+1,i}) with only an intercept term, an offset term equal to \link\tilde{m}_{t,j(i)}(\bar{s}_{t},A_{t,i},H_{t,i}) , and weights \omega_{t,i}(\bar{s}_{t}) , using all the data points in the sample. An outcome bounded in an interval [a,b] may be analyzed with logistic regression (i.e., \link=\logit ) by mapping it to an outcome (0,1) through the transformation (Y-a)/(b-a)(1-2\epsilon)+\epsilon for some small value \epsilon>0 . This approach has robustness advantages compared to fitting a linear model as it guarantees that the predictions in the next step remain within the outcome space ( 9 , see ) . • Let \hat{\epsilon} denote the maximum likelihood estimate, and update the estimates as \displaystyle\link\tilde{m}_{t,j(i)}^{\hat{\epsilon}}(\bar{s}_{t},A_{t,i},H_{t,i}) \displaystyle=\hat{\epsilon}+\link\tilde{m}_{t,j(i)}(\bar{s}_{t},A_{t,i},H_{t,i}) \displaystyle\link\tilde{q}_{t,j(i)}^{\hat{\epsilon}}(\bar{s}_{t-1},A_{t,i},H_{t,i}) \displaystyle=\hat{\epsilon}+\link\tilde{m}_{t,j(i)}((\bar{s}_{t-1},A_{t,i}),d((\bar{s}_{t-1},A_{t,i}),H_{t,i}),H_{t,i}). • Update \tilde{m}_{t,j(i)}=\tilde{m}_{t,j(i)}^{\hat{\epsilon}} , t=t-1 , and iterate. Step 4. The TMLE is defined as \hat{\theta}_{\mbox{\scriptsize tmle}}=\frac{1}{n}\sum_{i=1}^{n}\tilde{q}_{1,j(i)}(A_{1,i},H_{1,i}) . The iterative procedure and the score equation argument above guarantee that the above estimates solve the efficient influence function estimating equation, which is crucial to prove that \sqrt{n}(\hat{\theta}_{\mbox{\scriptsize tmle}}-\theta) converges to a random variable N(0;\var[\bar{\varphi}_{1}(Z;\eta)]) , provided that \mathsf{Rem}_{0}(\hat{\eta})=o_{P}(n^{-1/2}) and that the true and estimated weights \omega_{t}(s_{i}) are bounded. The proof of this result follows standard arguments in the analysis of TMLE presented elsewhere ( 33 ; 8 , e.g.,) and is therefore omitted. 5.2 Sequential regression estimator using SDR unbiased transformations In this section we use the multiply robust unbiased transformation in expression ( 3 ) to obtain an estimate of m_{t} . We say that \varphi_{t+1} is a multiply robust unbiased transformation ( 3 ; 21 ; 7 ; 13 ; 15 ) for m_{t} due to the following proposition, which is a straightforward consequence of Theorem 2 . Proposition 2 . Let \eta^{\prime} be such that either m^{\prime}_{s}=m_{s} or g^{\prime}_{s}=g_{s} for all s>t . Then we have \mathsf{E}\big[\varphi_{t+1}(\bar{s}_{t},Z;\eta^{\prime})\mid A_{t}=a_{t},H_{t}=h_{t}\big]=m_{t}(\bar{s}_{t},a_{t},h_{t}) . The above motivates the construction of the sequential regression estimator by regressing an estimate of the data transformation \varphi_{t+1}(\bar{s}_{t},Z;\eta) on (A_{t},H_{t}) for each \bar{s}_{t} , starting at \varphi_{\tau+1}(Z;\eta)=Y . For preliminary cross-fitted estimates \hat{g}_{1,j(i)},\ldots,\hat{g}_{\tau,j(i)} , the estimator is defined as follows: Step 1 Initialize \varphi_{\tau+1}(\bar{s}_{\tau},Z_{i};\underline{\check{\eta}}_{\tau,j(i)})=Y_{i} for i=1,\ldots,n . Step 2 For t=\tau,\ldots,1 and each s_{t}\in\mathcal{A}_{t} : • Compute the pseudo-outcome \check{Y}_{t+1,i}=\varphi_{t+1}(\bar{s}_{t},Z_{i};\underline{\check{\eta}}_{t,j(i)}) for all i=1,\ldots,n . • For j=1,\ldots,J : – Regress \check{Y}_{t+1,i} on (A_{t,i,}H_{t,i}) using any regression technique and using only data points i\in\mathcal{T}_{j} . – Let \check{m}_{t,j} denote the output, update \underline{\check{\eta}}_{t,j}=(\hat{g}_{t,j},\check{m}_{t,j},\ldots,\hat{g}_{\tau,j},\check{m}_{\tau,j}) , and iterate. Step 3 Define the sequential estimator regression as \hat{\theta}_{\mbox{\scriptsize sr}}=\frac{1}{n}\sum_{i=1}^{n}\varphi_{1}(Z_{i};\check{\eta}_{j(i)}) . Like its TMLE counterpart, this estimator satisfies \sqrt{n}(\hat{\theta}_{\mbox{\scriptsize sr}}-\theta) converges to a random variable N(0;\var[\bar{\varphi}_{1}(Z;\eta)]) , provided that \mathsf{Rem}_{0}(\hat{\eta})=o_{P}(n^{-1/2}) . This can be used to construct asymptotically valid confidence intervals and hypothesis tests, for example a (1-\alpha)100\% confidence interval may be constructed as \hat{\theta}_{\mbox{\scriptsize sr}}\pm z_{1-\alpha/2}\hat{\sigma}/\sqrt{n} , where \hat{\sigma}^{2} is the empirical variance of \bar{\varphi}_{1}(Z_{i};\check{\eta}_{j(i)}) . 6 Numerical illustration We use Monte Carlo simulation to illustrate the performance of the TMLE and SDR estimators. For n\in\left\{250,500,1000,5000,10000\right\} , data were simulated from the data generating mechanism given in Figure 3 . Let L_{0}\sim\mathcal{N}(0,1) , \epsilon_{t}\sim\mathcal{N}(0,1) , and A_{0}=0 , Y_{0}=0 . For t\in\{1,\dots,5\} , define: \displaystyle L_{t} \displaystyle=\begin{cases}0.5L_{t-1}+\epsilon_{t}&\text{if }Y_{t-1}=0\\ \emptyset&\text{if }Y_{t-1}=1\end{cases} \displaystyle P(A_{t}=1\mid L_{t}) \displaystyle=\begin{cases}\text{logit}^{-1}(-1.5+0.3L_{t})&\text{if }A_{t-1}=0\text{ and }Y_{t-1}=0\\ 1&\text{if }A_{t-1}=1\text{ and }Y_{t-1}=0\\ A_{t-1}&\text{if }Y_{t-1}=1\end{cases} \displaystyle P(Y_{t}=1\mid A_{t},L_{t}) \displaystyle=\begin{cases}\text{logit}^{-1}(-2+0.4L_{t}-0.8A_{t})&\text{if }Y_{t-1}=0\\ 1&\text{if }Y_{t-1}=1.\end{cases} Figure 3 : Data generating mechamism for the numerical illustration in § 6 This data generating mechanism corresponds to a survival analysis scenario with time-varying confounding and conditionally degenerate exposure assignment. At each sample size, we generated 1000 data-sets, and for each data-set, using both the SDR and TMLE, we estimated the expected value of Y_{5} under a hypothetical intervention where exposure ( A_{t} ) was delayed by one time-point. Outcome regressions were estimated using a generalized linear model with all possible two-way interactions; the propensity score at each time-point was estimated using a main-term generalized linear model with the logit link. Because the robustness results of the SDR and TMLE have been thoroughly discussed in the existing literature, (see e.g., 8 ; 31 ; 5 ) we limit the simulation study to the ideal scenario where all nuisance parameters are correctly specified and estimated at fast enough rates. Performance was summarized using absolute bias, mean squared error (MSE), n\times\text{MSE} , and nominal 95% confidence interval coverage. The true value under the intervention was calculated using Proposition 1 with correctly specified models and a “super-population” of one-million observations. The R code for the simulation can be found at https://github.com/CI-NYC/htlmtp-simulations . The results of the simulation study are presented in Table 1 . As expected, both the TMLE and SDR demonstrate improving metrics with increasing sample size. Interestingly, both estimators have greater than nominal confidence interval indicating conservative variance estimation at sample sizes greater than n=1000 . Further, at n=250 , the bias-variance tradeoff of the estimators appears slightly different with the SDR trading a slight increase in bias for smaller variance (as indicated by n\times\text{MSE} ), while the TMLE was opposite. However, we caution against drawing comparative conclusions regarding finite sample performance from such a simulation study as estimator performance can vary greatly with the data-generating mechanism ( 23 ) . Table 1 : Performance of the SDR and TMLE estimators in a simulation study. n Estimator \left|\text{Bias}\right| MSE n\times\text{MSE} 95% Cov. 250 SDR 0.007 0.002 0.527 0.920 TMLE 0.004 0.003 0.665 0.893 500 SDR 0.002 0.001 0.488 0.942 TMLE 0.002 0.001 0.495 0.941 1,000 SDR 0.000 0.000 0.472 0.959 TMLE 0.000 0.000 0.473 0.958 5,000 SDR 0.000 0.000 0.456 0.968 TMLE 0.000 0.000 0.455 0.967 10,000 SDR 0.000 0.000 0.454 0.966 TMLE 0.000 0.000 0.453 0.967 7 Motivating application We applied our proposed approach to estimate the effect of delaying a risky pain treatment by one month on 12-month incidence of opioid use disorder (OUD) or overdose diagnosis among non-dual-eligble, Medicaid-insured patients, aged 19-63 years, with lumbar spinal stenosis, N=12745. Day 0 was the first day of a pain treatment claim, which occurred within 30 days of a new-onset diagnosis of lumbar spinal stenosis (ICD-10 code M4806). Patients must have been continuously enrolled in Medicaid for the 6 months prior to day 0 (a “washout period”) to determine cohort eligibility, which, in addition to the above age and eligibility criteria, also included being opioid-naive, with no history of OUD, no cancer diagnosis, not pregnant, and community-dwelling. Additionally, they could not have been hospitalized in the 1 month prior to day 0. This cohort was drawn from Medicaid T-MSIS Analytic Files (TAF), Demographics, Other Services, Inpatient, and Pharmacy claims, 2016-2019 from the 25 states that implemented the ACA prior to 2014: AR, AZ, CA, CO, CT, DE, HI, IA, IL, KY, MA, MD, MI, MN, ND, NH, NJ, NM, NV, NY, OH, OR, RI, VT, WA, WV (excluded Maryland due to poor diagnostic code quality). We were interested in examining this treatment delay question among patients with a lumbar spinal stenosis diagnosis, because these patients typically receive multiple months of pain treatment and frequently receive pain treatments where concerns have been raised about their OUD risks. Starting from day 0, we discretized follow-up into 1-month intervals, ending at month 12. During each month, we evaluated 1) which pain treatments the patient had a claim for, 2) whether or not the patient was censored, and 3) presence of an OUD or overdose diagnosis code or receipt of medication for OUD. We collected information on the following baseline covariates during the washout period: age; sex; race/ethnicity; receipt of: TANF benefits, SSI benefits; veteran status; household size; marital status; english as primary language; diagnosis codes for: anxiety, bipolar, depression, ADHD, other psychiatric illness, alcohol use disorder, or other substance use disorder; mental health counseling claims; healthcare utilization variables: number of inpatient hospitalizations in the washout period prior to one month before day 0, number of outpatient visits, number of emergency department visits. We considered the following set of pain treatment variables: i) opioid analgesic prescription of \leq 7 days and dose \leq 50 morphine milligram equivalents (MME), ii) opioid analgesic prescription of >7 days and dose \leq 50 , iii) opioid analgesic prescription of dose >50 MME, iv) gabapentoid prescription, v) benzodiazepine prescription, vi) muscle relaxant prescription, vii) duloxetine prescription, viii) anti-inflammatory prescription, ix) interventional procedure, x) chiropractic visit, xi) physical therapy visit. Patients could have, and frequently did have, more than one treatment during a given month. For the purpose of this illustrative analysis we dichotomized treatment into the presence vs absence of any of the following higher-risk treatments, where this binary treatment variable, A_{t}=1 if any of treatments ii-v were present in month t and A_{t}=0 otherwise. Patients were censored in month t if: they had disenrolled from Medicaid by the end of month t , they became dual-eligible for Medicare, or 2019 ended. The study was approved by the Columbia University Institutional Review Board. All code to replicate cohort and variable creation and statistical analyses is given in the GitHub repository: https://github.com/CI-NYC/m4806-diagnosis . We were interested in the effect of delaying a higher-risk pain treatment by one month as compared to the observed average risk, assuming all censoring were prevented, \mathsf{E}[Y(\bar{A}^{d},\bar{C}=\bar{1})-Y(\bar{C}=\bar{1})] , where d_{t}(a_{t},\bar{a}_{t-1})=0 if a_{t}=1\text{ and }a_{s}=0\text{ for all }s\kappa.\end{cases} Thus, the rule at time t does not generally require the entire history \bar{a}_{t} , only the corresponding value of treatment a_{t-\kappa} . Equivalently, the sequential regression implementation only needs to augment the data by the relevant history. We discuss two examples. Example 1 (Delay by one time point) . Consider \tau=3 and a binary exposure representing an absorbing state. Consider an intervention given by d_{1}(a_{1},h_{1})=0 , d_{2}(\bar{a}_{2},h_{2}^{d})=a_{1} , and d_{3}(\bar{a}_{3},h_{3}^{d})=a_{2} . Then, the sequential regression formula equals the following: \displaystyle m_{3}(A_{3},H_{3}) \displaystyle=\mathsf{E}[Y\mid A_{3},H_{3}] \displaystyle q_{3}(\bar{s}_{2},A_{3},H_{3}) \displaystyle=m_{3}(d_{3}(\bar{s}_{2},A_{3},H_{3}),H_{3})=m_{3}(s_{2},H_{3}) \displaystyle m_{2}(\bar{s}_{2},A_{2},H_{2}) \displaystyle=\mathsf{E}[q_{3}(\bar{s}_{2},A_{3},H_{3})\mid A_{2},H_{2}] \displaystyle q_{2}(s_{1},A_{2},H_{2}) \displaystyle=m_{2}((s_{1},A_{2}),d_{2}((s_{1},A_{2}),H_{2}),H_{2})=m_{2}(A_{2},s_{1},H_{2}) \displaystyle m_{1}(s_{1},A_{1},H_{1}) \displaystyle=\mathsf{E}[q_{2}(s_{1},A_{2},H_{2})\mid A_{1},H_{1}] \displaystyle q_{1}(A_{1},H_{1}) \displaystyle=m_{1}(A_{1},d_{1}(A_{1},H_{1}),H_{1})=m_{1}(A_{1},0,H_{1}) \displaystyle\theta \displaystyle=\mathsf{E}[q_{1}(A_{1},H_{1})]. Although this display is written using the general notation \bar{s}_{2}=(s_{1},s_{2}) , the functions used in the recursion do not depend on all components of \bar{s}_{2} . Specifically, q_{3}(\bar{s}_{2},A_{3},H_{3}) depends on \bar{s}_{2} only through s_{2} . After substituting the intervention at time 2 , q_{2}(s_{1},A_{2},H_{2}) depends on the previous natural value s_{1} and the current natural value A_{2} , but not on a jointly augmented pair (s_{1},s_{2}) . Therefore, the one timepoint delay can be implemented by carrying only one natural-treatment value in the augmented data at each recursion step, as shown in Figure 4 . This example allows us to also illustrate the canonical gradients, which are equal to \displaystyle\psi_{3}(s_{2},Z) \displaystyle=\sum_{s_{3}}D_{2,3}(s_{2},s_{3})\frac{g_{3}(s_{3}\mid H_{3})}{g_{3}(A_{3}\mid H_{3})}(Y-m_{3}(A_{3},H_{3})) \displaystyle+q_{3}(s_{2},A_{3},H_{3})-m_{2}(s_{2},A_{2},H_{2}) \displaystyle\psi_{2}(s_{1},Z) \displaystyle=\sum_{s_{2},s_{3}}D_{1,3}(s_{2},s_{3})\frac{g_{2}(s_{2}\mid H_{2})}{g_{2}(A_{2}\mid H_{2})}\frac{g_{3}(s_{3}\mid H_{3})}{g_{3}(A_{3}\mid H_{3})}(Y-m_{3}(A_{3},H_{3})) \displaystyle+\sum_{s_{2}}D_{1,2}(s_{1},s_{2})\frac{g_{2}(s_{2}\mid H_{2})}{g_{2}(A_{2}\mid H_{2})}(q_{3}(s_{2},A_{3},H_{3})-m_{2}(s_{2},A_{2},H_{2})) \displaystyle+q_{2}(s_{1},A_{2},H_{2})-m_{1}(s_{1},A_{1},H_{1}) \displaystyle\psi_{1}(Z) \displaystyle=\sum_{s_{2},s_{3}}D_{0,3}(s_{2},s_{3})\frac{g_{1}(s_{1}\mid H_{1})}{g_{1}(A_{1}\mid H_{1})}\frac{g_{2}(s_{2}\mid H_{2})}{g_{2}(A_{2}\mid H_{2})}\frac{g_{3}(s_{3}\mid H_{3})}{g_{3}(A_{3}\mid H_{3})}(Y-m_{3}(A_{3},H_{3})) \displaystyle+\sum_{s_{1},s_{2}}D_{0,2}(s_{1},s_{2})\frac{g_{1}(s_{1}\mid H_{1})}{g_{1}(A_{1}\mid H_{1})}\frac{g_{2}(s_{2}\mid H_{2})}{g_{2}(A_{2}\mid H_{2})}(q_{3}(\bar{s}_{2},A_{3},H_{3})-m_{2}(\bar{s}_{2},A_{2},H_{2})) \displaystyle+\sum_{s_{1}}D_{0,1}(s_{1})\frac{g_{1}(s_{1}\mid H_{1})}{g_{1}(A_{1}\mid H_{1})}(q_{1}(s_{1},A_{2},H_{2})-m_{1}(s_{1},A_{1},H_{1})) \displaystyle+q_{1}(A_{1},H_{1})-\theta. Step 1. Estimate m_{3}(a_{3},h_{3}) by regressing Y on (A_{3},H_{3}) . Since q_{3} depends only on s_{2} , create an augmented dataset with one additional column s_{2} . \begin{array}[]{ccccc}i&A_{1}&A_{2}&A_{3}&Y\\ \hline\cr 1&a_{1,1}&a_{1,2}&a_{1,3}&y_{1}\\ \vdots&\vdots&\vdots&\vdots&\vdots\\ n&a_{n,1}&a_{n,2}&a_{n,3}&y_{n}\\ \end{array}\mapsto\begin{array}[]{cccccc}i&A_{1}&A_{2}&A_{3}&s_{2}&Y\\ \hline\cr 1&a_{1,1}&a_{1,2}&a_{1,3}&0&y_{1}\\ 1&a_{1,1}&a_{1,2}&a_{1,3}&1&y_{1}\\ \vdots&\vdots&\vdots&\vdots&\vdots&\vdots\\ n&a_{n,1}&a_{n,2}&a_{n,3}&1&y_{n}\\ \end{array} Step 2. Generate q_{3}(s_{2},A_{3})=m_{3}(s_{2},H_{3}) and estimate m_{2}(s_{2},A_{2},H_{2}) by regressing q_{3}(s_{2},A_{3}) on (s_{2},A_{2},H_{2}) . \begin{array}[]{ccccc}i&A_{1}&A_{2}&s_{2}&q_{3}(s_{2},A_{3})\\ \hline\cr 1&a_{1,1}&a_{1,2}&0&q_{3}(0,a_{1,3})\\ 1&a_{1,1}&a_{1,2}&1&q_{3}(1,a_{1,3})\\ \vdots&\vdots&\vdots&\vdots&\vdots\\ n&a_{n,1}&a_{n,2}&1&q_{3}(1,a_{n,3})\end{array} Step 3. To estimate m_{1} , augment the data with an additional column s_{1} . Generate q_{2}(s_{1},A_{2})=m_{2}(A_{2},s_{1},H_{2}) and regress it on (s_{1},A_{1},H_{1}) . \begin{array}[]{ccccc}i&A_{1}&A_{2}&s_{1}&q_{2}(s_{1},A_{2})\\ \hline\cr 1&a_{1,1}&a_{1,2}&0&q_{2}(0,a_{1,2})\\ 1&a_{1,1}&a_{1,2}&1&q_{2}(1,a_{1,2})\\ \vdots&\vdots&\vdots&\vdots&\vdots\\ n&a_{n,1}&a_{n,2}&1&q_{2}(1,a_{n,2})\end{array}\mapsto\begin{array}[]{cc}i&q_{1}(A_{1})\\ \hline\cr 1&q_{1}(a_{1,1})\\ \vdots&\vdots\\ n&q_{1}(a_{n,1})\end{array} Figure 4 : Reduced pooled-regression implementation of the one-timepoint delay in Example 1 , with \tau=3 , A_{t}\in\{0,1\} , and no covariates. Unlike the fully general implementation in Figure 2 , this intervention never requires simultaneous augmentation by (s_{1},s_{2}) : only the single natural value that may be used in a future delayed treatment decision is carried at each step. Example 2 (Delay by two time points) . Consider \tau=3 and a binary exposure representing an absorbing state. Consider a delay of two time units, where the intervention is given by d_{1}(a_{1},h_{1})=0 , d_{2}(\bar{a}_{2},h_{2}^{d})=0 , and d_{3}(\bar{a}_{3},h_{3}^{d})=a_{1} . Then, the sequential regression formula equals the following: \displaystyle m_{3}(A_{3},H_{3}) \displaystyle=\mathsf{E}[Y\mid A_{3},H_{3}] \displaystyle q_{3}(\bar{s}_{2},A_{3},H_{3}) \displaystyle=m_{3}(d_{3}((\bar{s}_{2},A_{3}),H_{3}),H_{3})=m_{3}(s_{1},H_{3}) \displaystyle m_{2}(\bar{s}_{2},A_{2},H_{2}) \displaystyle=\mathsf{E}[q_{3}(\bar{s}_{2},A_{3},H_{3})\mid A_{2},H_{2}] \displaystyle q_{2}(s_{1},A_{2},H_{2}) \displaystyle=m_{2}((s_{1},A_{2}),d_{2}((s_{1},A_{2}),H_{2}),H_{2})=m_{2}((s_{1},A_{2}),0,H_{2}) \displaystyle m_{1}(s_{1},A_{1},H_{1}) \displaystyle=\mathsf{E}[q_{2}(s_{1},A_{2},H_{2})\mid A_{1},H_{1}] \displaystyle q_{1}(A_{1},H_{1}) \displaystyle=m_{1}(A_{1},d_{1}(A_{1},H_{1}),H_{1})=m_{1}(A_{1},0,H_{1}) \displaystyle\theta \displaystyle=\mathsf{E}[q_{1}(A_{1},H_{1})]. A two timepoint delay requires the intervention at time 3 to only remember the natural treatment value at timepoint 1 . Similar to the illustration in Figure 4 , this implies that full data augmentation by \bar{s}_{2} is not necessary, augmenting the data with a single column s_{1} is sufficient. 4.1 Time horizon considerations. When natural treatment initiation occurs close to the end of follow-up, the delayed treatment initiation time may fall outside the study horizon. For example, in a vaccine study with \tau=10 monthly decision times and a five-month delay, individuals naturally vaccinated in months 6,\ldots,10 would remain unvaccinated throughout the observed ten-month horizon under the delayed intervention. This does not create a problem with our technical framework, but it may affect interpretation because, for late vaccines, the delayed intervention is equivalent within the study horizon to no vaccination.