Increasing Information for Model Predictive Control with Semi-Markov Decision Processes Rémy Hosseinkhan-Boucher, Stella Douka, Onofrio Semeraro, Lionel Mathelin Abstract Recent works in Learning-Based Model Predictive Control of dynamical systems show impressive sample complexity performances using criteria from Information Theory to accelerate the learning procedure. However, the sequential exploration opportunities are limited by the system local state, restraining the amount of information of the observations from the current exploration trajectory. This article resolves this limitation by introducing temporal abstraction through the framework of Semi-Markov Decision Processes. The framework increases the total information of the gathered data for a fixed sampling budget, thus reducing the sample complexity. \coltauthor ∗ ∗ Université Paris-Saclay, Orsay, France CNRS, Laboratoire Interdisciplinaire des Sciences du Numérique, Orsay, France keywords Expected Information Gain; Temporal Abstraction; Sample Complexity 1 Introduction Machine Learning Control (MLC) is an interdisciplinary area of statistical learning and control theory which solves model-free optimal control problems ( Duriez et al., 2016 ) . Among the multiple approaches of the vast field of data-driven control, two classes have received notable attention by the machine learning community: Learning-Based Model Predictive Control (LB-MPC) ( Hewing et al., 2020 ) and Model-Based Reinforcement Learning (MB-RL) ( Abbeel et al., 2006 ; Recht, 2018 ; Moerland et al., 2022 ) . The former refers to the combination of Model Predictive Control (MPC) , an optimisation method based on a sufficiently descriptive model of the system dynamics ( Grüne and Pannek, 2011 ) , and learning methods which enable the improvement of the prediction model from recorded data while possibly modeling uncertainty ( Aswani et al., 2013 ; Koller et al., 2019 ) . The latter combines general function approximators such as linear models ( Tsitsiklis and Van Roy, 1997 ) , or more generally neural networks ( Sutton et al., 1999a ) , with Dynamic Programming (DP) ( Bellman, 1957 ) principles to solve the underlying optimisation problem. * * footnotetext: Both authors contributed equally to this work. Despite the recent impressive results in learning complex dynamical models ( Ha and Schmidhuber, 2018 ) , the sample complexity of the learning process remains a major issue in the field of data-driven control ( Kakade, 2003 ; Li et al., 2021 , and see the references therein) , in which the sample complexity is defined as the sample size required to learn a good approximation of the target concept ( Mohri et al., 2018 ) . For this reason, recent works ( Mehta et al., 2022b ; Mehta et al., 2022a ) in LB-MPC have focused on the design of exploration strategies based on the Information Theory concept of Expected Information Gain (EIG) or negative Conditional Mutual Information (CMI) ( Lindley, 1956 ) . The resulting criterion allows for quantifying the gain of information given by a new state-control observation on the estimated optimal system trajectory. Hence, this tool can be used as an acquisition function to guide the exploration of the state-control space. The concept of acquisition function is borrowed from the field of Bayesian Optimisation (BO) . In particular, the work of Mehta et al. (2022b) relies on the broader black-box BO framework of Neiswanger et al. (2021) . In a setting where the data is collected along the trajectory of the dynamical system of interest, the diversity of the resulting dataset (which may be characterised by the quantity of information) is conditioned on the subsequent states of the system. Informally, the setting in which the sampling procedure is constrained by the current system state may introduce information redundancy if the system exhibits high auto-correlation or if the current state is in a slowly evolving region of the state space. Indeed, as shown in Figure 1 (auto-correlation from a perturbated fixed point of a controlled Lorenz 63’ system), the auto-correlation from an initial state can be high in average for a long period of time while the control intensity allows to reduce the correlation of the sequence of states. Figure 1: (Cov(X_{0},X_{k}))_{k\in\mathbb{N}} for the controlled Lorenz system x_{3} component under multiple control intensities. However, for dynamical systems characterised by a broad range of time scales, the notion of temporal abstraction , described in the below paragraphs, ( Precup, 2000 ; Machado et al., 2023 ) may play a key role in overcoming the issue mentioned here. Abstraction in Artificial Intelligence refers to a broad range of techniques in order to provide a more compact representation of the problem at hand ( Boutilier and Dearden, 1994 ; Banse et al., 2023 ) . In the framework of Markov Decision Process (MDP) , the work of Sutton et al. (1999b) sheds light on the limitation induced by standard MDP modeling: “There is no notion of a course of action persisting over a variable period of time. […] As a consequence, conventional MDP methods are unable to take advantage of the simplicities and efficiencies sometimes available at higher levels of temporal abstraction.” Temporal abstraction can refer to the concept of selecting the right level of time granularity to facilitate the description of the world model to achieve a given task. In simpler words, in the present case, temporal abstraction is the idea of representing and reasoning about actions and states at different time-scales and duration. In the present article, temporal abstraction through Semi-Markov Decision Processes (SMDP) modeling is introduced to improve the informativeness of the sequential exploration of the state-control space. SMDP modeling is shown to obtain a better sample complexity of the dynamics model estimator. This article thus extends the previous work of Mehta et al. (2022a) by introducing temporal abstraction to the acquisition function. The paper is organised as follows. Section 2 reviews the related works. Section 3 introduces the problem setting. Section 4 presents the hypothesis and the experimental setup while Section 5 presents the results and Section 6 concludes the paper. 2 Related Works Information Driven Model-Based Control The foundations of the Bayesian Experimental Design have been laid by the seminal work of Lindley (1956) where the author presents a measure of the information provided by an experiment. More recently, MacKay (1992) termed Expected Information Gain (EIG) a measure of the information provided by an observation allowing, in his own terms, to actively select particularly salient data points . In the field of LB-MPC, such a criterion has been used to cherry-pick the most informative state-control pair to learn the dynamics of the system ( Mehta et al., 2022b ; Mehta et al., 2022a ) . Their work is based on the broader Bayesian Optimisation method of Neiswanger et al. (2021) designed to optimise “blackbox” functions. An extensive review of Bayesian Optimisation and its applications is available in this latter paper. Learning-Based Model Predictive Control The history of learning-based models may be traced back to the seminal work by Stratonovich (1960) in probability theory which stimulated several contributions, notably the work of Kalman and Bucy (1961) , that were to compose a body of work generally referred to as filtering theory. More recently, Kamthe and Deisenroth (2018) model the system dynamics with Gaussian Processes (GP) and use MPC for data efficiency. GPs are also used in the PILCO model ( Deisenroth and Rasmussen, 2011 ) which has a high influence in MB-RL. Koller et al. (2019) model the uncertainty of the system dynamics for safe-RL. The work of Bonzanini and Mesbah (2020) presents a stochastic LB-MPC strategy to handle this uncertainty. Semi-Markov Decision Processes Temporal abstraction in reinforcement learning was pioneered in Sutton (1995) and Precup and Sutton (1997) ; Precup (2000) . Specifically, Sutton (1995) proposed learning a model and value function at different levels of temporal abstraction. The actions in SMDPs take variable amounts of time and are intended to model temporally-extended courses of action. Recent works for continuous-time control use variants of Neural Ordinary Differential Equations to model dynamics delays ( Du et al., 2020 ; Holt et al., 2023 ) . A classical use of SMDP is for queueing control and equipment maintenance ( Puterman, 2014 ) where time-delays are prominent. 3 Problem Setting 3.1 Control Model This work considers a control model given by the following d -dimensional discrete-time dynamical system X ( Duflo, 1997 ) on a probability space \left(\Omega,\mathcal{F},P\right) defined by \displaystyle X_{k+1}=F\left(X_{k},\,U_{k},\,\eta_{k}\right) (1) \displaystyle X_{0}\sim\mathcal{N}(x_{e},\,\sigma_{e}^{2}I_{d}) with X_{k}\in\mathcal{X} , U_{k}\in\mathcal{U} and \eta_{k}\in\mathcal{Z} for any k\in\mathbb{N} , where \mathcal{X} , \mathcal{U} and \mathcal{Z} are respectively the corresponding state, control and disturbance spaces. The initial state starts from a reference state x_{e} (a system equilibrium or fixed point 1 1 1 In this work a fixed point is considered as a point of the state space x_{e}\in\mathcal{X} such that F(x_{e},0)=x_{e} . ) on which centered Gaussian noise with diagonal covariance is additively applied, X_{0}\sim\mathcal{N}(x_{e},\,\sigma_{e}^{2}I_{d}) . The i.i.d. random process (\eta_{k})_{k\in\mathbb{N}} is such that \eta_{k} is independent of all previous states and controls for any k\in\mathbb{N} . The distribution of \eta_{k} for any k\in\mathbb{N} is denoted by P_{\eta} . Coupled with the dynamics, an instantaneous cost function c:X\times U\rightarrow\mathbb{R}_{+} is also given to define the control model. In the sequel, it will be convenient to define the control model as a Markov Control Model (MCM) ( Hernández-Lerma, 1989 ) defined by the following transition probability \mathcal{P} on \mathcal{X}\times\mathcal{U} : \mathcal{P}(B_{\mathcal{X}},\,(x,u))=\int_{\mathcal{Z}}1_{B_{\mathcal{X}}}\left(F(x,u,z)\right)P_{\eta}(dz)=P_{\eta}(\left\{z\in\mathcal{Z}\mid F(x,u,z)\in B_{\mathcal{X}}\right\}) (2) for any B_{\mathcal{X}}\in\mathcal{B}(\mathcal{X}) (Borel \sigma -algebra) and (x,u)\in\mathcal{X}\times\mathcal{U} . The function 1 is the indicator function. Hence, the conditional distribution of X_{k+1} given X_{k} and U_{k} is given by \mathcal{P}(B_{\mathcal{X}},(x,u))=P(X_{k+1}\in B_{\mathcal{X}}\mid X_{k}=x,U_{k}=u) (3) Additionally, in this context, a policy \pi is a transition probability on \mathcal{U} given \mathcal{X} , i.e. , a distribution on controls conditioned on states. In the rest of the paper, \pi\left(x,du\right)=\delta_{\left\{u\right\}} is the Dirac measure at u . Hence the notation is simplified to \pi\left(x\right)=u . Together, a control model, a policy \pi and an initial distribution on \mathcal{X} define a stochastic process with distribution P^{\pi} on the space of trajectories (\mathcal{X}\times\mathcal{U})^{T} . The distribution of the process is given by P(d{x_{0}}d{u_{0}}d{x_{1}}\dots)=P_{X_{0}}(d{x_{0}})\pi(d{u_{0}}\mid d{x_{0}})\mathcal{P}(d{x_{1}}\mid d{x_{0}},d{u_{0}})\dots More details on the stochastic process are given in Hernández-Lerma and Lasserre (1996) ; Puterman (2014) . Lastly, the history process (H_{k})_{k\in\mathbb{N}} is defined as H_{k}=(X_{0},\,U_{0},\dots,X_{k}) for any k\in\mathbb{N} . When k=T , H_{T} is called the trajectory of the process. The process (X_{k},U_{k},X_{k+1})_{k\in\mathbb{N}} is called the transition process and the marginal process \left(X_{k}\right)_{k\in\mathbb{N}} is called a Markov Decision Process (MDP) . 3.2 Control Problem The studied control problem is to find a policy \pi^{*} which minimises the following performance criterion J^{\pi}=\mathbb{E}^{\pi}\left[\sum_{k=0}^{T}c\left(X_{k},U_{k}\right)\right] (4) where T\in\mathbb{N} is a given time-horizon and \mathbb{E}^{\pi} denotes the expectation under the probability measure P^{\pi} . The quantity J^{\pi} is called the value function or objective function. The history process under \pi^{*} is called the optimal history process and is denoted by (H_{k}^{*})_{k\in\mathbb{N}} and the random variable H_{T}^{*} is called the optimal trajectory . In this work, the optimal policy \pi^{*} is estimated with Model Predictive Control (MPC) applied on a model of the dynamics. The MPC approach consists of solving a finite-horizon optimal control problem at each time step, formally it defines the following policy \displaystyle\pi^{\text{MPC}}(x)=u_{0}^{*} (5) \displaystyle s.t.\quad(u_{0}^{*},\,\ldots,\,u_{T^{\text{MPC}}}^{*})=\argmin_{(u_{0},\,\ldots,\,u_{T^{\text{MPC}}})}\mathbb{E}^{(u_{0},\,\ldots,\,u_{T^{\text{MPC}}})}\left[\sum_{k=0}^{T^{\text{MPC}}}c\left(X_{k},u_{k}\right)\mid X_{0}=x\right] (6) where T^{\text{MPC}}\leq T is the MPC planning horizon, x\in\mathcal{X} is the current state and the expectation is taken with the underlying probability measure P^{(u_{0},\,\ldots,\,u_{T^{\text{MPC}}})} is characterised by a deterministic policy (Dirac measures) concentrated at u_{k} for all 0\leq k\leq T^{\text{MPC}} . The policy obtained with MPC on \mathcal{P} is denoted by \pi^{\text{MPC}} . The history process under \pi^{\text{MPC}} is denoted by H^{\text{MPC}}=(H_{k}^{\text{MPC}})_{k\in\mathbb{N}} , it is an approximation of the optimal history process (H_{k}^{*})_{k\in\mathbb{N}} and the random variable H_{T}^{\text{MPC}} is an approximation of the optimal trajectory H_{T}^{*} . The objective function under \pi^{\text{MPC}} is denoted by J^{\text{MPC}} . The MPC procedure is here performed with the iCEM algorithm, an improved version of the Cross Entropy Method (CEM) ( Rubinstein and Kroese, 2004 ; Pinneri et al., 2021 ) , a zeroth-order optimisation algorithm based on Monte-Carlo estimation. 3.3 Gaussian Process Modeling The use of Gaussian Process (GP) regression to model relevant quantities of controlled dynamical systems has long been proposed ( Kuss and Rasmussen, 2003 ; Deisenroth and Rasmussen, 2011 ; Kamthe and Deisenroth, 2018 ) notably for its distributional nature thus its ability to model uncertainty. By definition, a GP is a stochastic process (here indexed by \mathcal{X}\times\mathcal{U} ) such that any finite collection of random variables has a joint Gaussian distribution. Continuing from the aforementioned papers, GP regression is used to model the transition probability \mathcal{P} with a model estimator \hat{\mathcal{P}}_{\mathcal{D}} such that \hat{\mathcal{P}}_{\mathcal{D}}(~\cdot~,\,(x,u))\sim\mathcal{N}\left(\mu(x,u),\,\Sigma\left((x,u),\,(x,u)\right)\mid\mathcal{D}\right) (7) where \mu and \Sigma are respectively the mean and covariance functions of the GP and \mathcal{D} is a dataset of observations from the transition process (X_{k},U_{k},X_{k+1})_{k\in\mathbb{N}} . The distribution \hat{\mathcal{P}}_{\mathcal{D}} of Equation ( 7 ) is the predictive posterior distribution of the GP conditioned on the dataset \mathcal{D} (the reader is referred to Rasmussen and Williams (2006) for more details on GP regression). The processes \hat{X} , \hat{U} and \hat{H} are respectively the state, control and history processes under the approximate model and the same rules of notation apply as for the original processes. The MPC policy obtained with the approximate model \hat{\mathcal{P}}_{\mathcal{D}} is denoted by \widehat{\pi}^{\text{MPC}} . The history process under \widehat{\pi}^{\text{MPC}} is denoted by \widehat{H}^{\text{MPC}}=(\widehat{H}_{k}^{\text{MPC}})_{k\in\mathbb{N}} and the objective function under \widehat{\pi}^{\text{MPC}} is denoted by \widehat{J}^{\text{MPC}} . Notably, this work focuses on the sample complexity required to estimate a model \hat{\mathcal{P}}_{\mathcal{D}} of the true dynamics \mathcal{P} accurate enough to obtain a MPC policy \widehat{\pi}^{\text{MPC}} that is close to the optimal policy \pi^{*} . Hence, two time units are considered: the sampling iteration n which represents the number of observations gathered from the system so far, and the time index k of the current state X_{k} of the underlying dynamical system X . It is supposed in the following that n\leq k : it is not possible to gather more observations than the number of time steps of the system. 3.4 Expected Information Gain For a fixed sampling budget n and a fixed configuration ( e.g. the horizon T^{\text{MPC}} , the number of samples for the Monte-Carlo estimation of the cost or the other hyper-parameters of the iCEM algorithm) to perform the MPC procedure \pi^{\text{MPC}} , the control performance mainly lies in the quality of the model estimator \hat{\mathcal{P}}_{\mathcal{D}_{n}} . It depends on two main elements: the choice a priori of the mean and kernel functions \mu and \Sigma and the collection \mathcal{D}_{n} of n observations. From the work of Mehta et al. (2022b) ; Mehta et al. (2022a) , the selection of the observations can be guided by the maximisation of the Expected Information Gain (EIG) on the optimal trajectory. Let suppose the time iteration k of the underlying observed process X is equal to the number of samples gathered, i.e. , k=n and the dataset is already collected 2 2 2 In this specific case of k=n , the dataset \mathcal{D}_{n} simply contains the whole past trajectory of X , it is a realisation of H_{n} , in other words \mathcal{D}_{n}=H_{n}(\omega) for some random outcome \omega\in\Omega . at the sampling iteration n such that \mathcal{D}_{n}=\left((x_{i},u_{i},{x^{\prime}}_{i})\right)_{i=0}^{n-1} and denote by \left(X_{n},U_{n}\right) a new random state-control pair to draw from the system. The goal is to select the state-control pair (x,u) that maximises the Expected Information Gain EIG on the optimal trajectory which is defined by \text{EIG}_{n}(x,u)=\mathcal{H}\left[\hat{H}_{T}^{*}\mid\mathcal{D}_{n}\right]-\mathbb{E}_{P_{X_{n+1}\mid\mathcal{D}_{n},\,X_{n}=x,\,U_{n}=u}}\left[\mathcal{H}\left[\hat{H}_{T}^{*}\mid\mathcal{D}_{n},X_{n}=x,U_{n}=u,X_{n+1}\right]\right] (8) where \mathcal{H} denotes the differential entropy of a random variable. In other words, given a level of uncertainty \mathcal{H}\left[\hat{H}_{T}^{*}\mid\mathcal{D}_{n}\right] on the optimal trajectory \hat{H}_{T}^{*} , the EIG measures the reduction of this uncertainty when the dataset of the model estimator is augmented with the transition tuple ( x,u,X_{n+1} ). An intriguing interpretation can be made by noticing that ( 8 ) is also equal to the negative Conditional Mutual Information (CMI) ( Pinsker, 1964 ; Cover and Thomas, 2006 ) of the optimal trajectory \hat{H}_{T}^{*} and the new state X_{n+1} given the dataset \mathcal{D}_{n} and the state-control pair (X_{n},U_{n}) 3 3 3 Here and after, a slight abuse of notation is made as the dataset \mathcal{D}_{n} should be written \mathcal{D}_{n}=\left((x_{i},u_{i},{x^{\prime}}_{i})\right)_{i=0}^{n-1} since the sole random quantities are X_{n} and \hat{H}_{T}^{*} but it is omitted for the sake of readability. . Thus, maximising the EIG is equivalent to minimising the CMI between the optimal trajectory and the new transition tuple hence tending to draw new states sharing less information with the optimal trajectory conditioned on the dataset \mathcal{D}_{n} and the event ( X_{n}=x,U_{n}=u ). Indeed, by definition, the CMI quantifies the independence between the distribution of the optimal trajectory and the distribution of the new state given both the dataset and the current state-action pair. By symmetry of the EIG, a more tractable formulation is given by \text{EIG}_{n}(x,u)=\mathcal{H}\left[X_{n+1}\mid\mathcal{D}_{n},\,X_{n}=x,\,U_{n}=u\right]-\mathop{\mathbb{E}}_{P_{\hat{H}_{T}^{*}\mid\mathcal{D}_{n}}}\left[\mathcal{H}\left[X_{n+1}\mid\mathcal{D}_{n},X_{n}=x,\,U_{n}=u,\,\hat{H}_{T}^{*}\right]\right] (9) It is in practice estimated by Monte-Carlo sampling as detailed in Section 4 . In the original work of Mehta et al. (2022b) , the EIG is maximised with a greedy Monte-Carlo algorithm (uniform sampling) that selects the next state-control pair (x,u) to interact with the true system and subsequently update the dataset \mathcal{D}_{n} with the new transition tuple (x,\,u,\,x^{\prime}) where x^{\prime} is sampled from the true transition probability \mathcal{P}(~\cdot~,(x,u)) . It assumes any state-control pair (x,u) can be evaluated and queried at any time step. The authors’ algorithm is called Bayesian Active Reinforcement Learning (BARL) ; the dataset and EIG obtained with this algorithm are denoted by \mathcal{D}_{n}^{\text{BARL}} and \text{EIG}^{\text{BARL}} respectively. In this setting, the dataset support is the whole state-control space, \support(\mathcal{D}_{n}^{\text{BARL}})=(\mathcal{X}\times\mathcal{U}\times\mathcal{X})^{n} . However, in many real-world applications, the system is not always controllable and the state-control pairs that can be queried are limited to a subset induced by the system trajectory. This constraint has been considered in the work following the original paper ( Mehta et al., 2022a ) where the authors proposed to restrict the dataset support to the trajectory of the system. This second algorithm is called 4 4 4 It is important to mention that the main asset of TIP is to provide a whole trajectory as input to the EIG, which is not used in this work. Thus, only the property of querying observation by following the trajectory of the system is used here. Trajectory Information Planning (TIP) and similarly the dataset and EIG obtained with this algorithm are denoted by \mathcal{D}_{n}^{\text{TIP}} and \text{EIG}^{\text{TIP}} respectively. In this case, the dataset support is limited to the trajectory of the system, \support(\mathcal{D}_{n}^{\text{TIP}})\subseteq\{((x_{k},\,u_{k},\,x_{k+1}))_{k=1}^{n}\in(\mathcal{X}\times\mathcal{U}\times\mathcal{X})^{n}\mid\exists(z_{k})_{k=1}^{n}\in\mathcal{Z}^{n},\,x_{k+1}=F(x_{k},\,u_{k},\,z_{k}),\,0\leq i\leq n\}\subseteq(\mathcal{X}\times\mathcal{U}\times\mathcal{X})^{n}=\support(\mathcal{D}_{n}^{\text{BARL}}) . This set inclusion implies that the optimal EIG obtained with TIP is lower than the one obtained with BARL provided the transition probability estimator \hat{\mathcal{P}}_{\mathcal{D}_{n}} are the same for both algorithms for a fixed current state x\in\mathcal{X} , \max_{\{(x,u),\,u^{\prime}\in\mathcal{U}\}}\text{EIG}(x,u^{\prime})\leq\max_{\{(x^{\prime},u^{\prime})\in\mathcal{X}\times\mathcal{U}\}}\text{EIG}(x^{\prime},u^{\prime}) . Besides, the latter algorithm (TIP) do not take into account the potential benefits of including dynamics time scales in the sampling process. In the next section, an extension of the TIP algorithm is proposed to increase the EIG for each of the sampling iteration through the introduction of delayed state-control pairs in the setting of Semi-Markov Decision Processes (SMDP) . The new algorithm builds upon TIP by considering the inclusion of temporally-extended actions in the data-collection procedure to reach more distant system states that are not reachable with the original TIP algorithm, hence increasing the amount of information gathered from the system. A similar use of action repetition improves learning in Deep-RL ( Sharma et al., 2017 ; Lakshminarayanan et al., 2017 ) . 3.5 Semi-Markov Decision Processes Extension A formal definition of temporal abstraction is given through the concept of options defined by Sutton et al. (1999b) where it refers to temporally extended courses of action . This concept has been shown by Parr (1998) to be equivalent to the construction of Semi-Markov Decision Processes (SMDP) which are defined below. Let call decision epoch the time index k of the underlying dynamics (X_{k})_{k\in\mathbb{N}} defined by equation ( 1 ). Semi-Markov Control Models (SMCM) generalise the concept of MCM by letting the decisions be random variables. Indeed, consider a strictly increasing random sequence (\kappa_{j})_{j\in\mathbb{N}} of integers. The random quantities \tau_{j}=\kappa_{j}-\kappa_{j-1} with support in some finite space \mathcal{T}\subsetneq\mathbb{N}\setminus\left\{0\right\} are called inter-decision times and the random index \kappa_{j} are called random decision epochs . The resulting stochastic process (X_{\kappa_{j}})_{j\in\mathbb{N}} is called a semi-Markov Decision Process . For a more detailed probabilistic construction, see ( Puterman, 2014 , p. 534) and ( Hernández-Lerma, 1989 , p. 15) . In the scope of this paper, SMDP are used to model the temporal extension of the control process . The corresponding SMCM is introduced by first extending the control space from \mathcal{U} to \mathcal{U}\times\mathcal{T} such that the temporal extension of the control is encoded in the last coordinate of the control tuple, and the new dynamics is given by \mathcal{P}^{\text{SMDP}}(dx^{\prime}\mid(x,(u,t)))=P(X_{k+t}\mid X_{k}=x,\,U_{k:k+t-1}=u) where U_{k:k+t-1}=u means that the control process is constant between k and k+t-1 . The latter definition illustrates the fact that during the inter-decision time \tau=t , the control process is constant and equal to u . From now on, this construction allows to enlarge the support of the dataset \mathcal{D}_{n} , for a fixed number of observations n while maintaining a rollout, trajectory-based sampling procedure. Indeed, the dataset support is now \support(\mathcal{D}_{n}^{\text{SM-TIP}})\subseteq\{((x_{k_{j}},\,u_{k_{j}},\,x_{k_{j}+1}))_{{j}=1}^{n}\in(\mathcal{X}\times\mathcal{U}\times\mathcal{X})^{n}\mid\exists(z_{k})_{k=1}^{n\sup(\mathcal{T})}\in\mathcal{Z}^{n\sup(\mathcal{T})},\,x_{k+1}=F(x_{k},u_{k},z_{k}),\,0\leq k\leq n\sup(\mathcal{T}),\,(k_{j})_{j=1}^{n}\in\mathcal{T}^{n},\,k_{j}1 for the SMDP algorithms (where t_{\max}>1 ). This shows that the sequential maximal EIG is approximately reached for inter-decision times that are larger than the original MDP decision times. This confirms the relevance of temporal abstraction to increase the information gathering process. However, the inter-decision times are not necessarily always equal to t_{\max} , suggesting the more informative observations are not always the temporally most distant ones. \subfigure [Lorenz] \subfigure [Pendulum] Figure 3: Inter-decision time \tau chosen by the SMDP during training. Moving on to the objective function, in the case of the Lorenz system (Figure 4 ), the evaluation performances show the learning speed is greater for the SM-TIP settings ( t_{\max}>1 ) than for the TIP setting ( t_{\max}=1 ). For the Pendulum case (Figure 4 ), except for the SM-TIP setting where t_{\max}=8 , the proposed approach shows better sample complexity since very few iterations are required to reach optimality (light blue curves ( t_{\max}\in\{2,4\} ) are below the grey curve ( t_{\max}=1 ) for the first (up to n=20 ) sampling iterations. Furthermore, one of the reasons the t_{\max}=8 fails to achieve optimal performances is likely the bootstrapping prediction error (not shown in this document) which increases with t_{\max} . Indeed, as mentioned in Section 4 due to the non-causal property of \text{EIG}^{\text{SM-TIP}} , there exists a trade-off between the temporal extension of the dynamics to reach the new region of the state space and the bootstrapping error which increases with the temporal extension. \subfigure [Lorenz] \subfigure [Pendulum] Figure 4: Evolution of the objective function J^{\widehat{\pi}^{\text{MPC}}} to evaluate the system during training. 6 Conclusion This study demonstrates that, when restricted to the trajectory of the system, the total information gathered for a given sampling budget can be increased by introducing temporal abstraction through the usage of SMDPs. Results show that learning the dynamics of the Inverted Pendulum and the Lorenz system is more data-efficient with the use of temporally-extended actions. Future work may extend this methodology to more complex systems, leveraging the flexibility of SMDPs. These systems have the potential to reach highly informative regions and efficiently capture rapid changes in system dynamics, as the information content can be increased when considering the time resolution as a decision variable. In summary, this work offers a concise yet comprehensive glimpse into the potential of SMDPs in Model Predictive Control. The results on known systems establish a robust foundation for broader applications and unveil potential future advancements in control strategies. acknowledgments-disclosure-of-funding. The authors acknowledge the support of the French Agence Nationale de la Recherche (ANR), under grant ANR-REASON (ANR-21-CE46-0008). This work was performed using HPC resources from GENCI-IDRIS (Grant 2023-[AD011014278]).