Increasing Information for Model Predictive Control with Semi-Markov Decision Processes
Rémy Hosseinkhan-Boucher, Stella Douka, Onofrio Semeraro, Lionel Mathelin · 2025
Read Increasing Information for Model Predictive Control with Semi-Markov Decision Processes by Rémy Hosseinkhan-Boucher, Stella Douka, Onofrio Semeraro, Lionel Mathelin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-17256.txt.
Introduction
Rémy Hosseinkhan-Boucher and co-authors improve a control method that learns from few trials, in a 2025 paper. Model predictive control plans actions for a moving system using a learned model of how it behaves. Recent learning based versions use ideas from information theory to learn faster, but each run of the system can only explore so much from its current state.
The team adds time abstraction through semi-Markov decision processes, where actions can span several steps. This raises the total information gathered from the data for the same number of samples. In effect, each trial teaches more, so the learning needs fewer trials overall. The change targets robots and other machines where real trials are costly or risky.
Catalog record
| CREATOR | Rémy Hosseinkhan-Boucher, Stella Douka, Onofrio Semeraro, Lionel Mathelin |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-17256
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Increasing Information for Model Predictive Control with Semi-Markov Decision Processes Rémy Hosseinkhan-Boucher, Stella Douka, Onofrio Semeraro, Lionel Mathelin Abstract Recent works in Learning-Based Model Predictive Control of dynamical systems show impressive sample complexity performances using criteria from Information Theory to accelerate the learning procedure. However, the sequential exploration opportunities are limited by the system local state, restraining the amount of information of the observations from the current exploration trajectory. This article resolves this limitation by introducing temporal abstraction through the framework of Semi-Markov Decision Processes. The framework increases the total information of the gathered data for a fixed sampling budget, thus reducing the sample complexity. \coltauthor ∗ ∗ Université Paris-Saclay, Orsay, France CNRS, Laboratoire Interdisciplinaire des Sciences du Numérique, Orsay, France keywords Expected Information Gain; Temporal Abstraction; Sample Complexity 1 Introduction Machine Learning Control (MLC) is an interdisciplinary area of statistical learning and control theory which solves model-free optimal control problems ( Duriez et al., 2016 ) . Among the multiple approaches of the vast field of data-driven control, two classes have received notable attention by the machine learning community: Learning-Based Model Predictive Control (LB-MPC) ( Hewing et al., 2020 ) and Model-Based Reinforcement Learning (MB-RL) ( Abbeel et al., 2006 ; Recht, 2018 ; Moerland et al., 2022 ) . The former refers to the combination of Model Predictive Control (MPC) , an optimisation method based on a sufficiently descriptive model of the system dynamics ( Grüne and Pannek, 2011 ) , and learning methods which enable the improvement of the prediction model from recorded data while possibly modeling uncertainty ( Aswani et al., 2013 ; Koller et al., 2019 ) . The latter combines general function approximators such as linear models ( Tsitsiklis and Van Roy, 1997 ) , or more generally neural networks ( Sutton et al., 1999a ) , with Dynamic Programming (DP) ( Bellman, 1957 ) principles to solve the underlying optimisation problem. * * footnotetext: Both authors contributed equally to this work. Despite the recent impressive results in learning complex dynamical models ( Ha and Schmidhuber, 2018 ) , the sample complexity of the learning process remains a major issue in the field of data-driven control ( Kakade, 2003 ; Li et al., 2021 , and see the references therein) , in which the sample complexity is defined as the sample size required to learn a good approximation of the target concept ( Mohri et al., 2018 ) . For this reason, recent works ( Mehta et al., 2022b ; Mehta et al., 2022a ) in LB-MPC have focused on the design of exploration strategies based on the Information Theory concept of Expected Information Gain (EIG) or negative Conditional Mutual Information (CMI) ( Lindley, 1956 ) . The resulting criterion allows for quantifying the gain of information given by a new state-control observation on the estimated optimal system trajectory. Hence, this tool can be used as an acquisition function to guide the exploration of the state-control space. The concept of acquisition function is borrowed from the field of Bayesian Optimisation (BO) . In particular, the work of Mehta et al. (2022b) relies on the broader black-box BO framework of Neiswanger et al. (2021) . In a setting where the data is collected along the trajectory of the dynamical system of interest, the diversity of the resulting dataset (which may be characterised by the quantity of information) is conditioned on the subsequent states of the system. Informally, the setting in which the sampling procedure is constrained by the current system state may introduce information redundancy if the system exhibits high auto-correlation or if the current state is in a slowly evolving region of the state space. Indeed, as shown in Figure 1 (auto-correlation from a perturbated fixed point of a controlled Lorenz 63’ system), the auto-correlation from an initial state can be high in average for a long period of time while the control intensity allows to reduce the correlation of the sequence of states. Figure 1: (Cov(X_{0},X_{k}))_{k\in\mathbb{N}} for the controlled Lorenz system x_{3} component under multiple control intensities. However, for dynamical systems characterised by a broad range of time scales, the notion of temporal abstraction , described in the below paragraphs, ( Precup, 2000 ; Machado et al., 2023 ) may play a key role in overcoming the issue mentioned here. Abstraction in Artificial Intelligence refers to a broad range of techniques in order to provide a more compact representation of the problem at hand ( Boutilier and Dearden, 1994 ; Banse et al., 2023 ) . In the framework of Markov Decision Process (MDP) , the work of Sutton et al. (1999b) sheds light on the limitation induced by standard MDP modeling: “There is no notion of a course of action persisting over a variable period of time. […] As a consequence, conventional MDP methods are unable to take advantage of the simplicities and efficiencies sometimes available at higher levels of temporal abstraction.” Temporal abstraction can refer to the concept of selecting the right level of time granularity to facilitate the description of the world model to achieve a given task. In simpler words, in the present case, temporal abstraction is the idea of representing and reasoning about actions and states at different time-scales and duration. In the present article, temporal abstraction through Semi-Markov Decision Processes (SMDP) modeling is introduced to improve the informativeness of the sequential exploration of the state-control space. SMDP modeling is shown to obtain a better sample complexity of the dynamics model estimator. This article thus extends the previous work of Mehta et al. (2022a) by introducing temporal abstraction to the acquisition function. The paper is organised as follows. Section 2 reviews the related works. Section 3 introduces the problem setting. Section 4 presents the hypothesis and the experimental setup while Section 5 presents the results and Section 6 concludes the paper. 2 Related Works Information Driven Model-Based Control The foundations of the Bayesian Experimental Design have been laid by the seminal work of Lindley (1956) where the author presents a measure of the information provided by an experiment. More recently, MacKay (1992) termed Expected Information Gain (EIG) a measure of the information provided by an observation allowing, in his own terms, to actively select particularly salient data points . In the field of LB-MPC, such a criterion has been used to cherry-pick the most informative state-control pair to learn the dynamics of the system ( Mehta et al., 2022b ; Mehta et al., 2022a ) . Their work is based on the broader Bayesian Optimisation method of Neiswanger et al. (2021) designed to optimise “blackbox” functions. An extensive review of Bayesian Optimisation and its applications is available in this latter paper. Learning-Based Model Predictive Control The history of learning-based models may be traced back to the seminal work by Stratonovich (1960) in probability theory which stimulated several contributions, notably the work of Kalman and Bucy (1961) , that were to compose a body of work generally referred to as filtering theory. More recently, Kamthe and Deisenroth (2018) model the system dynamics with Gaussian Processes (GP) and use MPC for data efficiency. GPs are also used in the PILCO model ( Deisenroth and Rasmussen, 2011 ) which has a