DATA-DRIVEN PRONTO : a Model-free Solution for Numerical Optimal Control

Marco Borghesi, Lorenzo Sforni, Giuseppe Notarstefano · 2025

Read DATA-DRIVEN PRONTO : a Model-free Solution for Numerical Optimal Control by Marco Borghesi, Lorenzo Sforni, Giuseppe Notarstefano (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-15465.txt.

Introduction

Marco Borghesi, Lorenzo Sforni, and Giuseppe Notarstefano gave a model-free method for optimal control in two thousand twenty-five. Optimal control seeks the best inputs to steer a system, but for many real systems no good model is known. Their setting allows repeated tests or simulations on the system itself.

Their algorithm, Data-Driven PRONTO, refines a trial solution step by step. At each step it perturbs the path, gathers the resulting runs under a tracking controller, and builds a local linear view from that data. They prove it converges to a small ball around a true optimum, with size set by the dither signal. Tests on an underactuated robot backed the theory.

Catalog record

CREATORMarco Borghesi, Lorenzo Sforni, Giuseppe Notarstefano
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE73 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-15465

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DATA-DRIVEN PRONTO : a Model-free Solution for Numerical Optimal Control Marco Borghesi, Lorenzo Sforni, Giuseppe Notarstefano Abstract This article addresses the problem of data-driven numerical optimal control for unknown nonlinear systems. In our scenario, we suppose to have the possibility of performing multiple experiments (or simulations) on the system. Experiments are performed by relying on a data-driven tracking controller able to steer the system towards a desired reference. Our proposed DATA-DRIVEN PRONTO algorithm iteratively refines a tentative solution of the optimal control problem by computing an approximate descent direction via a local trajectory perturbation. At each iteration, multiple trajectories are gathered by perturbing the current trajectory with a suitable dither signal, and then used to obtain a data-driven, time-varying linearization. The exploration is guided by the tracking controller, so that perturbed trajectories are obtained in closed loop. We show local convergence of DATA-DRIVEN PRONTO to a ball about an isolated optimal solution, whose radius depends on the amplitude of the dither signal. We corroborate the theoretical results by applying it to an underactuated robot. I Introduction Nonlinear optimal control problems are prevalent in various applications within the fields of Automation and Robotics, where the goal is to develop a control strategy that, when applied to a dynamical system, minimizes a specified performance index [ 1 , 2 , 3 , 4 ] . In general, the solution of Non-Linear Optimal Control Problems (NL-OCPs) strongly relies on a valid model of the system under control which, if inaccurate, can lead to the design of trajectories which are suboptimal for the true system. In this paper, we propose a combination of identification techniques together with optimal control which can solve a NL-OCP in a model-free setting. In particular, we focus on solving a finite-horizon NL-OCP where the dynamics is unknown but there is the possibility of performing multiple deterministic experiments (or simulations) from the same initial condition. Retrieving an optimal trajectory for this kind of setups is important, for example, in several industrial systems, where optimized execution of a repeated task results in significant savings. Pioneers in this field were the so called Repetitive Control (RC) and Iterative Learning Control (ILC) [ 5 , 6 ] by means of multiple repetitions of that specific tracking problem. Recent works address the problem of including a cost function in the ILC framework and provide suboptimality guarantees for this type of approach, [ 7 , 8 ] . Besides RC and ILC, we provide an overview of two fields that deal with the problem of data-driven optimal control, distinguishing between the Reinforcement Learning (RL) and Data-driven optimization approaches (where, since the two fields have lots in common, we stress the fact that this distinction is done arbitrarily, and several works may be placed in both categories). Reinforcement Learning Rooted in the principles of dynamic programming [ 9 ] , the field of RL addresses the problem of learning a control policy that minimizes (maximizes) a cumulative cost (reward) through interaction with the environment [ 10 ] . Learning occurs, in general, during a dedicated training phase, where the agent explores the environment and progressively associates good actions with rewards and bad actions with penalties. An early reference in this field is [ 11 ] , where the authors propose to solve the optimal control problem by imposing a linear parametrization of the value function, and finding its parameters via a linear program constrained by collected data samples. In [ 12 ] and references therein, Neural Networks (NN) are employed to parametrize the control policy, and gradient descent is used to find the optimal network weights. In [ 13 ] , the authors demonstrate through a practical application how combining a classical technique like ILC with RL can significantly reduce the amount of data required for the training. More recently, [ 14 ] explores the novel idea of using RL to obtain the open-loop optimal input, rather than deriving a feedback control policy. The challenge of guaranteeing the safety during the training phase is addressed, in general, by including a controller which leverages approximate system knowledge [ 15 , 16 , 17 ] to guarantee constraint satisfaction. A research field strictly related to data-driven optimal control is model predictive control, in which data-driven methods take into account the fact that the system [ 18 , 19 , 20 , 21 ] , or some of the constraints [ 22 ] may be unknown. Data-driven optimization Parallel to the RL paradigm, numerical optimization techniques capable of handling problems with parametric uncertainties were developed throught the years. Extremum seeking, black-box optimization, derivative-free optimization and simulation optimization [ 23 , 24 , 25 , 26 , 27 ] are techniques that solve an optimization problem when the explicit cost function, its derivatives, or the constraints are not available to the optimization process. In general, in these cases the idea is to substitute the missing knowledge by cleverly probing the cost function, possibly exploiting the knowledge of some known property (for example, in [ 28 , 29 , 30 ] the authors leverage on Lipschitz continuity). These methods require in general a high number of experiments, since they do not exploit the particular structure of NL-OCPs, which has been instead studied by several model-based iterative and efficient algorithms [ 1 , 2 , 3 , 4 ] . This peculiar structure is leveraged in [ 31 ] , where the authors propose an algorithm to iteratively solve a NL-OCP requiring only partial knowledge of the dynamics. We refer the reader to [ 32 , 33 , 34 ] for other ways of solving in a data-driven way NL-OCPs. Article Contribution The main contribution given in this paper is DATA-DRIVEN PRONTO , an algorithm for nonlinear optimal control extending the applicability of PRONTO [ 35 ] to the model-free scenario. In particular, DATA-DRIVEN PRONTO leverages the possibility to probe the system dynamics in a safe way via the application of an independently-designed control law. At each algorithm iteration, we perform L experiments (simulations) in which we use the given controller plus an exploration dither to gather data in the neighbourhood of the current trajectory. These data are then used to identify the dynamics linearizations about the current trajectory, substituting the need for a model in the optimization process. The proposed algorithm achieves the following desirable features: i) avoiding parametrization errors: DATA-DRIVEN PRONTO uses data to identify local first-order approximations of the dynamics, thus avoiding errors due to a global, inexact parametrization of the nonlinear dynamics or optimal policy; ii) being model-free: DATA-DRIVEN PRONTO itself does not require any knowledge of the system; however, partial system knowledge can be used to enhance the numerical stability of the algorithm, e.g., improving the design of a closed-loop control law; iii) being data-efficient: thanks to the efficient algorithm structure, the number of required experiments (simulations) needed is comparable, at each algorithm iteration, with the system dimension. We provide theoretical guarantees on the algorithm convergence properties, together with insights on how to choose its parameters. The proposed algorithm is proved to be locally uniformly ultimately bounded to a ball about the optimal solution whose radius depends on the amplitude of the dither used to achieve successful

Read the complete text