Continuous-time iterative linear-quadratic regulator

Juraj Lieskovský, Jaroslav Bušek, Tomáš Vyhlídal · 2025

Read Continuous-time iterative linear-quadratic regulator by Juraj Lieskovský, Jaroslav Bušek, Tomáš Vyhlídal (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-15525.txt.

Introduction

Juraj Lieskovsky and co-authors present a continuous time version of a well known method for planning optimal motion, in a paper from 2025. The method improves a path step by step by solving a series of simpler squared cost problems. Their version works directly in continuous time rather than in fixed steps.

A backtracking search keeps each update safe, and a new regularization trick is built into the core pass of the method. The design can use modern solvers that adapt their step size, so step counts vary as needed. On the classic cart and pole swing up task, the method handled a non convex cost well. The authors say it balances accuracy against computing effort.

Catalog record

CREATORJuraj Lieskovský, Jaroslav Bušek, Tomáš Vyhlídal
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE27 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-15525

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Continuous-time iterative linear-quadratic regulator Thanks: This work was supported by the project 25–16357S of the Czech Science Foundation and by the European Union under the project Robotics and advanced industrial production No. CZ.02.01.01/00/22_008/0004590. The first and the second author also acknowledge support from the Grant Agency of the Czech Technical University in Prague, student grant No. SGS23/157/OHK2/3T/12. Preprint submitted to IEEE Juraj Lieskovský, Jaroslav Bušek, Tomáš Vyhlídal Abstract We present a continuous-time equivalent to the well-known iterative linear-quadratic algorithm including an implementation of a backtracking line-search policy and a novel regularization approach based on the necessary conditions in the Riccati pass of the linear-quadratic regulator. This allows the algorithm to effectively solve trajectory optimization problems with non-convex cost functions, which is demonstrated on the cart-pole swing-up problem. The algorithm’s compatibility with state-of-the-art suites of numerical integration solvers allows for the use of high-order adaptive-step methods. Their use results in a variable number of time steps both between passes of the algorithm and across iterations, maintaining a balance between the number of function evaluations and the discretization error. I Introduction The development of non-linear trajectory optimization methods has been highly impactful in recent years, making approaches such as nonlinear model predictive control of highly non-linear underactuated systems such as legged robots and autonomous drones possible. The problem can be characterized as minimizing the predicted “cost” of a system’s behavior, when evaluated over a finite time horizon, starting from a single initial condition. Methods for solving this problem can generally be categorized as either direct or indirect. Direct methods transcribe the trajectory optimization problem into a finite-dimensional nonlinear programming problem, which can then be solved using general-purpose or specialized solvers. Indirect methods, on the other hand, formulate necessary conditions of optimality which a solution to the optimal control problem has to satisfy, dividing each iteration of the algorithm into multiple sub-problems that are solved sequentially. The most prominent representative of indirect methods is the differential dynamic programming algorithm, derived for systems described in the discrete time domain. Introduced in [ 1 ] the algorithm has experienced a resurgence in popularity. Compared to direct methods, it efficiently exploits the problem’s Markovian structure and, as a by-product of the optimization process, produces a local feedback controller. The latter allows for the stabilization of precomputed trajectories during execution, which can be taken advantage of in nonlinear model predictive control where the recalculation of the optimal trajectory may be performed over multiple periods of the stabilizing controller’s actions. Due to the time-critical nature of nonlinear model predictive control a variant of the algorithm known as the iterative linear-quadratic regulator, first presented in [ 2 ] , has become the standard in solvers [ 3 , 4 , 5 ] . Here it is also pertinent to mention the sequential linear-quadratic algorithm presented in [ 6 ] which is a lesser know equivalent of the iterative linear-quadratic regulator algorithm. Compared to the differential dynamic programming algorithm, iterative linear-quadratic regulator does not calculate terms containing second-order partial derivatives of the system’s dynamics. As a result the algorithm is not guaranteed to converge at the same rate but is computationally faster [ 2 , 7 ] . While originally both the iterative linear-quadratic regulator and sequential linear-quadratic were derived based on Hamiltonian dynamics, the iterative linear-quadratic regulator algorithm was later related to differential dynamic programming and the Bellman principle in [ 8 ] . A major disadvantage of the differential dynamic programming and iterative linear-quadratic regulator algorithms, compared to direct optimization methods, is their inability to handle constraints placed on states and inputs of the system and infeasible initial trajectories. Contemporary research has focused on addressing both. In [ 9 ] an approach to handling box-constrained inputs was incorporated, partially addressing the former, while a multiple-shooting variant of the iterative linear-quadratic regulator algorithm was introduced in [ 10 ] , addressing the latter. An augmented Langrangian approach to handling equality and inequality constraints was introduced in [ 3 ] , also allowing for infeasible initial trajectories through the use of slack variables. A similar approach to that in [ 10 ] was also implemented in [ 4 ] for the full differential dynamic programming algorithm and later extended in [ 11 ] with the input constraint handing introduced in [ 9 ] . The previously mentioned works have focused on optimizing trajectories of systems described in the discrete time domain, resorting to discretizing continuous-time systems using fixed timestep numerical integration. Comparatively, algorithms that produce continuous trajectories and inputs have been overlooked. As one of two notable exceptions, we may consider the sequential linear-quadratic algorithm with an augmented Langrangian approach to handling equality constraints that was introduced in [ 12 ] and later extended to handle inequality constraints in [ 13 ] . The second exception is the continuous-time differential dynamic programming algorithm presented in [ 14 ] . Developed to handle terminal constraints in [ 14 ] , it was later modified for a game-theoretic formulation of the trajectory optimization problem in [ 15 ] . While these works present interesting problem specific extensions, they do not address crucial topics such as regularization or the problem’s numerical properties and efficient implementation. I-A Contributions In this paper we provide a concise derivation of a continuous-time equivalent to the standard iterative linear-quadratic regulator algorithm, including terms of the full differential dynamic programming algorithm. A novel regularization approach, based on the necessary conditions in linear-quadratic regulator, is presented and a line-search policy is described. We also discuss details required for efficiently utilizing numerical integration during both the backward and forward pass of the algorithm and address the potential loss of certain properties due to numerical errors. The ability of the algorithm to converge to locally optimal policies is validated on two variants of the cart-pole swing-up problem. II Continuous-time trajectory optimization problem The problem of optimizing a continuous-time system’s trajectory over a finite horizon can be stated as finding the trajectory x(\tau)\in\mathbb{R}^{n_{x}} and control policy u(\tau)\in\mathbb{R}^{n_{u}} that solve the optimization problem \displaystyle\min_{x(\tau),u(\tau)} \displaystyle\quad\Phi(x(t_{f}))+\int_{t_{0}}^{t_{f}}l(x(\tau),u(\tau),\tau)\,\mathrm{d}\tau subject to \displaystyle\quad\dot{x}(\tau)=f(x(\tau),u(\tau),\tau),\quad\tau\in\langle t_{0},t_{f}\rangle \displaystyle\quad x(t_{0})=x_{0} where x_{0} is the initial state and f(x,u,t) are the continuous-time dynamics of the system. The problem’s objective function, made up of the final cost \Phi(x) and the running cost l(x,u,t) , is in the context of trajectory optimization referred to as the total cost. Bellman’s principle of optimality introduces additional quantities related to the total cost, namely the cost-go-to and the value function. The cost-to-go is defined as the “remaining cost” from any arbitrary state x(t) , t\in\langle t_{0},t_{f}\rangle : J(x(t),u(\tau),t)=\Phi(x(t_{f}))+\int_{t}^{t_{f}}l(x(\tau),u(\tau),\tau)\,\mathrm{d}\tau (1) attained by applying the control policy u(\tau) , \tau\in\langle t,t_{f}\rangle . The value function is consequently defined as the cost-to-go for the optimal control policy u^{*}(\tau) from the state x(t) , resulting in the system following the optimal trajectory x^{*}(\tau) : V(x(t),t)=J(x^{*}\!(\tau),u^{*}\!(\tau),t). (2) The value function then figures in the Hamilton-Jacobi-Bellman equation and the related necessary condition of optimality \displaystyle-\frac{\partial V}{\partial t}(x(t),t) \displaystyle=\min_{u(t)}\,Q(x(t),u(t),t) (3a) \displaystyle u^{*}\!(t) \displaystyle=\arg\min_{u(t)}\,Q(x(t),u(t),t), (3b) where Q(x,u,t)=\\ l(x,u,t)+{\left(\frac{\partial V}{\partial x}\right)}^{\top}\!\!(x,t)\,f(x,u,t). III Proposed Algorithm Similarly to its discrete-time counterpart, each iteration of our algorithm is centered around a dynamically feasible nominal trajectory \bar{x}(\tau) attained by applying the nominal control policy \bar{u}(\bar{x}(\tau),\tau) , where \tau\in\langle t_{0},t_{f}\rangle . Every iteration

Read the complete text