Accelerating Model-Based Reinforcement Learning using Non-Linear Trajectory Optimization
Marco Cali’, Giulio Giacomuzzo, Ruggero Carli, Alberto Dalla Libera · 2025
Read Accelerating Model-Based Reinforcement Learning using Non-Linear Trajectory Optimization by Marco Cali’, Giulio Giacomuzzo, Ruggero Carli, Alberto Dalla Libera (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-02767.txt.
Introduction
Marco Cali and co-authors wrote a 2025 paper on speeding up a learning method for robot control, where the name Cali appears with a grave accent in the byline. The base method, MC-PILCO, learns a model of how a system moves and then works out a good policy, but that step can be slow. The team pair it with a fast method that improves a path step by step.
Their combined method uses the fast path step to make useful trial paths and to start the policy off well. On the cart pole task, it cut run time by nearly half when both methods solved the task in four trials. It also kept a perfect success rate across trials. The gain held even in cases where the base method took fewer rounds to settle.
Catalog record
| CREATOR | Marco Cali’, Giulio Giacomuzzo, Ruggero Carli, Alberto Dalla Libera |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-02767
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Accelerating Model-Based Reinforcement Learning using Non-Linear Trajectory Optimization Marco Cali’, Giulio Giacomuzzo, Ruggero Carli, Alberto Dalla Libera Abstract This paper addresses the slow policy optimization convergence of Monte Carlo Probabilistic Inference for Learning Control (MC-PILCO), a state-of-the-art model-based reinforcement learning (MBRL) algorithm, by integrating it with iterative Linear Quadratic Regulator (iLQR), a fast trajectory optimization method suitable for nonlinear systems. The proposed method, Exploration-Boosted MC-PILCO (EB-MC-PILCO), leverages iLQR to generate informative, exploratory trajectories and initialize the policy, significantly reducing the number of required optimization steps. Experiments on the cart-pole task demonstrate that EB-MC-PILCO accelerates convergence compared to standard MC-PILCO, achieving up to 45.9 % \bm{45.9\%} reduction in execution time when both methods solve the task in four trials. EB-MC-PILCO also maintains a 𝟏𝟎𝟎 % \bm{100\%} success rate across trials while solving the task faster, even in cases where MC-PILCO converges in fewer iterations. I INTRODUCTION In recent years, Reinforcement Learning (RL) has emerged as a promising technology for deriving controllers for complex tasks across different domains, ranging from virtual environments to Robotics. Unlike classical control theory, which typically relies on models derived from first principles, RL derives the controller, also named policy, in a data-driven fashion. The perspective of a system that learns autonomously from experience makes RL an attractive alternative to traditional control methodologies. However, RL is not without its challenges. Key among these, in the Robotics domain, is the sample efficiency, namely, the amount of data the algorithm needs to converge to a good solution. Indeed, in actual systems data collection is a time-consuming and safety-critical operation. MBRL [ 1 ] has been proposed as a data-efficient alternative to traditional Model-Free RL (MFRL) [ 2 ] . Unlike MFRL algorithms, which estimate the value of an input directly from data, MBRL increases the information extracted from interactions by constructing a predictive model of the system. The policy is optimized on the model instead of the system, thus limiting experience on the actual system and potentially dangerous interactions. An interesting class of MBRL algorithms relies on Gaussian Process Regression (GPR) [ 3 ] to derive the system dynamics [ 4 , 5 , 6 , 7 , 8 , 9 ] . Among these solutions, Monte Carlo Probabilistic Inference for Learning COntrol (MC-PILCO) [ 5 ] is a state-of-the-art algorithm that proved particularly data-efficient in several actual applications [ 10 , 11 ] . GPR, thanks to its Bayesian formulation, returns a stochastic model that accounts for intrinsic and epistemic uncertainty, namely due to limited data availability. This fact opens the possibility of considering uncertainty during policy optimization and deriving more robust controllers. These advantages come at the price of computationally intensive and time-consuming policy optimization stages, that could limit the applicability of such algorithms from a technological point of view. In this work, we investigate the combination of MC-PILCO with iLQR [ 12 ] to reduce MC-PILCO’s convergence time. iLQR is a differential dynamic programming method that extends linear quadratic controllers to deterministic nonlinear systems by linearizing the dynamics around the current trajectory. Contrary to MC-PILCO, iLQR converges to a solution much more quickly. However, this solution is potentially local and less robust since it does not consider stochasticity. The main idea is that MC-PILCO convergence time can be reduced by using iLQR to explore the system in the neighborhood of low-cost regions and initialize the MC-PILCO policy, thus providing a warm start for MC-PILCO. To implement this strategy, we focused on two main aspects: (i) integration of Gaussian Process (GP) models in the iLQR framework [ 13 ] , (ii) definition of an effective strategy to pretrain the MC-PILCO policy from an iLQR controller. We evaluated the proposed approach on the cart-pole swing-up task through extensive simulations. The results demonstrate faster convergence compared to MC-PILCO, highlighting its potential for broader applications. We named the resulting framework Exploration-Boosted MC-PILCO (EB-MC-PILCO). II BACKGROUND This section reviews background notions on iLQR, GPR for dynamics model identification and the MC-PILCO algorithm. In the following, we consider a discrete-time dynamics system and point out with \bm{x}_{t}\in\mathbb{R}^{d_{x}} and \bm{u}_{t}\in\mathbb{R}^{d_{u}} its state and input at time t. We assume the following dynamics equation, \bm{x}_{t+1}=\bm{f}(\bm{x}_{t},\bm{u}_{t}). (1) Without loss of generality, we consider a mechanical system, and, as usual, we assume that the system state is given by \bm{x}_{t}=[\bm{q}_{t}^{\top},\bm{\dot{q}}_{t}^{\top}]^{\top} , where \bm{q}_{t} and \dot{\bm{q}}_{t} are the position and velocity of the system. II-A iLQR Iterative Linear Quadratic Regulator (iLQR) [ 12 ] optimizes control sequences for nonlinear systems by iteratively linearizing dynamics and solving a quadratic cost minimization problem. The goal is to minimize the cumulative cost: J(\bm{x}_{0},U)=\sum_{t=0}^{T-1}c(\bm{x}_{t},\bm{u}_{t})+c_{f}(\bm{x}_{T}), (2) where the running cost c(\bm{x}_{t},\bm{u}_{t}) and terminal cost c_{f}(\bm{x}_{T}) are quadratic: \displaystyle c(\bm{x}_{t},\bm{u}_{t}) \displaystyle=\frac{1}{2}\bm{x}_{t}^{\top}Q\,\bm{x}_{t}+\frac{1}{2}\bm{u}_{t}^{\top}R\,\bm{u}_{t}, (3a) \displaystyle c_{f}(\bm{x}_{T}) \displaystyle=\frac{1}{2}\bm{x}_{T}^{\top}Q_{f}\,\bm{x}_{T}. (3b) Here Q\in\mathbb{R}^{d_{x}\times d_{x}} and Q_{f}\in\mathbb{R}^{d_{x}\times d_{x}} are the state cost matrices and R\in\mathbb{R}^{d_{u}\times d_{u}} is the input cost matrix. The algorithm starts from a nominal trajectory \bm{\zeta} , generated by applying the control sequence U to the initial state \bm{x}_{0} . If a good initialization is not known, then we can set U=0 [ 14 ] . The algorithm consists of two main steps, detailed below. II-A 1 Backward Pass The backward pass computes the optimal control modification using the Q -function, which approximates the cost-to-go J_{\tau} : \displaystyle J_{\tau}(\bm{x},U_{\tau}) \displaystyle=\sum_{t=\tau}^{T-1}c(\bm{x}_{t},\bm{u}_{t})+c_{f}(\bm{x}_{T}), (4) \displaystyle Q(\delta\bm{x},\delta\bm{u}) \displaystyle=c(\bm{x}+\delta\bm{x},\bm{u}+\delta\bm{u})+V^{\prime}(\bm{f}(\bm{x}+\delta\bm{x},\bm{u}+\delta\bm{u})) (5) where V(\bm{x})=\min_{\bm{u}}\left[c(\bm{x},\bm{u})+V^{\prime}(\bm{f}(\bm{x},\bm{u}))\right] and V^{\prime} is the value function at the next time step, and \delta\bm{x}=\hat{\bm{x}}-{\bm{x}} and \delta\bm{u}=\hat{\bm{u}}-{\bm{u}} are the deviations from the nominal trajectory. The Q -function is expanded quadratically around the nominal trajectory, and its derivatives are computed: \displaystyle Q_{\bm{x}} \displaystyle=c_{\bm{x}}+f_{\bm{x}}^{\top}V_{\bm{x}}^{\prime}, (6a) \displaystyle Q_{\bm{u}} \displaystyle=c_{\bm{u}}+f_{\bm{u}}^{\top}V_{\bm{x}}^{\prime}, (6b) \displaystyle Q_{\bm{xx}} \displaystyle=c_{\bm{xx}}+f_{\bm{x}}^{\top}V_{\bm{xx}}^{\prime}f_{\bm{x}}, (6c) \displaystyle Q_{\bm{uu}} \displaystyle=c_{\bm{uu}}+f_{\bm{u}}^{\top}V_{\bm{xx}}^{\prime}f_{\bm{u}}, (6d) \displaystyle Q_{\bm{ux}} \displaystyle=c_{\bm{ux}}+f_{\bm{u}}^{\top}V_{\bm{xx}}^{\prime}f_{\bm{x}}. (6e) The optimal control modification is given by \delta\bm{u}^{\star}=\argmin_{\delta\bm{u}}Q(\delta\bm{x},\delta\bm{u})=\bm{k}+K\delta\bm{x}, (7) with open-loop term \bm{k}\triangleq-Q_{\bm{uu}}^{-1}Q_{\bm{u}} and feedback gain term K\triangleq-Q_{\bm{uu}}^{-1}Q_{\bm{ux}} . Finally, the value function and its derivatives are updated for the next time step: \displaystyle V_{\bm{x}} \displaystyle=Q_{\bm{x}}-K^{\top}Q_{\bm{uu}}\bm{k}=Q_{\bm{x}}-Q_{\bm{ux}}^{\top}Q_{\bm{uu}}^{-1}Q_{\bm{u}}, (8a) \displaystyle V_{\bm{xx}} \displaystyle=Q_{\bm{xx}}-K^{\top}Q_{\bm{uu}}K=Q_{\bm{xx}}-Q_{\bm{ux}}^{\top}Q_{\bm{uu}}^{-1}Q_{\bm{ux}} (8b) To ensure stability, Q_{\bm{uu}}^{-1} is regularized via eigendecomposition. Small eigenvalues are increased by a regularization parameter \gamma , which is lowered as cost improves. II-A 2 Forward rollout The new trajectory \bm{\zeta} is computed in the forward pass: \displaystyle\hat{\bm{x}}_{0} \displaystyle=\bm{x}_{0}, (9) \displaystyle g(\hat{\bm{x}}_{t})=\hat{\bm{u}}_{t} \displaystyle=\bm{u}_{t}+\alpha\bm{k}_{t}+K_{t}(\hat{\bm{x}}_{t}-\bm{x}_{t}), (10) \displaystyle\hat{\bm{x}}_{t+1} \displaystyle=\bm{f}(\hat{\bm{x}}_{t},\hat{\bm{u}}_{t}). (11) The backtracking parameter \alpha , initially set to 1, ensures cost improvement by selecting the largest possible step size. Hyperparameters \alpha_{f} (step size reduction rate) and \alpha_{\text{min}} (minimum step size) control convergence: higher \alpha_{f} speeds up convergence but risks oscillation, while lower \alpha_{\text{min}} allows broader exploration but may slow convergence. II-B GPR for dynamics model identification In the MBRL algorithm considered in this work the transition function ( 1 ) is identified from data by means of GPR. Function \bm{f}(\bm{x},\bm{u}) is reconstructed through d_{x} independent GPs with input \tilde{\bm{x}}_{t}=[\bm{x}_{t}^{\top}\;\bm{u}_{t}^{\top}]^{\top} , hereafter pointed out by \Delta^{(i)}(\tilde{\bm{x}}_{t}) with i=1\dots d_{x} . The i -th GP \Delta^{(i)} models the delta between the i -th components of consecutive states, namely \Delta^{(i)}(\tilde{\bm{x}}_{t})=\bm{x}_{t+1}^{(i)}-\bm{x}_{t}^{(i)}. (12) By definition of GP, \Delta^{(i)}(\cdot)\sim GP(m(\cdot),k(\cdot,\cdot)) where m(\cdot) is the prior mean, while the kernel function k(\cdot,\cdot) is a function that defines the prior covariance between \Delta^{(i)}(\tilde{\bm{x}}_{t}) and \Delta^{(i)}(\tilde{\bm{x}}_{p}) , that is, k(\tilde{\bm{x}}_{t},\tilde{\bm{x}}_{p})=Cov[\Delta^{(i)}(\tilde{\bm{x}}_{t}),\Delta^{(i)}(\tilde{\bm{x}}_{p})] . In the following, we will adopt the compact notation K(\tilde{{X}},\tilde{{X}}) to point out the kernel matrix associated to a set of input