Safety Reinforced Model Predictive Control (SRMPC): Improving MPC with Reinforcement Learning for Motion Planning in Autonomous Driving

Johannes Fischer, Marlon Steiner, Ömer Şahin Taş, Christoph Stiller · 2025

Read Safety Reinforced Model Predictive Control (SRMPC): Improving MPC with Reinforcement Learning for Motion Planning in Autonomous Driving by Johannes Fischer, Marlon Steiner, Ömer Şahin Taş, Christoph Stiller (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-03774.txt.

Introduction

Motion planning for self-driving cars often relies on model predictive control, which optimises a short stretch of driving again and again. Johannes Fischer and colleagues note a flaw in a 2025 robotics preprint. Real-time planning uses convex approximations, and the best solution may lie outside the simplified region being searched. Their fix lets learning widen the search without giving up safety.

The proposed method, Safety Reinforced Model Predictive Control, uses safe reinforcement learning to propose a fresh reference trajectory for the controller. A constrained learning setup keeps that proposal inside safe regions, defined through a handcrafted energy-based index. A multiplier that adapts with the state is learned alongside the policy, enforcing the safety constraint during training. In a highway scenario, the combined planner beat both plain predictive control and learning alone on safety and performance measures.

Catalog record

CREATORJohannes Fischer, Marlon Steiner, Ömer Şahin Taş, Christoph Stiller
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-03774

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Safety Reinforced Model Predictive Control (SRMPC): Improving MPC with Reinforcement Learning for Motion Planning in Autonomous Driving Johannes Fischer, Marlon Steiner, Ömer Şahin Taş, Christoph Stiller Abstract Model predictive control (MPC) is widely used for motion planning, particularly in autonomous driving. Real-time capability of the planner requires utilizing convex approximation of optimal control problems (OCPs) for the planner. However, such approximations confine the solution to a subspace, which might not contain the global optimum. To address this, we propose using safe reinforcement learning (SRL) to obtain a new and safe reference trajectory within MPC. By employing a learning-based approach, the MPC can explore solutions beyond the close neighborhood of the previous one, potentially finding global optima. We incorporate constrained reinforcement learning (CRL) to ensure safety in automated driving, using a handcrafted energy function-based safety index as the constraint objective to model safe and unsafe regions. Our approach utilizes a state-dependent Lagrangian multiplier, learned concurrently with the safe policy, to solve the CRL problem. Through experimentation in a highway scenario, we demonstrate the superiority of our approach over both MPC and SRL in terms of safety and performance measures. I Introduction Autonomous vehicles have the potential to fundamentally change traffic by reducing risky driving behavior, congestion, carbon dioxide emissions, and transportation costs, as well as improving road safety [ 1 ] . An important area in the field of autonomous driving is motion planning, which aims to generate optimal trajectories in a given state that at the same time do not pose a threat to road traffic. Motion planning is often performed by model predictive control (MPC) [ 2 , 3 , 4 , 5 , 6 ] . For real-time applications, linear time-varying MPC (LTV-MPC) is often preferred because of the ability to faster solution calculation [ 7 , 8 ] . Though, LTV-MPC only performs a local optimization which leads to the fact that only a locally optimal solution can be found. Recent studies also focus on motion planning and control with reinforcement learning (RL). In RL applications it is a huge challenge to address constraint satisfaction, especially in safety critical applications like autonomous driving. There is a lot of research trying to achieve safety in RL, which is called safe RL (SRL) [ 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 ] . Most of the latest approaches are not tested on driving tasks yet, and they often require further research. However, full autonomous driving is still confronted with the task to make feasible and especially safe decisions and planning in complex and uncertain dynamic traffic scenarios. In order to address the existing issues, we present a novel approach which combines both methods, MPC and RL, to improve the motion planning in terms of safety and performance. This leads to a novel method that combines the mathematical optimization approach with a learning-based approach. Our main contributions are: • The application of a SRL approach with state dependent Lagrange multiplier and safety index for autonomous driving tasks. • Utilization of SRL for switching between local optima and ideally finding the global optimum, which serves as the reference for the locally optimizing MPC. • Experimental results on a highway scenario, demonstrating that the proposed method achieves better results in terms of safety and performance than SRL and LTV-MPC. II Related Work To the best of our knowledge there are no other approaches, which use RL to improve the motion planning of an MPC controller by generating a reference trajectory. There are several approaches of so called learning-based MPC , which use machine learning to learn dynamic models or parameters of the MPC controller [ 17 , 18 , 19 , 20 , 21 ] . RL in combination with MPC is often used to imitate the control strategy of the MPC by end-to-end learning [ 22 , 23 , 24 ] . However, approaches where the MPC benefits from the learned value function of the RL algorithm can rarely be found in the literature. The authors in [ 25 ] integrate the RL value function into the objective function of the MPC. But they solved the optimal control problem (OCP) in a combinatorial way and not analytically. This makes the proposed approach only applicable for problems with a small and discrete action space. Another approach which is closely related to ours is presented in [ 26 ] . In this paper the authors use RL to define a terminal area for the MPC, so that the controlled vehicle can be guided through an unknown environment. In contrast to [ 26 ] , we do not define a terminal area but use RL to generate a whole trajectory as a reference respectively initial solution for the OCP. Furthermore, the learned trajectory in our work is not forced to be tracked by the MPC but can shift the optimization problem from one locally optimal solution to another. III Fundamentals In this section, we formalize the definition of the MPC and of the RL problem as a constrained Markov decision process (CMDP) and present the algorithms that are used to solve it. III-A MPC and Linear Time-Varying MPC MPC is a control strategy that predicts future system behavior and determines optimal control actions \bm{u} based on the modeled system dynamics. It computes control actions for a finite time horizon by solving an OCP, but only applying the first action to the system. The process is repeated at every time step, taking into account new measurements and updated predictions for the state \bm{x} . Due to often nonlinear constraints in the OCP within an autonomous driving task, a nonlinear MPC (NMPC) for the control of such a system is required. The NMPC solves an optimization problem \displaystyle\underset{\bm{u}_{1:N},\bm{x}_{1:N+1}}{\text{min}}\quad \displaystyle J(\bm{u}_{1:N},\bm{x}_{1:N+1}) (1a) s.t. \displaystyle\bm{x}_{k+1}=\mathbf{f}(\bm{x}_{k},\bm{u}_{k}),\ k=1,\ldots,N, (1b) \displaystyle\bm{x}_{1}=\hat{\bm{x}}, (1c) \displaystyle\underline{\mathbf{g}_{\text{col}}}\leq\mathbf{g}_{\text{col}}(\bm{x}_{k},\bm{u}_{k})\leq\overline{\mathbf{g}_{\text{col}}}, \displaystyle\phantom{---------}k=2,\ldots,N\!+\!1, (1d) \displaystyle\underline{\bm{u}}\leq\bm{u}_{k}\leq\overline{\bm{u}},\ k=1,\ldots,N, (1e) \displaystyle\underline{\bm{x}}\leq\bm{x}_{k}\leq\overline{\bm{x}},\ k=1,\ldots,N\!+\!1, (1f) in every time step specified by the objective function ( 1a ), the system dynamics ( 1b ), the initial state condition ( 1c ), collision avoidance constraints ( 1d ) denoted by general collision functions \mathbf{g}_{\mathrm{col}} and box constraints for the control and state variables ( 1e )–( 1f ). In the OCP ( 1 ) the variable N represents the prediction horizon length. Applying NMPC to an on-board system is challenging due to real-time constraints and the computational complexity of the optimization problem. LTV-MPC addresses this challenge by incorporating time-varying dynamics through the linearization of the optimization problem constraints at every time step. Together with a quadratic objective function to optimize, the problem becomes convex and a quadratic programming (QP) solver can be applied to solve it. However, this leads to the fact that only a locally optimal solution is calculated which depends on the trajectory used for linearization. III-B Constrained Markov Decision Process Analogous to a Markov decision process (MDP) [ 27 ] , a CMDP [ 28 ] is a model that describes a problem of sequential decision-making between an agent and its environment. The difference is that the CMDP allows a clear separation of reward and safety signals. A CMDP can be defined as a tuple (\mathcal{S},\mathcal{A},\mathcal{P}_{0},r,c,d,\gamma) , where \mathcal{S} is the state space, \mathcal{A} is the

Read the complete text