Sampling-based Model Predictive Control Using Trust Regions

Markus Walker, Marcel Reith-Braun, Daniel Frisch, Uwe D. Hanebeck · 2026

Read Sampling-based Model Predictive Control Using Trust Regions by Markus Walker, Marcel Reith-Braun, Daniel Frisch, Uwe D. Hanebeck (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-07801.txt.

Introduction

Markus Walker and three co-authors bring trust regions to sampling-based model predictive control, the family that includes the popular MPPI algorithm. These controllers sample candidate action sequences, score their trajectories, and update the sampling distribution, but adjusting knobs like temperature or momentum has relied on heuristics and hand tuning.

Their setup bounds each update of the proposal distribution by a set divergence limit, with an entropy floor if wanted. The adaptation values then come out of the optimization rather than a rule of thumb. Deterministic local sampling further improves speed. On two benchmark environments the trust region version converged faster and used samples better in low-sample, low-iteration regimes, exactly where sampling controllers struggle.

Catalog record

CREATORMarkus Walker, Marcel Reith-Braun, Daniel Frisch, Uwe D. Hanebeck
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.07801

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Sampling-based Model Predictive Control Using Trust Regions Thanks: This work is part of the German Research Foundation (DFG) AI Research Unit 5339 regarding the combination of physics-based simulation with AI-based methodologies for the fast maturation of manufacturing processes. Thanks: Markus Walker, Marcel Reith-Braun, Daniel Frisch and Uwe D. Hanebeck are with the Intelligent Sensor-Actuator-Systems Laboratory (ISAS), Institute for Anthropomatics and Robotics, Karlsruhe Institute of Technology, Germany (e-mail: { firstname.lastname } @kit.edu ). Markus Walker, Marcel Reith-Braun, Daniel Frisch, Uwe D. Hanebeck Abstract Sampling-based model predictive control algorithms, such as model predictive path integral, enable approximate, gradient-free solutions to optimal control problems by drawing samples from a proposal distribution, evaluating their trajectory costs, and updating the proposal parameters accordingly. However, these approaches typically rely on heuristics for adjusting hyperparameters, such as temperature or momentum, or manual tuning. We propose a trust region formulation for sampling-based model predictive control that constrains updates of the proposal distribution via a principled Kullback–Leibler divergence bound and, optionally, an entropy lower bound. This replaces heuristic hyperparameter adaptation with values that are optimal w.r.t. the underlying Lagrangian. We further improve sample efficiency and convergence by combining the trust region update with deterministic localized cumulative distribution-based sampling. Experiments on two benchmark environments demonstrate that the proposed trust region update achieves faster convergence and better sample efficiency in low-sample and low-iteration regimes, especially when paired with deterministic localized cumulative distribution-based sampling. I Introduction Sampling-based model predictive control methods, such as cross-entropy method–model predictive control [ 1 ] and model predictive path integral [ 2 ] , have gained popularity for solving complex optimal control problems due to their ability to efficiently parallelize computations and handle arbitrary cost functions and system dynamics. These methods iteratively refine a parameterized proposal probability density function over control sequences by drawing samples, evaluating their trajectory costs, and updating the proposal parameters accordingly. Importance-weighting-based variants such as standard model predictive path integral [ 2 ] , and iterative extensions thereof [ 3 ] assign weights to the sampled control sequences based on their associated trajectory costs, thereby using a temperature parameter to control the sharpness of the weighting. This parameter is typically held constant or adjusted heuristically during optimization [ 4 ] . Similarly, cross-entropy method-based methods rely on momentum-style smoothing [ 5 ] to prevent premature collapse of the proposal distribution. These heuristics are often difficult to tune and may compromise convergence or performance across different problem settings. Motivated by the success of trust region methods in reinforcement learning [ 6 , 7 ] , where principled Kullback–Leibler-constrained updates have largely replaced heuristic parameter tuning, we propose to transfer these ideas to sampling-based model predictive control. Specifically, for each proposal update, we constrain the new proposal to remain within a bounded divergence from the previous one. The resulting constrained optimization problem is solved via Lagrange multipliers, with optimal hyperparameters obtained from the dual formulation. The updated proposal is then projected back onto the original distribution family (e.g., Gaussian), yielding a feasible distribution from which control sequences are sampled in the subsequent iteration. Orthogonal to the choice of update rule, the sampling method directly affects the quality and smoothness of the control. In our previous work [ 8 , 9 ] , we demonstrated that exchanging random samples with precomputed, optimally-placed deterministic samples based on localized cumulative distributions [ 10 ] yield improved sample efficiency and smoother control sequences. localized cumulative distribution-based sampling shows superior coverage and inherently positions the samples homogeneously, i.e., it induces a smooth distribution of the sizes of the sample-free gaps between them. It is therefore well-suited for integration and optimization tasks. In particular, it integrates well with trust region updates, which restrict the proposal to remain close to the original and thus to regions with high sample coverage. Contribution We propose a trust region formulation for sampling-based model predictive control that replaces heuristic proposal adaptations with principled Kullback–Leibler-constrained updates derived from dual optimization. Building on our previous work [ 8 , 9 ] , we further combine the trust region update with deterministic localized cumulative distribution-based sampling to improve sample efficiency. Both contributions are evaluated on two benchmark control tasks and compared against heuristic baselines across multiple sampling strategies, including random and low-discrepancy sampling (Sobol [ 11 ] and Halton [ 12 ] ). Notation In this paper, underlined letters, e.g., {\underline{x}} , denote vectors, boldface letters, such as {\boldsymbol{\underline{x}}} , represent random variables, while boldface capital letters, such as {{\mathbf{A}}} , indicate matrices. The expectation of a random variable is denoted by \hat{\cdot} , e.g., \hat{{\underline{x}}} , and covariance matrices are denoted by {{\mathbf{C}}} . II Background We consider a finite-horizon optimal control problem with cumulative cost \displaystyle J_{k} \displaystyle=g_{N_{\mathrm{H}}}({\underline{x}}_{k+N_{\mathrm{H}}})+\sum_{n=0}^{N_{\mathrm{H}}-1}g_{n}({\underline{x}}_{k+n},{\underline{u}}_{k+n})\kern 5.0pt, (2) where k denotes the current time step, N_{\mathrm{H}} the prediction horizon, and n the time step within the prediction horizon. Furthermore, {\underline{x}}_{k}\in\mathds{R}^{d_{x}} represents the system state, and {\underline{u}}_{k}\in\mathds{R}^{d_{u}} the control input. The stage cost is given by g_{n}(\cdot,\cdot)\colon\mathds{R}^{d_{x}}\times\mathds{R}^{d_{u}}\to\mathds{R} , and the terminal cost by g_{N_{\mathrm{H}}}(\cdot)\colon\mathds{R}^{d_{x}}\to\mathds{R} . The discrete-time system dynamics are described by \displaystyle{\underline{x}}_{n+1} \displaystyle={\underline{a}}_{n}({\underline{x}}_{n},{\underline{u}}_{n})\kern 5.0pt,\quad n=k,\ldots,k+N_{\mathrm{H}}-1\kern 5.0pt, (3) with transition function {\underline{a}}_{k}(\cdot,\cdot)\colon\mathds{R}^{d_{x}}\times\mathds{R}^{d_{u}}\to\mathds{R}^{d_{x}} . The objective is to determine the optimal control sequence {\underline{u}}_{k:k+N_{\mathrm{H}}-1}^{*}=\left({\underline{u}}_{k}^{*},{\underline{u}}_{k+1}^{*},\ldots,{\underline{u}}_{k+N_{\mathrm{H}}-1}^{*}\right) that minimizes J_{k} . From this optimal sequence, the first control input {\underline{u}}_{k}^{*} is applied to the system, and the optimization is repeated at the next time step k+1 in a receding horizon fashion. For compact notation, we represent the stacked control sequence {\underline{u}}_{k:k+N_{\mathrm{H}}-1} by {\underline{\xi}}\in\mathds{R}^{d_{{\xi}}} , i.e., {\underline{\xi}}=[{\underline{u}}_{k}^{\top},{\underline{u}}_{k+1}^{\top},\ldots,{\underline{u}}_{k+N_{\mathrm{H}}-1}^{\top}]^{\top}\in\mathds{R}^{N_{\mathrm{H}}\cdot d_{u}} (omitting time index k from now on for brevity) and write the associated cumulative cost as J({\underline{\xi}}) . In sampling-based model predictive control, this optimal control problem is solved iteratively by drawing N control sequences \{{\underline{\xi}}^{(i)}\}_{i=1}^{N} from a proposal probability density function f({\underline{\xi}};{\underline{\theta}}_{j}) , parameterized by {\underline{\theta}}_{j} at iteration j=0,\dots,j_{\mathrm{max}}-1 . For each sampled control sequence {\underline{\xi}}^{(i)} , the corresponding state trajectory is predicted using the system dynamics, its cost is evaluated, and the proposal is subsequently updated based on all trajectories and their costs. A common choice for the proposal in the considered Euclidean action space is a Gaussian distribution \mathcal{N}({\underline{\xi}};\hat{{\underline{\xi}}}_{j},{{\mathbf{C}}}_{j}) , where the mean \hat{{\underline{\xi}}}_{j} and covariance {{\mathbf{C}}}_{j} constitute the parameters collected in {\underline{\theta}}_{j} . III Related Work III-A Sampling-based MPC A prominent approach to sampling-based model predictive control is the cross-entropy method, which iteratively updates a proposal probability density function by selecting low-cost elite samples and maximizing their likelihood [ 5 ] . For Gaussian proposals, the updated moments for the next iteration are computed as the empirical mean and covariance of the selected control sequences [ 5 ] . To improve numerical robustness and reduce premature concentration of the proposal, many implementations introduce a momentum (or smoothing) term across iterations [ 5 , 1 ] . For the mean update, this is typically written as \displaystyle\hat{{\underline{\xi}}}_{j+1}=\alpha\,\hat{{\underline{\xi}}}_{j}+(1-\alpha)\,\hat{{\underline{\xi}}}_{j}^{\prime}\kern 5.0pt, (4) where \hat{{\underline{\xi}}}_{j}^{\prime} denotes the (unsmoothed) mean of the elite set. Related model predictive path integral-like methods also perform (iterative) proposal refinement, but they rely on importance weighting of sampled control sequences instead of hard elite selection [ 9 ] . The weights are given by \displaystyle w^{(i)}\propto\exp\left(-\frac{1}{\lambda}J\left({\underline{\xi}}^{(i)}\right)\right)\kern 5.0pt, (5) and are used to compute the updated Gaussian moments as the control sequences’ weighted sample mean and covariance. The inverse temperature \lambda

Read the complete text