Tempered Sequential Monte Carlo for Trajectory and Policy Optimization with Differentiable Dynamics

Heng Yang · 2026

Read Tempered Sequential Monte Carlo for Trajectory and Policy Optimization with Differentiable Dynamics by Heng Yang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-21456.txt.

Introduction

Heng Yang casts control design as inference for path and policy tuning under smooth dynamics. Lowering a KL-regularized average path cost yields a Boltzmann-tilted spread over control settings that focuses on low-cost solutions as cooling proceeds. Tempered sequential Monte Carlo anneals samples from a prior to the sharp, multi-peaked target, adjusting weights and resampling with Hamiltonian Monte Carlo refresh using rollout slopes. Policy work pins down the initial-state spread and treats rollout randomness as extra variables; benchmarks show TSMC compares well with rivals.

Catalog record

CREATORHeng Yang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE126067
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.21456

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Tempered Sequential Monte Carlo for Trajectory and Policy Optimization with Differentiable Dynamics Heng Yang Abstract We propose a sampling-based framework for finite-horizon trajectory and policy optimization under differentiable dynamics by casting controller design as inference. Specifically, we minimize a KL-regularized expected trajectory cost, which yields an optimal “Boltzmann-tilted” distribution over controller parameters that concentrates on low-cost solutions as temperature decreases. To sample efficiently from this sharp, potentially multimodal target, we introduce tempered sequential Monte Carlo (TSMC): an annealing scheme that adaptively reweights and resamples particles along a tempering path from a prior to the target distribution, while using Hamiltonian Monte Carlo rejuvenation to maintain diversity and exploit exact gradients obtained by differentiating through trajectory rollouts. For policy optimization, we extend TSMC via (i) a deterministic empirical approximation of the initial-state distribution and (ii) an extended-space construction that treats rollout randomness as auxiliary variables. Experiments across trajectory- and policy-optimization benchmarks show that TSMC is broadly applicable and compares favorably to state-of-the-art baselines. I Introduction Consider a discrete-time dynamical system \mathchar 29048_{\mathchar 29044\mathchar 8235\mathchar 28721}\mathchar 12349\mathchar 29030\delimiter 67273472\mathchar 29048_{\mathchar 29044}\mathchar 24891\mathchar 29045_{\mathchar 29044}\delimiter 84054785\mathchar 24891\quad\mathchar 29044\mathchar 12850\mathbb{\mathchar 29006} (1) where \mathchar 29048_{\mathchar 29044}\mathchar 12850{{\mathbb{\mathchar 29010}}^{\mathchar 29038}} is the state and \mathchar 29045_{\mathchar 29044}\mathchar 12850{{\mathbb{\mathchar 29010}}^{\mathchar 29037}} is the control input. We focus on deterministic dynamics for clarity of exposition and discuss extensions to stochastic dynamics in Remark 3 . For a finite horizon \mathchar 29012 , we denote a state-control trajectory by \mathchar 28956\mathchar 12349\delimiter 67273472\mathchar 29048_{\mathchar 28720}\mathchar 24891\mathchar 29045_{\mathchar 28720}\mathchar 24891\mathchar 29048_{\mathchar 28721}\mathchar 24891\mathchar 29045_{\mathchar 28721}\mathchar 24891\ldots\mathchar 24891\mathchar 29048_{\mathchar 29012\mathchar 8704\mathchar 28721}\mathchar 24891\mathchar 29045_{\mathchar 29012\mathchar 8704\mathchar 28721}\mathchar 24891\mathchar 29048_{\mathchar 29012}\delimiter 84054785 , with total cost \mathchar 29002\delimiter 67273472\mathchar 28956\delimiter 84054785\mathchar 12349\mathchar 4944\displaylimits_{\mathchar 29044\mathchar 12349\mathchar 28720}^{\mathchar 29012\mathchar 8704\mathchar 28721}\mathchar 352_{\mathchar 29044}\delimiter 67273472\mathchar 29048_{\mathchar 29044}\mathchar 24891\mathchar 29045_{\mathchar 29044}\delimiter 84054785\mathchar 8235\mathchar 352_{\mathchar 29012}\delimiter 67273472\mathchar 29048_{\mathchar 29012}\delimiter 84054785\mathchar 314 (2) Here \mathchar 352_{\mathchar 29044} for \mathchar 29044\mathchar 12349\mathchar 28720\mathchar 24891\dots\mathchar 24891\mathchar 29012\mathchar 8704\mathchar 28721 are stage costs and \mathchar 352_{\mathchar 29012} is a terminal cost. Let \mathchar 28950 denote a distribution over initial states \mathchar 29048_{\mathchar 28720} , and let \mathchar 28946\mathchar 12850{{\mathbb{\mathchar 29010}}^{\mathchar 29028}} parameterize a (state-feedback) controller \mathchar 28953_{\mathchar 28946}\mathchar 12346{{\mathbb{\mathchar 29010}}^{\mathchar 29038}}\mathchar 12833{{\mathbb{\mathchar 29010}}^{\mathchar 29037}} . Under \mathchar 28953_{\mathchar 28946} , the control input is \mathchar 29045_{\mathchar 29044}\mathchar 12349\mathchar 28953_{\mathchar 28946}\delimiter 67273472\mathchar 29048_{\mathchar 29044}\delimiter 84054785 , which together with the dynamics induces a trajectory rollout \mathchar 28956\delimiter 67273472\mathchar 29048_{\mathchar 28720}\mathchar 24891\mathchar 28946\delimiter 84054785 . For simplicity, we focus on deterministic controllers. We then consider the unified objective \min_{\mathchar 28946\mathchar 12850{{\mathbb{\mathchar 29010}}^{\mathchar 29028}}}\mathbb{\mathchar 28997}_{\mathchar 29048_{\mathchar 28720}\mathchar 12824\mathchar 28950}\left\delimiter 67482370\mathchar 29002\delimiter 67273472\mathchar 28956\delimiter 67273472\mathchar 29048_{\mathchar 28720}\mathchar 24891\mathchar 28946\delimiter 84054785\delimiter 84054785\right\delimiter 84267779\mathchar 314 (3) Problem ( 3 ), henceforth referred to as trajectory and policy optimization (TPO), subsumes two fundamental settings in optimal control and reinforcement learning: • Trajectory Optimization (TO) [ 81 , 16 , 21 , 55 , 78 ] : fix the initial state by setting \mathchar 28950\mathchar 12349\mathchar 28942_{\mathchar 29048_{\mathchar 28720}} (a Dirac measure), and choose an open-loop parameterization \mathchar 28946\!\mathchar 12349\!\delimiter 67273472\mathchar 29045_{\mathchar 28720}\mathchar 24891\dots\mathchar 24891\mathchar 29045_{\mathchar 29012\mathchar 8704\mathchar 28721}\delimiter 84054785 (equivalently, \mathchar 28953_{\mathchar 28946}\delimiter 67273472\mathchar 29048_{\mathchar 29044}\delimiter 84054785\mathchar 12817\mathchar 29045_{\mathchar 29044} ). Then ( 3 ) reduces to optimizing a single control sequence for a given \mathchar 29048_{\mathchar 28720} . • Policy Optimization (PO) [ 92 , 86 , 88 , 53 , 35 ] : let \mathchar 28950 represent a distribution over initial conditions ( e.g., uniform over a region in the state space), and parameterize \mathchar 28953_{\mathchar 28946} as a state-feedback policy ( e.g., with a neural network). Then ( 3 ) becomes the problem of minimizing expected trajectory cost over the initial-state distribution. Motivated by the recent efforts in differentiable physics simulation—particularly contact solvers [ 60 , 47 , 89 , 29 , 77 , 74 , 46 ] —and in learning differentiable world models [ 79 , 63 , 111 , 3 , 7 , 2 , 107 ] , we assume the dynamics \mathchar 29030\delimiter 67273472\mathchar 29048_{\mathchar 29044}\mathchar 24891\mathchar 29045_{\mathchar 29044}\delimiter 84054785 is differentiable in both \mathchar 29048_{\mathchar 29044} and \mathchar 29045_{\mathchar 29044} . This enables gradient-based approaches that “exploit” differentiability: in TO, nonlinear programming techniques (single or multiple shooting) optimize an open-loop control sequence [ 81 , 9 , 101 ] ; in PO, recent “first-order” methods [ 20 , 108 , 7 , 8 , 104 ] leverage differentiability to compute policy gradients more efficiently and improve sample efficiency. A drawback, however, is that gradient-based optimization can get trapped in undesirable local minima, and gradients from differentiable simulators can be uninformative in the presence of nonsmooth dynamics [ 91 ] . In contrast, sampling-based methods such as model predictive path integral control (MPPI) [ 105 , 106 ] “explore” the control space and use the dynamics model only to evaluate trajectory costs; they can be less sensitive to poor local minima but often require many samples. This motivates our central question: can we design a unified and principled algorithm for TPO that combines the benefits of sampling and gradient-based optimization under differentiable dynamics? From Optimization to Inference. Towards this, instead of optimizing a single parameter \mathchar 28946 in ( 3 ), we consider optimizing a distribution \mathchar 29040 over \mathchar 28946\mathchar 12850{{\mathbb{\mathchar 29010}}^{\mathchar 29028}} . Let \mathchar 29040_{\mathchar 28720} be a prior distribution with full support on {{\mathbb{\mathchar 29010}}^{\mathchar 29028}} ( i.e., with density \mathchar 29040_{\mathchar 28720}\delimiter 67273472\mathchar 28946\delimiter 84054785\mathchar 12606\mathchar 28720 for all \mathchar 28946\mathchar 12850{{\mathbb{\mathchar 29010}}^{\mathchar 29028}} ; e.g., a Gaussian). We study the KL-regularized problem \min_{\mathchar 29040}\mathbb{\mathchar 28997}_{\mathchar 29048_{\mathchar 28720}\mathchar 12824\mathchar 28950\mathchar 24891\mathchar 28946\mathchar 12824\mathchar 29040}\left\delimiter 67482370\mathchar 29002\delimiter 67273472\mathchar 28956\delimiter 67273472\mathchar 29048_{\mathchar 28720}\mathchar 24891\mathchar 28946\delimiter 84054785\delimiter 84054785\right\delimiter 84267779\mathchar 8235\mathchar 28949\mathchar 28996_{\mathrm{\mathchar 29003\mathchar 29004}}\delimiter 67273472\mathchar 29040\delimiter 69645069\mathchar 29040_{\mathchar 28720}\delimiter 84054785\mathchar 24891 (4) where \mathchar 28996_{\mathrm{\mathchar 29003\mathchar 29004}}\delimiter 67273472\mathchar 29040\delimiter 69645069\mathchar 29040_{\mathchar 28720}\delimiter 84054785\mathchar 12346\mathchar 12349\mathchar 4946\nolimits_{{{\mathbb{\mathchar 29010}}^{\mathchar 29028}}}\mathchar 29040\delimiter 67273472\mathchar 28946\delimiter 84054785\log{{\mathchar 29040\delimiter 67273472\mathchar 28946\delimiter 84054785\over\mathchar 29040_{\mathchar 28720}\delimiter 67273472\mathchar 28946\delimiter 84054785}}\mathchar 29028\mathchar 28946 is the Kullback–Leibler (KL) divergence and \mathchar 28949\mathchar 12606\mathchar 28720 controls the strength of regularization (equivalently, a “temperature”). At \mathchar 28949\mathchar 12349\mathchar 28720 , problem ( 4 ) is equivalent to the TPO problem ( 3 ): they have the same optimal value and any distribution supported on the optimal solution set of ( 3 ) is optimal for ( 4 ). As \mathchar 28949\mathchar 12833\mathchar 28720^{\mathchar 8235} , the optimal \mathchar 29040^{\mathchar 8511} concentrates its mass on minimizers of ( 3 ). A key advantage of ( 4 ) is that it admits a closed-form solution. Theorem 1 (Optimal Distribution) . Assume \mathchar 352_{\mathchar 29044}\delimiter 67273472\mathchar 8705\mathchar 24891\mathchar 8705\delimiter 84054785\mathchar 12821\mathchar 28720 for \mathchar 29044\mathchar 12349\mathchar 28720\mathchar 24891\dots\mathchar 24891\mathchar 29012\mathchar 8704\mathchar 28721 and \mathchar 352_{\mathchar 29012}\delimiter 67273472\mathchar 8705\delimiter 84054785\mathchar 12821\mathchar 28720 . Define the energy function \mathchar 28997\delimiter 67273472\mathchar 28946\delimiter 84054785\mathchar 12349\mathbb{\mathchar 28997}_{\mathchar 29048_{\mathchar 28720}\mathchar 12824\mathchar 28950}\left\delimiter 67482370\mathchar 29002\delimiter 67273472\mathchar 28956\delimiter 67273472\mathchar 29048_{\mathchar 28720}\mathchar 24891\mathchar 28946\delimiter 84054785\delimiter 84054785\right\delimiter 84267779\mathchar 24891 (5) then the optimal distribution \mathchar 29040^{\mathchar 8511} of ( 4 ) is given by \mathchar 29040^{\mathchar 8511}\delimiter 67273472\mathchar 28946\delimiter 84054785\mathchar 12349{{\mathchar 28721\over\mathchar 29018}}\,\mathchar 29040_{\mathchar 28720}\delimiter 67273472\mathchar 28946\delimiter 84054785\exp\left\delimiter 67273472\mathchar 8704{{\mathchar 28721\over\mathchar 28949}}\mathchar 28997\delimiter 67273472\mathchar 28946\delimiter 84054785\right\delimiter 84054785\mathchar 24891 (6) where \mathchar 29018\mathchar 12349\mathchar 4946\nolimits_{{{\mathbb{\mathchar 29010}}^{\mathchar 29028}}}\mathchar 29040_{\mathchar 28720}\delimiter 67273472\mathchar 28946\delimiter 84054785\exp\left\delimiter 67273472\mathchar 8704{{\mathchar 28721\over\mathchar 28949}}\mathchar

Read the complete text