Dreaming Smoothly and Sample Efficiently with Gradient Penalized Latent Dynamics

Romil V. Sonigra, P. R. Kumar · 2026

Read Dreaming Smoothly and Sample Efficiently with Gradient Penalized Latent Dynamics by Romil V. Sonigra, P. R. Kumar (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-23089.txt.

Introduction

Sonigra and Kumar propose GPLD, a gradient-based latent dynamics penalty for the DreamerV3 world model. Existing latent world models do not enforce local smoothness in learned transition dynamics. GPLD applies a row-wise Jacobian cost to the posterior latent distribution. The penalty mirrors finite-difference smoothing of state laws in discrete MDPs, estimated cheaply with Hutchinson-style stochastic probes. Across DeepMind Control proprioceptive tasks it improves sample efficiency, most strongly on complex locomotion, and reaches high-return quadruped behavior earlier.

Catalog record

CREATORRomil V. Sonigra, P. R. Kumar
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42402
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.23089

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Dreaming Smoothly and Sample Efficiently with Gradient Penalized Latent Dynamics Romil V. Sonigra, P. R. Kumar Abstract Model-based reinforcement learning improves sample efficiency by learning a world model. However, existing latent world models such as DreamerV3 do not explicitly enforce local smoothness in their learned transition dynamics, leaving a useful inductive bias for transition dynamics learning unexploited. We propose GPLD, a gradient-penalized latent dynamics regularizer for DreamerV3 that applies a row-wise Jacobian penalty to the posterior latent distribution to encourage locally smooth transition learning. We show that this penalty can be interpreted as the continuous-latent analog of finite-difference smoothing of transition laws in discrete embedded-state MDPs, and estimate it efficiently using Hutchinson-style stochastic probes. Empirically, across DeepMind Control proprioceptive tasks, GPLD improves aggregate sample efficiency, with particularly strong gains on higher-complexity locomotion environments. On more challenging quadruped tasks, GPLD reaches high-return behavior earlier and exhibits more consistent late-stage learning over longer horizons. Explicit local smoothness regularization is a simple and effective way to improve latent world models for smooth continuous control environments. Code for GPLD is available at github.com/romils9/gpld-mbrl . 1 Introduction Model-based reinforcement learning (MBRL) improves sample efficiency by learning a predictive world model that can be used for planning and policy optimization ( 11 ) . DreamerV3 has demonstrated that latent world models can scale to diverse continuous-control tasks by learning compact recurrent state-space representations and training policies through imagined rollouts ( 12 ) . Despite this success, standard latent world models do not explicitly exploit a basic structural prior of many continuous-control systems: nearby states induce similar short-horizon transition behavior. Encoding this prior directly in the learned latent dynamics can help the world model share information across nearby states, reducing the amount of interaction data needed to learn useful transition structure. We study how to exploit this local smoothness prior through gradient-penalized latent dynamics (GPLD), a differentiable regularizer for latent world-model dynamics. This regularizer is motivated through a discrete-to-continuous smoothness argument. In a finite embedded-state MDP, local smoothness can be imposed by penalizing differences between transition laws at neighboring states. When the state representation is continuous and the transition model is differentiable, these normalized finite differences become directional derivatives of the learned dynamics. Averaging over local directions yields a Frobenius Jacobian penalty. This provides a principled route from neighborhood smoothing in tabular MDPs to the gradient penalty used by GPLD. We implement GPLD in DreamerV3 and evaluate it on DeepMind Control tasks. Empirically, GPLD improves aggregate sample efficiency on proprioceptive tasks, with the largest gains attained on higher-complexity locomotion environments. On difficult quadruped tasks, the benefit becomes clearer over longer horizons, where GPLD reaches high-return behavior earlier and exhibits more consistent late-stage learning. Pixel-observation results show milder aggregate gains, suggesting that the effect of latent smoothness regularization is weaker when dynamics learning is coupled to high-dimensional visual encoding. Our contributions are: 1. A discrete-to-continuous justification showing how finite-difference smoothing of tabular transition laws leads to Frobenius Jacobian regularization in differentiable latent models. 2. GPLD, a posterior Jacobian regularizer for DreamerV3 that explicitly encourages local smoothness in latent probability maps, yielding a normalized aggregate gain of 17.7\% across the DeepMind Control proprioceptive benchmark. 3. Evaluation of GPLD on DeepMind Control, showing a 34.6\% normalized aggregate gain on higher-complexity proprioceptive locomotion tasks, clearer long-horizon benefits on difficult quadruped tasks, and ablations over the main design choices. Code for GPLD is available at github.com/romils9/gpld-mbrl . 2 Related Work Smoothness in reinforcement learning. Smoothness has long been recognized as important for generalization, stable prediction, and control in reinforcement learning ( 3 ; 13 ; 1 ; 5 ) . Unconstrained neural networks can produce high-frequency oscillations that degrade stability and generalization ( 3 ; 15 ; 5 ) , and such errors can compound in multi-step value or dynamics prediction ( 17 ; 19 ) . In model-based RL, 1 show that Lipschitz constraints on learned transition models yield tighter multi-step estimation error bounds, directly linking transition smoothness to world-model accuracy. These works motivate smoothness as a useful inductive bias for learned dynamics, but do not provide a local Jacobian regularizer for stochastic latent world models. Gradient penalties and control smoothness. Gradient penalties provide a differentiable way to encourage local smoothness and have been widely used in representation and generative modeling ( 10 ; 16 ; 9 ) . In reinforcement learning, related methods often regularize value functions, policies, or action trajectories. For example, 20 apply gradient penalties to Q-functions in model-free RL, while 4 introduce Lipschitz-constrained policies as a differentiable alternative to task-specific reward-smoothing heuristics. Other control approaches encourage smooth behavior through penalties on mechanical energy or low-pass filtering ( 7 ; 14 ) . In contrast, GPLD targets the learned world model itself: it applies a row-wise Jacobian penalty to the posterior latent probability map in DreamerV3, improving the latent dynamics used for inference and imagination. Global versus local smoothness constraints. Global smoothness methods such as spectral normalization constrain the Lipschitz constant of neural networks by bounding layer-wise spectral norms ( 8 ; 2 ) . However, global constraints can be overly restrictive because the global Lipschitz constant may arise from rare, unseen, or irrelevant regions of the input space ( 6 ) . GPLD instead imposes a local, data-dependent smoothness penalty on sampled latent inputs during world-model training. This distinguishes GPLD from global normalization methods and from policy-smoothing approaches: GPLD regularizes the posterior transition representation used by a model-based agent, while leaving the rest of the world model unconstrained except through the training objective. 3 A discrete-to-continuous justification for GPLD We justify GPLD by connecting a discrete local-smoothness prior on transition laws to the continuous Jacobian penalty used in our latent world-model setting. We begin with a finite embedded-state MDP, where smoothness can be imposed by penalizing finite differences of neighboring transition distributions. We then consider a differentiable transition model over continuous state representations and show that these finite differences become directional Jacobian norms, whose isotropic average yields the Frobenius Jacobian norm. This produces the row-wise posterior regularizer used in GPLD. 3.1 Discrete local smoothness prior We begin with a finite state and finite action MDP. Let \mathcal{S} denote the state space, \mathcal{A} the action space, and let e:\mathcal{S}\to\mathbb{R}^{d} be an embedding of states into a metric space. For each action a\in\mathcal{A} , let P(\cdot\mid s,a)\in\Delta^{|\mathcal{S}|-1} denote the transition probability vector, and define the \varepsilon -neighborhood of s by \mathcal{N}_{\varepsilon}(s):=\{k\in\mathcal{S}:\|e(k)-e(s)\|_{2}\leq\varepsilon\}. Given a dataset of transitions \mathcal{D}=\{(s,a,s^{\prime})\}_{t=1}^{T} , let N(s,a,s^{\prime}) denote the number of observed transitions from (s,a) to s^{\prime} . The standard maximum-likelihood estimator minimizes the negative log-likelihood \mathcal{L}_{\mathrm{MLE}}(P;\mathcal{D})=-\sum_{s,a,s^{\prime}}N(s,a,s^{\prime})\log P(s^{\prime}\mid s,a). A natural structural prior is that nearby states should induce similar transition laws. We encode this by coupling neighboring transition vectors through a finite-difference regularizer: \mathcal{L}_{\mathrm{FD}}(P)=\sum_{a}\sum_{s}\sum_{k\in\mathcal{N}_{\varepsilon}(s)}\sum_{s^{\prime}}\left(\frac{P(s^{\prime}\mid k,a)-P(s^{\prime}\mid s,a)}{\|e(k)-e(s)\|_{2}}\right)^{2}. (1) This yields the regularized objective \min_{P}\;\mathcal{L}_{\mathrm{MLE}}(P;\mathcal{D})+\lambda\mathcal{L}_{\mathrm{FD}}(P). This is the discrete precursor of GPLD: it penalizes normalized local differences of transition laws across neighboring states and therefore explicitly favors locally smooth transition estimates. 3.2 Continuous-state limit The discrete finite-difference regularizer in ( 1 ) leads to a Jacobian penalty when the transition law is represented by a differentiable function over continuous state embeddings. Consider a neighboring state k\in\mathcal{N}_{\varepsilon}(s) . In the embedding space, its displacement from s can be

Read the complete text