Temporal Straightening for Latent Planning

Ying Wang, Oumayma Bounou, Gaoyue Zhou, Randall Balestriero, Tim G. J. Rudner, Yann LeCun, Mengye Ren · 2026

Read Temporal Straightening for Latent Planning by Ying Wang, Oumayma Bounou, Gaoyue Zhou, Randall Balestriero, Tim G. J. Rudner, Yann LeCun, Mengye Ren (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-12231.txt.

Introduction

Wang and co-authors straighten latent paths over time to make world model planning work better. Pretrained visual encoders give strong features, but they are not shaped for planning tasks. A curvature regularizer nudges latent trajectories toward local straight lines during training. Encoder and predictor of a JEPA world model learn together under that rule. Straight paths make Euclidean distance a better stand-in for true geodesic distance. Gradient-based planning then runs more stably and hits goal tasks at far higher rates.

Catalog record

CREATORYing Wang, Oumayma Bounou, Gaoyue Zhou, Randall Balestriero, Tim G. J. Rudner, Yann LeCun, Mengye Ren
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59348
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.12231

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Temporal Straightening for Latent Planning Ying Wang, Oumayma Bounou, Gaoyue Zhou, Randall Balestriero, Tim G. J. Rudner, Yann LeCun, Mengye Ren Abstract Learning good representations is essential for latent planning with world models. While pretrained visual encoders produce strong semantic visual features, they are not tailored to planning and contain information irrelevant—or even detrimental—to planning. Inspired by the perceptual straightening hypothesis in human visual processing, we introduce temporal straightening to improve representation learning for latent planning. Using a curvature regularizer that encourages locally straightened latent trajectories, we jointly learn an encoder and a predictor. We show that reducing curvature this way makes the Euclidean distance in latent space a better proxy for the geodesic distance and improves the conditioning of the planning objective. We demonstrate empirically that temporal straightening makes gradient-based planning more stable and yields significantly higher success rates across a suite of goal-reaching tasks. Temporal Straightening for Latent Planning Wang et al. 0 0 footnotetext: * Equal advising. 1 Introduction Figure 1 : Latent trajectories encoded by a pretrained visual encoder are usually highly curved, increasing the difficulty of prediction and planning. We learn a representation space where feasible trajectories are straighter. Latent world models offer a compelling solution for planning because they introduce abstraction that improves efficiency and generalization ( 42 ; 53 ; 22 ; 23 ; 25 ; 26 ; 28 ; 27 ) . They compress high-dimensional observations into compact latent representations, learn predictive dynamics in that latent space, and enable imaginary rollouts for action optimization. Compared to operating directly in pixel or state space, the latent abstraction reduces dimensionality and ignores noise, making dynamics learning easier and more efficient. At test time, planning is typically posed as optimizing an action sequence by rolling the model forward and minimizing a cost function between the goal and the predicted terminal state in the latent space. In practice, however, optimization in the learned latent space remains challenging. The induced planning objective is typically highly non-convex, potentially causing gradient-based optimizers to struggle. As a result, many successful practices ( 24 ; 27 ; 60 ; 51 ; 54 ) rely on search-based methods such as CEM ( 48 ) or MPPI ( 57 ) , which achieve competitive performance but introduce a substantial compute burden and latency. Moreover, commonly used goal cost metrics based on Euclidean distance can be misleading if the embedding space is not properly regularized. In particular, when latent trajectories are highly curved, straight-line distances in embedding space misrepresent the geodesic distance along feasible transitions. These challenges call for better representations that are tailored for latent planning. What is a “good” representation for latent planning? Large-scale visual pretraining provides powerful semantic-aware features, but it is not tailored to the dynamics of the environments and often retains plenty of planning-irrelevant low-level details. We argue that planning could benefit from representations that are (i) sufficient for predicting dynamics but without task-irrelevant information and (ii) properly regularized so that embedding distances reflect the geodesic distance and gradient-based optimization is reliable. With such representations, we can exploit the differentiability of latent world models and enable efficient gradient-based planning, bypassing the need for computationally expensive search-based methods. Figure 2 : Latent trajectories before vs. after straightening. The upper PushT example is a rotation and the bottom UMaze example shows the agent traveling from the left top to the right top, with the star denoting the target. Straightening yields less curved and smoother trajectories, and makes Euclidean distance a more faithful proxy for geodesic progress towards the goal. More examples are shown in Section D.2 . Inspired by the perceptual straightening hypothesis in human vision ( 32 ) , which posits that visual systems transform complex videos into straighter internal representations, we introduce a simple approach to straighten latent trajectories for planning. Concretely, we jointly learn an encoder and a predictor of a world model, while imposing regularization on the curvature of latent trajectories during training. The resulting encoded trajectories are significantly straighter, with Euclidean distances better aligned with geodesic distances ( Figure 2 ). We prove that reducing curvature improves convergence of gradient-based planners, and observe superior empirical gains across a suite of goal-reaching tasks: open-loop planning success improves by 20–60% and MPC by 20-30% with a simple gradient-based planner. 2 Related Work While early visual world models directly predict in pixel spaces and use generated images for control ( 44 ; 18 ; 16 ; 15 ) , an increasing number of recent works first encode high-dimensional sensory inputs into compact latent representations and plan in the resulting latent space. Learning representations is central to these latent world models. To obtain meaningful representations for world modeling, prior methods add reconstruction-based objectives when training the encoder along with the predictor ( 56 ; 59 ; 39 ; 22 ; 24 ; 23 ; 40 ; 47 ) . However, these reconstruction objectives overemphasize low-level visual details that are unnecessary for planning and may fail to capture task-relevant information. More recent approaches decouple perception from dynamics by leveraging strong pretrained visual encoders ( 41 ; 60 ; 6 ; 20 ; 4 ; 2 ) . Closest to our setup, DINO-WM ( 60 ) trains task-agnostic predictors and plans directly in frozen DINOv2 ( 45 ) feature space. While DINOv2 features provide high-quality visual representations, they are not optimized for planning and may lead to planning objectives that are challenging to optimize. In this work, we improve the representation space for planning by introducing a straightening regularization during world model training. Joint-Embedding Predictive Architecture (JEPA) emerges as a promising paradigm for world models by learning representations through prediction ( 38 ; 7 ; 2 ) . It aims to capture predictable structure without retaining unpredictable low-level details, making it more effective and efficient than reconstruction-based objectives ( 1 ) . This paradigm has been shown to be effective for predictive modeling and planning, with training from scratch on offline simulator data ( 51 ) and large-scale real-world video pretraining followed by action-conditioned post-training for robotic planning ( 2 ) . Our work also belongs to the JEPA family and focuses on learning better representations. Temporal Contrastive Learning ( 49 ; 13 ; 17 ; 58 ) is also a popular paradigm for learning representations that can reflect the temporal relationships. It encourages temporally close frames to have similar embeddings while distant frames have more dissimilar embeddings through InfoNCE loss ( 46 ) . However, how to choose positive and negative samples requires careful tuning and this objective might push away geodesically close states if suboptimal trajectories are used. Instead, our regularization-based method doesn’t require negatives and applies local straightening without requiring expert trajectories. Motivated by the perceptual straightening hypothesis in human vision ( 32 ) , some prior works have examined implicit straightening in pretrained visual encoders ( 29 ; 35 ) or used it as an objective to obtain robust video models ( 43 ; 3 ) . Implicit straightening is also observed in autoregressive language models optimized for next-word prediction ( 34 ; 33 ) . To the best of our knowledge, however, none of these prior works have studied the impact of straightening on planning. 3 Temporal

Read the complete text