Motion Planning in Compressed Representation Spaces

Lukas Lao Beyer, Sertac Karaman · 2026

Read Motion Planning in Compressed Representation Spaces by Lukas Lao Beyer, Sertac Karaman (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-30940.txt.

Introduction

Lukas Lao Beyer and Sertac Karaman plan robot motion inside a compressed code instead of the original trajectory space. Learned planners capture rich behaviour from data but are hard to steer; classical search and optimisation are flexible yet work in bulky representations. The pair first train an autoencoder with a high compression ratio that turns trajectories into hierarchically ordered, discrete tokens, coarse details first and refinements later. Planning then becomes a search over token sequences, scored by whatever objective the user sets at test time.

Because the decoder is generative, the searched plans stay realistic. On the nuPlan benchmark and the Waymo Open Motion Dataset, latent search performed strongly at closed-loop motion planning. It also synthesised guided multi-agent scenarios, all without any task-specific training.

Catalog record

CREATORLukas Lao Beyer, Sertac Karaman
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE65 KB
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2606.30940

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Motion Planning in Compressed Representation Spaces Lukas Lao Beyer, Sertac Karaman Abstract Deep learning methods have vastly expanded the capabilities of motion planning in robotics applications, as learning priors from large-scale data has been shown to be essential in capturing the highly complex behavior required for solving tasks such as manipulation or navigation for autonomous vehicles. At the same time, model-based planning algorithms based on search or optimization remain an essential tool due to their flexibility, efficiency, and the ability to incorporate domain knowledge via expert-designed algorithms and objective functions. We propose a new generative framework to unify these two paradigms. First, we learn an autoencoder with a high compression ratio and a latent space of hierarchically ordered, discrete-valued tokens. Leveraging both the dimensionality reduction and the hierarchical coarse-to-fine structure learned by this autoencoder, we then perform motion planning by directly searching in the latent space of tokens. This search can optimize arbitrary objective functions specified at test time, providing a large degree of flexibility while maintaining efficiency and producing realistic solutions by relying on the generative capabilities of the highly compressed autoencoder. We evaluate our method on nuPlan and the Waymo Open Motion Dataset, showing how latent space search can be used for a variety of guided behavior generation tasks, achieving strong performance for closed-loop motion planning and multi-agent guided scenario synthesis without requiring any task-specific training. (a) Environment-conditioned trajectory decoder. search \xrightarrow{\hskip 61.45854pt} 1 token 2 tokens 3 tokens Goal A Goal B (b) Latent token search with test-time objective (no retraining). Figure 1 : Token search for guided trajectory synthesis. We learn a highly compressed, environment-conditioned autoencoder that maps trajectories to an ordered discrete code z . At test time we perform greedy token search: at step i , we fix z_{1:i-1} , enumerate z_{i} , decode each candidate trajectory, score according to the user-specified objective chosen at test time , and keep the best prefix (here we illustrate a goal-reaching objective). Because tokens are causally ordered and learned with nested dropout, each additional token refines a feasible prefix so greedy search follows the representation’s intended coarse-to-fine semantics. Token semantics are not hand-designed and emerge through compression. 1 Introduction In real-world settings, robotic motion planning must produce trajectories that satisfy task- and user-specific objectives while remaining realistic and safe in complex environments. Classical model-based planners offer flexibility through explicit cost functions and constraints, and can be highly robust and performant when deployed in controlled settings ( Foehn et al. 2021 ; Moore et al. 2014 ; Goh & Gerdes 2016 ) . However, purely model-based approaches struggle in unstructured and open-ended domains, where realistic behavior depends on rich priors learned from data. This has driven a shift toward learning-based approaches that leverage large-scale data to acquire such priors. However, many learning-based planners bake task structure into the model via conditioning ( Cheng et al. 2024 ) , training-time reward shaping ( Zhang et al. 2025 ) , or specialized inference procedures ( Dauner et al. 2023 ) , which can make it nontrivial to accommodate new objectives or constraints. In this paper, we propose a generative framework aimed at unifying these paradigms by separating trajectory realism from test-time control. A key ingredient enabling this separation is compression. Consider mapping high-dimensional trajectories to short sequences of low-dimensional or even discrete tokens: this simultaneously (i) reduces the dimensionality of the decision space the planner must search over and (ii) concentrates the burden of producing realistic motion in a decoder, i.e., a learned reconstruction model that expands tokens back into full trajectories. Figure 1 previews our approach, which implements this principle by learning an environment-conditioned autoencoder that maps trajectories to a highly compressed, ordered latent representation; motion planning then reduces to tree search over latent tokens guided by a user-specified test-time objective. This yields a simple interface for incorporating domain knowledge or preferences while relying on the decoder to produce realistic, context-appropriate behavior. From a motion-planning viewpoint, the environment-conditioned, ordered tokenization induces a context-dependent library of maneuvers. Unlike classical maneuver libraries with a small fixed set of hand-engineered primitives ( Frazzoli et al. 2005 ; Pivtoraiko et al. 2009 ) , the library here is learned from data and automatically adapts to the scene through environment conditioning. Furthermore, the hierarchical ordering of this library enables efficient traversal of a combinatorially large set of maneuvers via coarse-to-fine refinement. From the point of view of guided generation, our method chooses a different route than the iterative gradient-based guidance commonly used with diffusion and flow models, offering unique advantages. For example, partial token prefixes decode to trajectories in the original state space, so the objective can be applied directly without requiring additional estimators. Furthermore, latent token search naturally supports objectives that are not smooth or differentiable. We evaluate our approach on the Waymo Open Motion Dataset ( Ettinger et al. 2021 ) and nuPlan ( Caesar et al. 2021 ) , using autonomous driving as a demanding testbed for guided generation with diverse objectives. Overall, our results indicate that a highly compressed, ordered latent representation can turn guided generation into a direct search problem. Throughout our experiments, we find that greedy search in the learned token space provides a simple yet effective mechanism for test-time control, supporting a variety of objectives, yielding strong performance on closed-loop planning, and achieving high realism metrics on guided multi-agent scenario synthesis, all without task-specific training. 2 Related Work Test-time guidance for generative models. A large body of work studies how to steer generative models at test time. In diffusion models ( Ho et al. 2020 ) , guidance is often implemented by modifying the sampling dynamics using gradients of a conditioning signal, including classifier guidance ( Dhariwal & Nichol 2021 ) , classifier-free guidance ( Ho & Salimans 2021 ) , and guidance using more general functions ( Song et al. 2023 ; Bansal et al. 2023 ) . These techniques have also been adopted for trajectory generation in robotics, e.g., Diffusion Policy ( Chi et al. 2023 ) , and for controllable motion prediction in multi-agent settings, e.g., MotionDiffuser ( Jiang et al. 2023a ) . Our work differs in the mechanism used to realize test-time guidance: instead of gradient-based steering during sampling, we perform decode-and-score search over ordered latent tokens, which naturally supports objectives that are non-differentiable and can be evaluated directly on decoded trajectories even in intermediate search iterations. Tokenization and ordered representations. Tokenization via autoencoders is a standard technique for reducing the dimensionality of generative modeling problems. In image modeling, generators commonly operate in learned latent spaces of continuous VAEs ( Kingma & Welling 2014 ) or discrete tokenizers such as VQ-VAE/VQGAN ( van den Oord et al. 2017 ; Esser et al. 2021 ) to improve efficiency and scalability ( Rombach et al. 2022 ; Chang et al. 2022 ; Li et al. 2024 ) . Recently, nested dropout ( Rippel et al. 2014 ) has been used to learn ordered and variable-length image tokenizations, inducing a coarse-to-fine hierarchy in which shorter codes capture global structure and longer codes add detail ( Wen et al. 2025 ; Miwa et al. 2025 ; Bachmann et al. 2025 ) . We build on this

Read the complete text