State-Covering Trajectory Stitching for Diffusion Planners
Kyowoon Lee, Jaesik Choi · 2025
Read State-Covering Trajectory Stitching for Diffusion Planners by Kyowoon Lee, Jaesik Choi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-00895.txt.
Introduction
Kyowoon Lee and Jaesik Choi stitch short runs into long ones to teach plan-making models in a 2025 paper. Such models dream a path to a goal, but their dreams are capped by the runs they were fed, which tend to be short and few.
Their stitch tool first learns a map that keeps the true time gaps between states. It then sews runs end to end, steered by a thirst for new ground. The sewn set is both wide and long, with no need for prize marks in the data. Planners fed the sewn set did far better on tests that ask for long hops and joins. Plain goal tools gained as well, across more than one world.
Catalog record
| CREATOR | Kyowoon Lee, Jaesik Choi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-00895
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
State-Covering Trajectory Stitching for Diffusion Planners Kyowoon Lee, Jaesik Choi Abstract Diffusion-based generative models are emerging as powerful tools for long-horizon planning in reinforcement learning (RL), particularly with offline datasets. However, their performance is fundamentally limited by the quality and diversity of training data. This often restricts their generalization to tasks outside their training distribution or longer planning horizons. To overcome this challenge, we propose State-Covering Trajectory Stitching (SCoTS), a novel reward-free trajectory augmentation method that incrementally stitches together short trajectory segments, systematically generating diverse and extended trajectories. SCoTS first learns a temporal distance-preserving latent representation that captures the underlying temporal structure of the environment, then iteratively stitches trajectory segments guided by directional exploration and novelty to effectively cover and expand this latent space. We demonstrate that SCoTS significantly improves the performance and generalization capabilities of diffusion planners on offline goal-conditioned benchmarks requiring stitching and long-horizon reasoning. Furthermore, augmented trajectories generated by SCoTS significantly improve the performance of widely used offline goal-conditioned RL algorithms across diverse environments. 1 Introduction In many real-world applications, agents must plan over hundreds of steps, often receiving sparse or delayed feedback until they reach a distant goal. Perfect knowledge of the environment allows powerful planners like MPC Tassa et al. 2012 and MCTS Silver et al. 2016 ; Silver et al. 2017 ; Lee et al. 2018 to excel. However, most real-world tasks instead require learning environment dynamics from data. Model-based reinforcement learning (MBRL) Sutton 2018 constructs such world models, offering sample-efficient learning and improved generalization Ha & Schmidhuber 2018 ; Hafner et al. 2019 ; Kaiser et al. 2020 . However, autoregressive predictions from learned models accumulate small errors into a cascade of inaccuracies. This compounding error can cause planners to exploit model inaccuracies and generate trajectories that are suboptimal or even physically infeasible, especially in long-horizon tasks Talvitie 2014 ; Asadi et al. 2018 ; Janner et al. 2019 ; Voelcker et al. 2022 ; Chen et al. 2024a . To address these limitations, diffusion planners Janner et al. 2022 ; Ajay et al. 2023 ; Liang et al. 2023 ; Chen et al. 2024c have recently emerged as a promising alternative for trajectory generation in sequential decision-making. Instead of rolling out one step at a time, diffusion planners treat each trajectory as a single high-dimensional sample, learning a denoising process that transforms noise drawn from a simple prior into trajectories that match the target distribution Ho et al. 2020 ; Song et al. 2021 . By operating on entire trajectories simultaneously, these methods inherently prevent the compounding of prediction errors that undermine autoregressive dynamics models. Moreover, the generative nature of diffusion models allows for flexible conditioning and guidance mechanisms, enabling the synthesis of plans with properties like reaching specific goals or maximizing expected returns Dhariwal & Nichol 2021 . (a) Offline Data (b) HD (c) Ours Figure 1: Improved generalization with SCoTS. (a) Examples from the training dataset, illustrating limited coverage. (b) Plans generated by Hierarchical Diffuser (HD) Chen et al. 2024c , which fail to generalize well to these out-of-distribution tasks due to insufficient coverage of the training data. (c) Plans generated by HD trained on SCoTS-augmented data, demonstrating significantly improved trajectory stitching capability and generalization to unseen tasks. Each color corresponds to one of 10 plans generated by the planner. Despite these advantages, the effectiveness of diffusion planners remains fundamentally limited by the quality, diversity, and coverage of the offline training data. First, their effective planning horizon is inherently coupled to the maximum trajectory length observed during training, making it challenging to generate coherent plans that significantly exceed this length. Second, their generalization capability is often confined to the specific types of trajectories and transitions represented in the training data. For instance, if the dataset predominantly features certain movement patterns, the planner may struggle to synthesize solutions for novel tasks requiring different compositions of behaviors (as illustrated in Figure 1 ). While exhaustively collecting data for all conceivable scenarios could mitigate this, such an approach is prohibitively expensive. Trajectory stitching Ziebart et al. 2008 offers a promising alternative by composing novel, longer sequences from existing short segments. However, existing methods rely heavily on extrinsic rewards for segment selection, and maintaining the dynamic consistency and feasibility of stitched trajectories remains challenging. In this paper, we propose S tate- Co vering T rajectory S titching ( SCoTS ), a reward-free trajectory augmentation framework that systematically extends trajectories to cover diverse, unexplored regions of the state space. Specifically, SCoTS employs a three-stage approach: First, we learn a temporal distance-preserving latent representation by training a model to encode states based on learned optimal temporal distances, facilitating efficient identification of viable trajectory segments. Second, we introduce a novel iterative stitching strategy that balances directed exploration with state-space coverage. In this process, trajectory segments are selected based on their progress along a learned direction in the latent space and their novelty relative to previously explored regions within the rollout. Finally, we refine the resulting stitched trajectories using a diffusion-based refinement procedure. Consequently, the resulting trajectories exhibit broader state-space coverage while preserving dynamic feasibility. To summarize, our contribution in this paper is the introduction of SCoTS, a reward-free trajectory augmentation approach designed to generate diverse, long-horizon trajectories that enhance diffusion planners. Extensive experiments across diverse and challenging benchmark tasks show that SCoTS significantly enhances the stitching capabilities and long-horizon generalization of diffusion planners. Furthermore, augmented trajectories generated by SCoTS notably boost the performance of widely used offline goal-conditioned reinforcement learning (GCRL) algorithms in across multiple trajectory stitching benchmarks. 2 Planning with Diffusion Models † † footnotetext: Codes are available at https://github.com/leekwoon/scots . Figure 2: Overview of the SCoTS stitching process. (a) Temporal Distance-Preserving Search: Given the currently composed trajectory (red), we identify candidate segments (gray) by searching in a latent space learned to preserve temporal distances. Candidates are selected based on proximity to the endpoint of the current trajectory in latent space. (b) Exploratory Segment Selection: Among the retrieved candidate segments, we select the segment (blue) that best balances directional progress toward a randomly sampled latent direction and novelty relative to previously visited states in latent space. (c) Diffusion-based Stitching Refinement: To ensure smooth transitions, a diffusion model refines the stitching point between segments, generating dynamically consistent trajectories. Diffusion-based planners Janner et al. 2022 ; Liang et al. 2023 ; Chen et al. 2024c provide a promising framework for long-horizon decision-making by modeling entire trajectories as joint distributions. A trajectory \boldsymbol{\tau} is typically represented as a sequence of states \boldsymbol{s}_{t} and actions \boldsymbol{a}_{t} over a planning horizon T : \displaystyle\boldsymbol{\tau}=\begin{bmatrix}\boldsymbol{s}_{1}&\boldsymbol{s}_{2}&\lx@intercol\hfil\hbox{\multirowsetup$\ldots$}\hfil\lx@intercol&\boldsymbol{s}_{T}\\ \boldsymbol{a}_{1}&\boldsymbol{a}_{2}&&\boldsymbol{a}_{T}\end{bmatrix}, (1) where \boldsymbol{s}_{t} and \boldsymbol{a}_{t} denote the state and action at time step t , respectively. Diffusion planners utilize diffusion probabilistic models Sohl-Dickstein et al. 2015 ; Ho et al. 2020 to learn a trajectory distribution p_{\theta}(\boldsymbol{\tau}^{0}) over noise-free trajectories \boldsymbol{\tau}^{0} . This involves a predefined forward noising process and a learned reverse denoising process. The forward process incrementally adds Gaussian noise to the trajectories through M discrete diffusion timesteps with a variance schedule \{{\beta_{i}}\}_{i=1}^{M} : \displaystyle q(\boldsymbol{\tau}^{i}|\boldsymbol{\tau}^{i-1})\coloneqq{\cal N}(\boldsymbol{\tau}^{i};\sqrt{1-\beta_{i}}\boldsymbol{\tau}^{i-1},\beta_{i}\mathbf{I}). (2) A key property is the direct sampling of intermediate trajectories: \displaystyle q({\boldsymbol{\tau}}^{i}\mid{\boldsymbol{\tau}}^{0})={\cal N}({\boldsymbol{\tau}}^{i};\sqrt{\alpha_{i}}{\boldsymbol{\tau}}^{0},(1-\alpha_{i})\mathbf{I}), (3) where \alpha_{i}\coloneqq\prod_{s=1}^{i}(1-\beta_{s}) . The schedule ensures that \boldsymbol{\tau}^{M} approximates a standard Gaussian