Planning-Query-Guided Model Generation for Model-Based Deformable Object Manipulation
Alex LaGrassa, Zixuan Huang, Dmitry Berenson, Oliver Kroemer · 2025
Read Planning-Query-Guided Model Generation for Model-Based Deformable Object Manipulation by Alex LaGrassa, Zixuan Huang, Dmitry Berenson, Oliver Kroemer (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-19199.txt.
Introduction
Alex LaGrassa and co-authors teach robots to model soft objects only as closely as the task demands. Their 2025 paper addresses planning with deformable things, where full-fidelity models are slow, yet crude ones fail at delicate work.
Their generator builds a task-specific model whose detail varies by region. A diffusion model reads the start and goal shapes of a planning query and predicts where fine modeling matters. Training data comes from a two-stage search that first trusts a rough prior, then checks choices in closed-loop trials. On a tree-shaping task the method doubled planning speed with only a small drop in success against a full-resolution model. The authors aim at reusing past planning data for new tasks.
Catalog record
| CREATOR | Alex LaGrassa, Zixuan Huang, Dmitry Berenson, Oliver Kroemer |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-19199
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Planning-Query-Guided Model Generation for Model-Based Deformable Object Manipulation Alex LaGrassa, Zixuan Huang, Dmitry Berenson, Oliver Kroemer Abstract Efficient planning in high-dimensional spaces, such as those involving deformable objects, requires computationally tractable yet sufficiently expressive dynamics models. This paper introduces a method that automatically generates task-specific, spatially adaptive dynamics models by learning which regions of the object require high-resolution modeling to achieve good task performance for a given planning query . Task performance depends on the complex interplay between the dynamics model, world dynamics, control, and task requirements. Our proposed diffusion-based model generator predicts per-region model resolutions based on start and goal pointclouds that define the planning query. To efficiently collect the data for learning this mapping, a two-stage process optimizes resolution using predictive dynamics as a prior before directly optimizing using closed-loop performance. On a tree-manipulation task, our method doubles planning speed with only a small decrease in task performance over using a full-resolution model. This approach informs a path towards using previous planning and control data to generate computationally efficient yet sufficiently expressive dynamics models for new tasks. I Introduction Deep learning on general-purpose data representations, such as images and point clouds, has enabled predictive world models of high-dimensional deformable objects with complex dynamics [ 1 , 2 ] . Furthermore, scalable generative models trained on large, diverse datasets can broadly generalize across domains. However, computation on high-dimensional data comes at a cost. Many large models have slow inference times due to using billions of parameters. These computation times make them impractical for applications like model-based planning, which require repeated sequential evaluation for good performance. This paper focuses on graph neural networks (GNNs), commonly used for modeling deformable objects [ 3 , 4 , 5 , 1 ] , operate on graph-based representation whose size, defined by the number of vertices and edges, determines their resolution . Higher-resolution graphs capture more detail at greater computational cost. State representations and corresponding models must strike a balance. They must be expressive enough to solve the task, but not so large that planning becomes computationally intractable [ 6 , 7 ] . GNNs are particularly powerful since they can generalize across highly different graph structures and node quantities [ 8 ] . Fig. 1 : Given a planning query and a segmented pointcloud, the model generator outputs a dynamics model that simplifies regions that are irrelevant to task performance. In this example, the circled branch moves between the start and goal. The model generator outputs a graph that represents the end of that branch and the base in high resolution, and simplifies all other regions. Although the tree has 1317 particles, only 227 particles are needed to represent the tree dynamics accurately. The simplified model is then used for faster planning. This paper addresses the problem of simplifying predictive models without sacrificing end-to-end closed-loop task performance by learning which parts of the state needs high-resolution modeling and which can be safely simplified to a lower resolution. Task-relevant simplification depends on a complex interplay between the dynamics model, task objectives, underlying world dynamics, planner, and low-level controller. Consider a robot tasked with moving a particular tree branch to inspect or remove fruit from it (Fig. 3 ). Intuitively, some regions, such as branches far from the apple, may be modeled coarsely without harming performance. However, the required resolutions depend on factors such as if they need to be pushed or if they may be moved by other branches to solve the task. Resolution-adaptive representations from computer graphics are a helpful starting point for our work. However, region-specific resolution selection in planning depends on the ability to make good decisions during planning, instead of the more directly optimizable geometric fidelity. In fact, prior work has shown that low-resolution models can be surprisingly effective for control in some cases [ 9 , 10 ] . Our contributions are as follows: 1. A diffusion-based model generator that outputs region-specific resolutions that optimize computational efficiency for a given planning query with minimal impact to closed-loop task performance. The input is represented as a start and goal point cloud. The output is a vector that specifies how to encode the point cloud into a graph for planning. To train this model generator, we construct a dataset of planning queries paired with optimized resolution vectors 2. A two-stage resolution optimization procedure for generating this training data. Since evaluating performance with closed-loop MPC is computationally expensive, the first stage produces a prior based on dynamics model accuracy, which is then refined using closed-loop task performance. II Related Work Task-Relevant Model Selection: The most relevant area of work to ours is model selection: selecting a dynamics model that a controller uses for optimization. Predictive accuracy is not necessary or even necessarily correlated with task performance. Only some parts of the state-space are relevant to the planning query. Some methods in search-based planning select the necessary DoF needed at various parts of the search to compute a plan to a specified goal [ 11 , 12 ] or use a heuristic such as the presence of contacts to decide when to use a more accurate but slower physics model [ 13 ] . In high-dimensional problems such as deformable object manipulation, the relationship between predictive accuracy and planning performance is even more complex [ 9 ] . With access to a complete and accurate model, an assumption we do not make, some methods can directly optimize state-space and action-space reductions that preserve task performance while making computation tractable [ 14 , 15 , 16 ] . Without accurate models, optimal model selection for task performance requires evaluating controller performance on the true dynamical system. Since evaluation on the true system can be expensive, [ 17 ] frames model selection as a multi-arm bandit problem where the utility of a model is measured by how much a generated command reduces the current error to the goal. Although this work acknowledges the computational benefit of more simplified models, unlike in our method, computational cost is not reflected in the model selection process. However, [ 18 ] incorporates the higher computational cost of higher resolution models in their optimization process by providing a fixed computation budget for each trial. Our proposed method optimizes model selection for task performance while accounting for computational cost by optimizing model complexity subject to a specifiable sub-optimality tolerance. Additionally, our method contributes spatially local resolution optimization, which allows some object regions to be higher resolution than others, while [ 18 ] optimizes a global resolution parameter for a granular media manipulation task, which can also be made more computationally tractable using neural density fields rather than particle-based models [ 19 ] . However, many common deformable objects such as clothes and rope, are better represented when some object regions have higher particle density than others [ 20 ] . Our method informs tradeoffs between model complexity and task performance using the knowledge of semantically distinct regions of the object. Adaptive Mesh Simplification for Predictive Dynamics Models: Our strategies for adapting model resolution builds on existing methods that prioritize geometric and physical fidelity. The core objective of geometric simplification is to preserve spatial topology as much as possible while reducing mesh