Meta-Optimization and Program Search using Language Models for Task and Motion Planning
Denis Shcherba Eckart Cobo-Briesewitz TU Berlin TU Berlin [email protected] [email protected] Cornelius V. Braun Marc Toussaint TU Berlin TU Berlin · 2025
Read Meta-Optimization and Program Search using Language Models for Task and Motion Planning by Denis Shcherba Eckart Cobo-Briesewitz TU Berlin TU Berlin [email protected] [email protected] Cornelius V. Braun Marc Toussaint TU Berlin TU Berlin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-03725.txt.
Introduction
Denis Shcherba and co-authors let language models search for robot plans written as programs in a paper from 2025. Robots must reason about tasks and motion together. Too much abstraction makes plans vague; too little drowns the planner in joint angles. Foundation models plan fast but need the right interface.
Their method searches over programs that pose trajectory optimization problems. The language model proposes plan structure; the optimizer fills in real motion. Meta-optimization tunes how that search itself runs. Natural language instructions can drive the whole chain. Program-shaped plans stay readable and fixable, unlike raw joint commands. The interface question in robot planning gets a concrete, testable answer.
Catalog record
| CREATOR | Denis Shcherba Eckart Cobo-Briesewitz TU Berlin TU Berlin [email protected] [email protected] Cornelius V. Braun Marc Toussaint TU Berlin TU Berlin |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-03725
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Meta-Optimization and Program Search using Language Models for Task and Motion Planning Denis Shcherba Eckart Cobo-Briesewitz TU Berlin TU Berlin [email protected] [email protected] Cornelius V. Braun Marc Toussaint TU Berlin TU Berlin Abstract Intelligent interaction with the real world requires robotic agents to jointly reason over high-level plans and low-level controls. Task and motion planning (TAMP) addresses this by combining symbolic planning and continuous trajectory generation. Recently, foundation model approaches to TAMP have presented impressive results, including fast planning times and the execution of natural language instructions. Yet, the optimal interface between high-level planning and low-level motion generation remains an open question: prior approaches are limited by either too much abstraction (e.g., chaining simplified skill primitives) or a lack thereof (e.g., direct joint angle prediction). Our method introduces a novel technique employing a form of meta-optimization to address these issues by: (i) using program search over trajectory optimization problems as an interface between a foundation model and robot control, and (ii) leveraging a zero-order method to optimize numerical parameters in the foundation model output. Results on challenging object manipulation and drawing tasks confirm that our proposed method improves over prior TAMP approaches. Keywords: Task and Motion Planning, LLMs as Optimizers, Trajectory Optimization 1 Introduction Intelligent interaction with the world requires flexible action plans that are robust and satisfy real-world constraints. To achieve a long-horizon goal, agents are required not only to reason over symbolic decisions, but also to make geometry-based decisions to seamlessly integrate high-level reasoning and low-level control. The resulting decision-making problems are challenging to solve due to the combinatorial complexity of the search space with increasing plan lengths. Commonly, the challenges of these decision-making problems are approached through the lens of integrated task and motion planning (TAMP) [ 1 , 2 , 4 , 5 ] . The core idea of TAMP is to make long-horizon sequential manipulation planning tractable by introducing a symbolic domain that links discrete high-level decisions with continuous motions [ 6 ] . Over the past decades, this framework has been shown capable of solving a wide variety of tasks – from table-top manipulation [ 7 , 8 , 3 ] , to puzzle solving [ 10 ] , to architectural construction [ 11 ] . While highly general, classical TAMP approaches trade expressivity for tractability: they fix the mapping between symbolic skills and the underlying trajectory optimization, a function that must be designed by an expert engineer. This imposes two key limitations: First, solution quality depends on the designer’s insight and experience. Second, the flexibility of these methods is severely limited because the reasoning over NLPs occurs only at the symbolical level, while the specific timings, scalings, and other parameters of the trajectory constraints are abstracted away during planning. This rigid separation often leads to plans that are only locally optimal or not feasible at all [ 1 , 12 ] . Figure 1: Overview diagram of our method MOPS and its empirical performance. Left: MOPS solves the problem by iterating a meta-optimization loop: an LLM optimizes the selection of constraints, a blackbox optimizer (BBO) improves the continuous parameters of these functions and a gradient-based method solves the induced non-linear program (NLP) for a trajectory, which is then simulated to compute the full cost. Based on the back-propagated costs, the constraints of the NLP are adapted until convergence. Right: Average normalized performance across two domains, with three tasks each. MOPS outperforms prior methods that search over action sequences (PRoC3S) or simple code snippets (CaP). Results for each task are averaged over 5 independent runs ( \pm 1.96 standard deviation). Recently, foundation model (FM)-based methods have been presented as an alternative to classical TAMP methods, since they do not require hand-crafted symbolic predicates to solve tasks. Instead, they can select control policies from a pre-trained repertoire [ 13 ] , write code that controls the robot [ 14 ] , or directly predict joint states [ 15 ] given a high-level goal specification. While FMs have now been applied successfully to a wide range of tasks, most prior approaches are built on simplifying and restrictive assumptions. A common limiting factor is their reliance on atomic skills, originating from the lack of fine - grained spatial reasoning in current foundation models [ 14 , 16 , 17 , 18 ] . This allows them to solve simple pick-and-place tasks, but fails for tasks that require precise placement locations or low tolerances for errors. Curtis et al. [12] address this shortcoming by formulating language-instructed TAMP as a constraint satisfaction problem. To solve this problem, the authors propose using LLMs to reason over parameterized skills, for which the parameters can be sampled so that the specified constraints hold during manipulation. However, their approach requires human users to specify relevant task constraints and uses uniform sampling to obtain skill parameters. The result is sample - inefficient rejection sampling [ 19 ] , and an approach that heavily hinges on the capabilities of the user to specify all relevant constraints upfront, which can be challenging for many real-world tasks. In this paper, we introduce Meta-Optimization and Program Search using Language Models for TAMP (MOPS) , a novel approach that overcomes these limitations by treating the TAMP problem as a meta-optimization problem in which both the motion constraints and the resulting trajectories are explicit optimization targets. Concretely, we solve this problem by nesting three optimization levels. In the first step, we use an FM as an optimizer [ 20 ] to propose parameterized constraint functions c(x,\alpha_{c}) that define a nonlinear mathematical program (NLP) and sensible initialization heuristics for each numerical constraint parameter in a plan. Second, we perform black-box optimization to optimize the numerical constraint parameters \alpha_{c} for overall task success and reward. Third, we perform gradient-based trajectory optimization. This step solves the fully defined NLP to produce smooth, and collision-free trajectories that satisfy all task constraints c(x,\alpha_{c}) . Our contributions are summarized as follows: • We propose a novel perspective on TAMP by formulating language conditioned TAMP as a search over constraint sequences instead of actions sequences ( Section 2.1 ). • We introduce a novel multi-level optimization method for sequential robotic manipulation that combines foundation models with parameterized NLPs and gradient-based trajectory optimization, which enables efficient complex robot manipulation ( Section 2.3 ). • We validate our method, MOPS, on a range of problems and demonstrate that it improves over prior TAMP approaches ( Section 4.2 ). 2 Meta-Optimization and Program Search using Language Models for TAMP This section introduces our method, MOPS (Meta-Optimization and Program Search), a hierarchical framework that casts language conditioned TAMP as a meta-optimization problem. At the top level, a foundation model (FM) performs a semantic search over discrete constraints; at the middle level, a black-box optimizer refines continuous constraint parameters; and at the lowest level, a gradient-based solver produces a smooth trajectory. 2.1 Problem Formulation We start by defining the language-conditioned TAMP setting, which follows the general TAMP problem structure [ 1 ] , but assumes that the task goal is specified in natural language instead of a planning language such as PDDL. A language-conditioned TAMP problem is specified by the tuple ({\cal S},{\cal C},\,s_{0},\,G,\,{\cal J},\,g,\,h) . Here, {\cal S} denotes the fully observable