Large-Language-Model-Guided State Estimation for Partially Observable Task and Motion Planning

Yoonwoo Kim, Raghav Arora, Roberto Martín-Martín, Peter Stone, Ben Abbatematteo, Yoonchang Sung · 2026

Read Large-Language-Model-Guided State Estimation for Partially Observable Task and Motion Planning by Yoonwoo Kim, Raghav Arora, Roberto Martín-Martín, Peter Stone, Ben Abbatematteo, Yoonchang Sung (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-03704.txt.

Introduction

Yoonwoo Kim and five colleagues let a language model whisper common sense into a robot's map of the unseen. A robot planning in a cluttered room cannot see every object, and naive planners ignore the stray objects they stumble on. Their 2026 framework, CoCo-TAMP, builds two commonsense rules into state estimation: objects have likely homes, and similar objects cluster together while dissimilar ones separate.

Writing those rules by hand would be endless, so a large language model supplies them on the fly. Its guidance shapes a hierarchical estimate of where task-relevant objects probably are, focusing the search. In simulation, planning and execution time fell by nearly two thirds against a commonsense-free baseline. In real-world robot demonstrations, the saving grew to nearly three quarters. Long-horizon chores become tractable when the robot guesses like a person.

Catalog record

CREATORYoonwoo Kim, Raghav Arora, Roberto Martín-Martín, Peter Stone, Ben Abbatematteo, Yoonchang Sung
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49762
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.03704

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Large-Language-Model-Guided State Estimation for Partially Observable Task and Motion Planning Yoonwoo Kim, Raghav Arora, Roberto Martín-Martín, Peter Stone, Ben Abbatematteo, Yoonchang Sung Abstract Robot planning in partially observable environments, where not all objects are known or visible, is a challenging problem, as it requires reasoning under uncertainty through partially observable Markov decision processes. During the execution of a computed plan, a robot may unexpectedly observe task-irrelevant objects, which are typically ignored by naive planners. In this work, we propose incorporating two types of common-sense knowledge: (1) certain objects are more likely to be found in specific locations; and (2) similar objects are likely to be co-located, while dissimilar objects are less likely to be found together. Manually engineering such knowledge is complex, so we explore leveraging the powerful common-sense reasoning capabilities of large language models (LLMs). Our planning and execution framework, CoCo-TAMP, introduces a hierarchical state estimation that uses LLM-guided information to shape the belief over task-relevant objects, enabling efficient solutions to long-horizon task and motion planning problems. In experiments, CoCo-TAMP achieves an average reduction of 62.7 % 62.7\% in planning and execution time in simulation, and 72.6 % 72.6\% in real-world demonstrations, compared to a baseline that does not incorporate either type of common-sense knowledge. I Introduction Robots performing long-horizon manipulation tasks must reason over discrete decisions, such as which objects to interact with, and continuous motions for manipulation and navigation. Task and motion planning (TAMP) [ 8 ] provides a principled approach for such problems. However, in realistic settings with uncertainty over object poses and occlusions, plans from deterministic TAMP solvers can fail. This work addresses the partially observable task and motion planning (PO-TAMP) problem, aiming to enable robots to effectively plan to manipulate objects that may not be directly visible due to partial observability. We introduce CoCo-TAMP , a TAMP system that uses large language models (LLMs) to provide common-sense priors and co-location cues that shape beliefs during planning and execution. While LLMs have limitations as planners [ 21 , 28 ] , they can serve as a rich, approximate source of knowledge [ 11 , 16 ] . LLM-Modulo [ 16 ] provides a framework that leverages the strength of LLMs with external model-based verifiers in a “generate and verify” loop, allowing flexibility in problem specification while guaranteeing the completeness of the framework. Following this paradigm, CoCo-TAMP queries an LLM to form priors over rooms/surfaces via multiple-choice question answering (MCQA) and uses LLM sentence embeddings to build a similarity-based co-location model that propagates evidence across objects. Beliefs are maintained in a hierarchical Bayesian filter over rooms, surfaces, and continuous poses, with a visibility-aware observation model that addresses misses under partial coverage and a lightweight “co-location toggler” that disables co-location when semantics suggest broad dispersion ( e.g. , switches). Integrated with a belief-space planner, these components yield efficient information gathering and execution. In simulation and on a real robot, CoCo-TAMP reduces cumulative planning and execution time by \mathbf{62.7\%} and \mathbf{72.6\%} , respectively, compared to a baseline without LLM priors or co-locations. In summary, this paper makes the following contributions: • We propose an interleaved planning–execution framework for PO-TAMP that leverages LLMs to provide commonsense knowledge. By providing informative priors and guiding belief updates, CoCo-TAMP enables more accurate object state estimation and makes belief-space planning practical under partial observability. • We demonstrate the effectiveness of our approach through large-scale household simulations and real-world robot experiments. Across diverse environments, our framework substantially reduces planning and execution time compared to baselines, and maintains robustness even in adversarial settings where commonsense priors are misleading. II Problem Description We assume a known semantic layout (rooms and surfaces) obtained from semantic SLAM system [ 14 ] , but only partial information about objects is available. The environment contains O manipulable objects, R rooms, and S surfaces. For each task-relevant object k\in\{1,\dots,K\} , let x_{r,t}^{k}\in\mathcal{R} , x_{s,t}^{k}\in\mathcal{S} , and x_{p,t}^{k}\in SE(3) denote its room, surface, and pose at time t , with beliefs \text{bel}(x_{r,t}^{k}) , \text{bel}(x_{s,t}^{k}) be categorical and \text{bel}(x_{p,t}^{k}) be continuous. We are given initial beliefs \text{bel}(x_{r,0}^{k}) , \text{bel}(x_{s,0}^{k}) , and \text{bel}(x_{p,0}^{k}) over K<O task‑relevant objects. Building on SS-Replan [ 10 ] , we model PO‑TAMP as a hybrid discrete–continuous belief‑space stochastic shortest‑path problem [ 1 ] where actions have strictly positive cost and the minimum‑cost plan maximizes success likelihood. CoCo-TAMP takes a TAMP specification (\mathcal{O},\mathcal{P},\mathcal{I},\mathcal{G},\mathcal{A}) , where \mathcal{O} is the set of manipulable objects, \mathcal{P} the predicates, \mathcal{I} the initial literals, \mathcal{G} the goal literals, and \mathcal{A} a finite set of parameterized actions with preconditions and effects. Throughout planning and execution, CoCo-TAMP maintain beliefs over (x_{r,t}^{k},x_{s,t}^{k},x_{p,t}^{k}) for each task‑relevant object k and triggers replanning whenever execution failures occur. PDDLStream [ 9 ] was used as the underlying TAMP planner to couple symbolic actions with streams that sample and certify continuous variables ( e.g. , poses and trajectories). In addition to standard pick, place, and move actions, we include an observation action detect . The detect action is parameterized by the object ?o , surface ?s , room ?r , sampled object pose from \text{bel}(x_{p,0}^{k}) ?pb , base configuration ?bq , head configuration ?hq , and head trajectory ?ht of the robot. Following the self-loop MDP heuristic in SS-Replan, the cost of taking the detect action ( DetectCost ) is set inversely to the belief over the object’s state, which serves as a proxy for detection likelihood ( i.e. , cheaper when belief mass and visibility predict success). Concretely, this cost is computed as \frac{c_{a}}{\text{bel}(x_{r,t}^{k},x_{s,t}^{k},x_{p,0}^{k})} where c_{a}=1 and the resulting cost steers the planner toward informative views. We are particularly interested in improving the efficiency of solving PO-TAMP problems by achieving accurate beliefs guided by LLM-driven initializations and object co-location models. To this end, we evaluate CoCo-TAMP using two metrics: the cumulative planning and execution time until successful task completion, and the number of replanning iterations required to solve the problem. The latter metric is motivated by the fact that the former can vary significantly depending on implementation details, system specifications, and the task itself. As such, the number of execution failures serves as a useful proxy metric for evaluating efficiency. III COCO-TAMP Fig. 1 : The initial beliefs about the semantic locations of objects, \text{bel}(x_{r,0}^{k}) and \text{bel}(x_{s,0}^{k}) , are derived from LLMs, while the initial beliefs about their poses, \text{bel}(x_{p,0}^{k}) , are uniformly distributed across all surfaces. The TAMP problem specification (\mathcal{O,P,I,G,A}) , where the beliefs are incorporated into the cost of the observation action, is provided to a TAMP planner PDDLStream. The planner outputs a plan, which the robot executes. Upon executing the observation action, the co-location toggler determines whether to use the co-location model or not, based on the observed object. Then, the beliefs are updated using the proposed observation model, and the planner replans with the updated beliefs. Planning and execution are repeated until the goal state \mathcal{G} is reached. Further implementation details, derivations, additional experimental results, and link to the code are available at our project page: https://coco-tamp.github.io . The overview of CoCo-TAMP is presented in Figure 1 , which illustrates an integrated planning and execution framework. LLM-guided initial belief generation is discussed in Section III-A . Section III-B describes how the beliefs are updated during execution using

Read the complete text