Jointly Learning Predicates and Actions Enables Zero-Shot Skill Composition

Benedict Quartey, Sebastian Castro, Eric Rosen, Wil Thomason, George Konidaris, Stefanie Tellex, Brown University Robotics & AI Institute · 2026

Read Jointly Learning Predicates and Actions Enables Zero-Shot Skill Composition by Benedict Quartey, Sebastian Castro, Eric Rosen, Wil Thomason, George Konidaris, Stefanie Tellex, Brown University Robotics & AI Institute (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-20648.txt.

Introduction

Benedict Quartey and five colleagues propose Predicate Action Skills, a way for robots to compose learned skills without retraining. Learning from demonstration often fails on new combos of known skills, since policies model motion alone. Their PACTS policies jointly model action paths and the symbolic results those actions bring about. Generating both together improves motion and result prediction, and planning over the joint model enables zero-shot composition of skills.

Catalog record

CREATORBenedict Quartey, Sebastian Castro, Eric Rosen, Wil Thomason, George Konidaris, Stefanie Tellex, Brown University Robotics & AI Institute
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41437
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.20648

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Jointly Learning Predicates and Actions Enables Zero-Shot Skill Composition Benedict Quartey, Sebastian Castro, Eric Rosen, Wil Thomason, George Konidaris, Stefanie Tellex, Brown University Robotics & AI Institute Abstract Learning from Demonstration (LfD) enables robots to learn complex behaviors from expert examples, yet existing approaches often fail to generalize to new compositions of known skills without retraining. Modern generative policies model distributions over action trajectories alone, thus are unable to reason about the symbolic outcomes required for robust composition. We propose that skills should jointly model action trajectories and the symbolic outcomes they induce. To address this gap, we introduce Predicate-Action Skills ( Pacts ), a class of closed-loop visuomotor policies that model skills as a joint generative process over action and predicate belief trajectories, producing coherent action–outcome rollouts within a single model. Jointly generating actions and predicates enables Pacts to learn internal representations that improve both action generation and predicate classification. Furthermore, we demonstrate zero-shot composition of learned skills via planning by leveraging online predicate predictions from Pacts as a symbolic interface for sequencing and monitoring execution. Project website: planpacts.github.io Fig. 1: Predicate-Action Skills . Conditioned on current observations \mathbf{o} , we model skills as a joint generative process over an action trajectory \mathbf{x} and a predicate-belief trajectory \mathbf{z} by learning the coupled distribution p(\mathbf{x},\mathbf{z}\mid\mathbf{o}) . Starting from noise ( \mathbf{x}_{T} , \mathbf{z}_{T} ), our model iteratively refines both modalities to produce temporally coherent action–outcome rollouts (\mathbf{x}_{0},\mathbf{z}_{0}) . The resulting predicate-belief trajectory \mathbf{z}_{0} provides an online symbolic interface for monitoring skill execution and planning-based skill composition using off-the-shelf planners. I Introduction Learning from Demonstration (LfD) has become a standard approach for teaching robots complex behaviors. Recent generative visuomotor policies are particularly compelling in this setting: by modeling distributions over action trajectories, they effectively capture multi-modal behavior. Yet, the gap between learning short-horizon behaviors and solving long-horizon, multi-step tasks remains open. In long-horizon execution, distribution shift compounds over time and the space of possible task variations and compositions grows combinatorially. Closing this gap typically requires large-scale demonstration datasets that cover many such variations or retraining whenever the desired composition changes. A promising route to long-horizon generalization is to introduce modular structure: reusable skills that capture short-horizon behavior, and symbolic predicates that describe state and skill effects in a form amenable to planning. By abstracting both temporal behavior (via skills) and state (via predicates), prior work has shown that robots can generalize to unseen task compositions by sequencing learned skills to achieve goals [ 14 , 8 ] . However, these prior works typically decouple these components, training one model to generate actions and a separate model to classify predicates from observations. This separation obscures a central fact about manipulation: actions and symbolic outcomes are tightly coupled . For composition, what matters is not only which actions are plausible, but which symbolic conditions those actions will make true. This paper starts from a simple alternative viewpoint: skills should not be represented only as a distribution over action trajectories, but as distributions over action–outcome rollouts . We represent outcomes as a predicate-belief trajectory —a compact trace of soft truth values over predicates that describes what the model expects to become true (and with what confidence) as the skill unfolds. Given an observation \mathbf{o} , we therefore model a coupled distribution over an action trajectory \mathbf{x} and an outcome trace \mathbf{z} : (\mathbf{x},\mathbf{z})\sim p(\mathbf{x},\mathbf{z}\mid\mathbf{o}). Sampling from this joint distribution yields coherent pairs, as the same sample that commits to an action mode also commits to the corresponding symbolic trace. We instantiate this idea with Predicate-Action Skills ( Pacts ), closed-loop visuomotor policies that model skills as a joint generative process over action trajectories and predicate-belief trajectories. Pacts iteratively refines both modalities to produce temporally aligned rollouts (\mathbf{x}_{0},\mathbf{z}_{0}) within a single model (see Fig. 1 ). Crucially, the resulting predicate-belief trajectory \mathbf{z}_{0} provides a practical interface for planning-based skill composition: it can be used to check preconditions and effects, monitor execution, and trigger replanning when expectations are violated—enabling zero-shot composition of learned skills with off-the-shelf planners. Our formulation supports both diffusion-style denoising and flow-matching objectives, and instantiates naturally with different sequence backbones (e.g., UNet- and Transformer-based policies). We systematically evaluate Pacts on a compositional 2D benchmark for comprehensive ablations and on two 3D RoboMimic tasks [ 17 , 16 ] , spanning a broad grid of model variants. Across these settings, jointly generating actions and predicate beliefs maintains competitive performance and often improves both action performance and predicate prediction relative to action-only generative policies and pipelines that pair action generation with a separate predicate predictor. Finally, we demonstrate planning-based composition of learned subskills in simulation and in a real-world environment. In summary, we make three contributions: (1) a joint generative skill formulation that models action trajectories and predicate traces to produce coherent action–outcome rollouts; (2) planning-based skill composition using predicate-belief traces as an online interface for sequencing and monitoring with off-the-shelf planners; and (3) an open-source skill segmentation and labeling toolkit that converts monolithic demonstrations into skill-centric datasets for joint (\mathbf{x},\mathbf{z}) learning in real-world deployments. II Related Work Generative visuomotor policies for imitation learning Behavior cloning learns skills by mapping observations to actions [ 25 ] , and recent work increasingly models distributions over action trajectories to capture multi-modality and long-horizon structure. Diffusion-based policies [ 4 ] and conditional flow matching [ 5 ] are prominent instantiations, with complementary approaches improving data efficiency via pretrained vision-language representations [ 28 , 2 ] or architectural/representational structure (e.g., equivariance and object-/affordance-centric models) [ 26 , 27 , 23 ] . Diffusion has also been used for planning: Diffuser [ 13 ] denoises state–action trajectories to enable flexible test-time conditioning, whereas Diffusion Policy [ 4 ] treats control as learning the distribution over action sequences, conditioning on observations without denoising them for efficient real-time inference. We adopt this conditional-policy perspective, but depart from action-only modeling by learning a coupled generator p(\mathbf{x},\mathbf{z}\mid\mathbf{o}) that jointly samples actions and a low-dimensional predicate-belief rollout \mathbf{z} aligned with PDDL preconditions/effects, yielding an outcome-aware interface for execution-time monitoring and planning-based composition. Vision–language–action models similarly couple control with language semantics, but their semantics are typically part of the conditioning interface rather than an explicit planner-consumable outcome trajectory. Skills, abstraction, and planning-based composition Automated task planning in robotics [ 10 ] specifies a domain (often in PDDL [ 9 ] ) in terms of abstract actions with preconditions and effects, and uses search to synthesize sequences that achieve goal conditions. In robotic systems, these abstract actions are typically instantiated as hand-designed controllers (e.g., finite-state machines [ 11 ] or behavior trees [ 6 ] ), and predicates are grounded through engineered perception and state estimation modules [ 20 ] . Several frameworks support such symbolic planning architectures [ 7 , 18 , 19 , 21 , 22 ] . Recent learning-based approaches aim to reduce manual engineering by learning both skill policies and predicate evaluators from data: policies are learned via behavior cloning or reinforcement learning, while predicates are grounded either with supervised classifiers or with pretrained vision–language models that infer symbolic propositions from observations. A common design pattern in learned planning pipelines is to train a policy separately from a predicate evaluator (classifier or pretrained

Read the complete text