Graph-of-Constraints Model Predictive Control for Reactive Multi-agent Task and Motion Planning
Anastasios Manganaris, Jeremy Lu, Ahmed H. Qureshi, Suresh Jagannathan · 2026
Read Graph-of-Constraints Model Predictive Control for Reactive Multi-agent Task and Motion Planning by Anastasios Manganaris, Jeremy Lu, Ahmed H. Qureshi, Suresh Jagannathan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-18400.txt.
Introduction
Manganaris and co-authors present Graph-of-Constraints Model Predictive Control for multi-agent task and motion planning. Constraint chains are central to such planning, but past methods assume fixed agent assignments. Disturbances that change allocations break those static schemes. Their framework supports partial task orders, dynamic coordination, and online recovery within model predictive control. Constraints over tracked 3D keypoints let it solve varied manipulation tasks from vision alone.
Catalog record
| CREATOR | Anastasios Manganaris, Jeremy Lu, Ahmed H. Qureshi, Suresh Jagannathan |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37245 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.18400
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Graph-of-Constraints Model Predictive Control for Reactive Multi-agent Task and Motion Planning Anastasios Manganaris, Jeremy Lu, Ahmed H. Qureshi, Suresh Jagannathan Abstract Sequences of interdependent geometric constraints are central to many multi-agent Task and Motion Planning (TAMP) problems. However, existing methods for handling such constraint sequences struggle with partially ordered tasks and dynamic agent assignments. They typically assume static assignments and cannot adapt when disturbances alter task allocations. To overcome these limitations, we introduce GoC-MPC ( GoC-MPC ), a generalized sequence-of-constraints framework integrated with MPC. GoC-MPC naturally supports partially ordered tasks, dynamic agent coordination, and disturbance recovery. By defining constraints over tracked 3D keypoints, our method robustly solves diverse multi-agent manipulation tasks—coordinating agents and adapting online from visual observations alone, without relying on training data or environment models. Experiments demonstrate that GoC-MPC achieves higher success rates, significantly faster TAMP computation, and shorter overall paths compared to recent baselines, establishing it as an efficient and robust solution for multi-agent manipulation under real-world disturbances. Our supplementary video and code can be found at https://sites.google.com/view/goc-mpc/home . I INTRODUCTION The effective deployment of multi-robot teams promises to automate significantly more complex and useful real-world tasks. For example, two robotic arms benefit from a substantially larger effective workspace, the ability to manipulate objects designed for two-handed human use, and the capacity to perform work in parallel. Scaling to larger teams of arms or mobile manipulators further increases throughput and task diversity. However, these capabilities come at the cost of much higher complexity in performing TAMP ( TAMP ). Despite this complexity, a variety of successful methods for TAMP have been developed and remain applicable in the multi-robot setting. A particularly well-studied family of approaches leverages modern constrained optimization to directly solve TAMP problems formulated as sequences of constraints [ 28 , 23 , 24 ] . Just as trajectory optimization underlies online control when used in a receding-horizon loop (e.g., with MPC ( MPC )), optimization-based formulations of TAMP have also been extended to reactive execution, enabling high-level symbolic plans and low-level continuous motions to be recomputed during runtime [ 27 ] . This line of work demonstrates how re-solving optimization problems in response to new information provides a natural mechanism for handling failures and disturbances. Moreover, recent research has shown that defining constraints in terms of universally applicable geometric primitives—such as spatial keypoints tracked in the robot’s workspace—enables general-purpose manipulation across a wide range of tasks. This allows such methods to achieve impressive generality and avoid the need for precise system models or extensive task-specific training data [ 12 ] . Fig. 1: GoC ( GoC ) can naturally express partially-ordered multi-agent tasks using a DAG of arbitrary system constraints. Our method, GoC-MPC , can use a GoC with constraints defined in terms of workspace keypoints to find optimal solutions for general multi-agent manipulation tasks using only visual observations. For this two-cup manipulation task, GoC-MPC finds an initial solution in 0.373 seconds, and all subsequent solutions in 0.065 seconds on average, demonstrating extremely fast performance. Although these methods have achieved success in single-agent domains, we identify two fundamental limitations that currently limit their usefulness for general multi-agent tasks. In particular, using singular sequences of constraints imposes a total-order on the steps carried out by the agents and it imposes a static assignment of agents to steps. By artificially imposing ordering constraints between steps that could be otherwise completed in parallel, such as the first two steps in Fig. 1 , the system can unnecessarily halt the progress of one agent when waiting for the other to complete its task. Disturbances can exacerbate this issue arbitrarily, by preventing a single agent from completing an early assigned task, and therefore delaying the entire team, even if they are not affected by the disturbance at all. Static agent assignments can also contribute to this problem, as a poor choice in static assignments without considering the current state of the system can lead to unnecessary extra work or even complete infeasibility. Disturbances should also be able to prompt reassignments of agents if necessary. In this work, we address these limitations by realizing the generalization of Sequences-of-Constraints to Graphs -of-Constraints (GoCs). GoC are DAG of constraints derived from symbolic, high-level actions, and therefore can naturally represent partially-ordered sets of task steps. Furthermore, constraints within a GoC are defined with respect to a dynamic assignment of agents to subtasks, which allows for optimizing over this assignment when obtaining the solution. To solve problems formulated with GoC , we also contribute GoC-MPC , which decomposes and solves the optimization problem resulting from a GoC at real-time speeds, therefore enabling reactive multi-agent TAMP . We evaluate our algorithm on several bimanual manipulation tasks that take advantage of properties of the GoC formulation, including block stacking (featuring parallelized pick-ups and sequenced placements), transferring liquids in cups (featuring coordination between agents based on the objects they are manipulating), and tablecloth folding (featuring exact synchronization of agents). II RELATED WORK In general, the field of TAMP is concerned with extending symbolic, discrete planning algorithms [ 6 , 31 , 11 ] with continuous motion planning [ 13 , 20 ] to solve complex, long-horizon robotics tasks [ 7 , 29 ] . Approaches to TAMP include constraint-based [ 5 ] , sampling-based [ 7 ] , and the aforementioned optimization-based methods [ 34 ] . The latter TAMP methods formulate a continuous or mixed-integer optimization problem [ 28 , 23 , 24 , 34 ] and therefore benefit from naturally being able to incorporate diverse costs and constraints. Optimization-based approaches have also been extended to multi-agent settings [ 1 ] . Reactive TAMP is a closely related area to TAMP that, instead of assuming that TAMP solutions can be perfectly executed without failure or external disturbances, focuses on the ability to quickly recompute plans online during execution. Specific strategies include synthesizing hierarchical policies for symbolic actions via reactive synthesis from temporal logic specifications [ 15 , 9 , 10 ] , defining object-relative plans that remain valid despite object motion [ 18 ] , and re-solving optimization problems derived from symbolic skeleton plans [ 3 , 32 , 33 , 27 ] . To our knowledge, we contribute the first multi-agent-capable reactive TAMP method belonging to this latter category. We also note that this last class of solutions that reactively optimize with respect to a task skeleton has received increased attention as a way to relax the heavy modeling assumptions of traditional TAMP . In particular, ReKep [ 12 ] are constraints on the geometric relationships of keypoints within an environment, which enables the execution of general manipulation skills without requiring exhaustive world modeling or large, task-specific datasets as with other methods [ 17 , 22 , 4 ] . Our method is also designed to use this constraint representation, and therefore achieves similar generality, while still addressing the gaps in prior reactive TAMP methods for supporting dynamic agent assignment and parallelizable task structures. III METHODS In this section, we present our approach for achieving reactive multi-agent TAMP . We begin by discussing the preliminaries for our method, including necessary notations and the limitations of prior methods in supporting multi-agent tasks (Section III-A ). We then