Toward Scalable Multirobot Control: Fast Policy Learning in Distributed MPC

Xinglong Zhang, Wei Pan, Cong Li, Xin Xu, Xiangke Wang, Ronghua Zhang, Dewen Hu · 2024

Read Toward Scalable Multirobot Control: Fast Policy Learning in Distributed MPC by Xinglong Zhang, Wei Pan, Cong Li, Xin Xu, Xiangke Wang, Ronghua Zhang, Dewen Hu (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-19669.txt.

Introduction

Zhang leads a team making predictive control for robot swarms learnable instead of solver-bound in a 2024 paper. Distributed model predictive control coordinates many robots well, but its online optimization is heavy. Big, nonlinear swarms break its scalability.

Their framework learns explicit control policies with a distributed actor-critic scheme. Policies update step by step in a receding-horizon fashion, with closed-loop stability guaranteed. Learned policies then deploy directly, even to swarms of different sizes. A force-field twist adds safe learning for collision-sensitive settings. Experiments covered wheeled robots and drones at scales up to ten thousand units. The authors report fast learning and deployment. The method promises real-time control for huge robot teams.

Catalog record

CREATORXinglong Zhang, Wei Pan, Cong Li, Xin Xu, Xiangke Wang, Ronghua Zhang, Dewen Hu
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE132 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-19669

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Toward Scalable Multirobot Control: Fast Policy Learning in Distributed MPC Xinglong Zhang, Wei Pan, Cong Li, Xin Xu, Xiangke Wang, Ronghua Zhang, Dewen Hu Abstract Distributed model predictive control (DMPC) is promising in achieving optimal cooperative control in multirobot systems (MRS). However, real-time DMPC implementation relies on numerical optimization tools to periodically calculate local control sequences online. This process is computationally demanding and lacks scalability for large-scale, nonlinear MRS. This article proposes a novel distributed learning-based predictive control (DLPC) framework for scalable multirobot control. Unlike conventional DMPC methods that calculate open-loop control sequences, our approach centers around a computationally fast and efficient distributed policy learning algorithm that generates explicit closed-loop DMPC policies for MRS without using numerical solvers. The policy learning is executed incrementally and forward in time in each prediction interval through an online distributed actor-critic implementation. The control policies are successively updated in a receding-horizon manner, enabling fast and efficient policy learning with the closed-loop stability guarantee. The learned control policies could be deployed online to MRS with varying robot scales, enhancing scalability and transferability for large-scale MRS. Furthermore, we extend our methodology to address the multirobot safe learning challenge through a force field-inspired policy learning approach. We validate our approach’s effectiveness, scalability, and efficiency through extensive experiments on cooperative tasks of large-scale wheeled robots and multirotor drones. Our results demonstrate the rapid learning and deployment of DMPC policies for MRS with scales up to 10,000 units. Source codes and multimedia materials are available online at https://sites.google.com/view/pl-dpc/ . I Introduction Multirobot systems (MRS) represent a collective of autonomous robots interconnected through communication networks [ 1 ] , enabling collaborative control tasks. This networked structure endows MRS with the capability to pursue global objectives that exceed the capabilities of individual robots. However, achieving optimal coordination in MRS often involves optimizing a global performance index, which poses a significant large-scale optimal control problem [ 2 ] . Centralized solutions to the above problems may struggle to adequately address the complexities arising from interactions between multiple robots [ 3 ] . Consequently, recent decades have witnessed considerable attention in developing distributed optimal control approaches for MRS [ 4 , 5 , 6 , 7 , 8 , 9 , 10 ] . Among them, distributed model predictive control (distributed MPC, DMPC) is a primary methodology for multirobot control under constraints [ 11 , 12 , 13 , 14 , 15 ] , formulating control problems as optimization tasks over prediction horizons to achieve optimized performance. In DMPC, each robot calculates the local control sequences online by solving the optimization problem with numerical solvers [ 16 , 17 , 18 , 19 ] , which could be computationally intensive for nonlinear MRS. In real-world applications such as small-size mobile robots, the computational efficiency and real-time performance optimization hinge on several influencing factors: i) Onboard computing resources are inherently limited in scale and processing capability; ii) The nonlinear dynamics and complex interactions among robots compound the computational load. Consequently, the real-time resolution of large-scale DMPC problems presents significant challenges [ 17 , 18 ] , deemed inapplicable for real-world large-scale yet small-sized MRS. This motivates us to propose a computationally fast and efficient policy learning approach to generate explicit closed-loop DMPC policies, rather than calculating the open-loop control sequences, i.e., implicit policies, with numerical solvers. To our knowledge, no previous work has developed policy learning techniques for designing the DMPC policies. As a class of policy learning techniques, reinforcement learning (RL) has made significant progress for robot control (cf. [ 20 , 21 , 22 , 23 ] ). RL enables the acquisition of control policies directly from data [ 20 ] , or through model predictions [ 21 ] , to improve sample efficiency. In the context of multirobot control, numerous approaches have been proposed leveraging multiagent RL paradigms [ 24 , 25 , 26 ] . Despite the prevalence of deep RL frameworks, such as asynchronous advantage actor-critic [ 24 ] , challenges persist in training scalability, sample efficiency, and the absence of closed-loop guarantees in policy learning. These challenges highlight the crucial need for scalable policy learning with stability guarantees. Our approach achieves this goal from two perspectives. First, we design a distributed online actor-critic learning algorithm in which the training process is executed incrementally to generate control policies efficiently. Second, we introduce a policy training approach grounded in control theory, integrating the receding horizon optimization strategy into policy updates. This ensures closed-loop stability and improves learning efficiency. In addition to guaranteed stability, safety constraints, such as collision avoidance, must be met to ensure the persistent and reliable operation of MRS. However, ensuring control safety in RL remains a nontrivial task, even in the model-based scenario [ 27 , 28 , 29 , 30 ] . Recent efforts in safe learning control have focused primarily on the centralized control structure [ 29 ] . Few works have been dedicated to the safe multiagent policy learning [ 31 , 32 ] , drawing inspiration from cost-shaping-based RL [ 27 , 28 ] . However, the cost-shaping design would lead to weight divergence in the actor-critic framework [ 33 ] , which lacks online learning ability and safety guarantees. Gaining insights from interior point optimization [ 34 ] , we design a novel safe policy learning algorithm with a force field-inspired policy structure. This design can balance the objective and constraint-associated forces acting on the MRS during policy learning, ensuring safe learning with physical interpretations. The contributions of this article are summarized below. i) We propose a novel distributed learning-based predictive control (DLPC) framework for large-scale MRS. In contrast to conventional numerical DMPC, which calculates open-loop control sequences, our approach generates closed-loop DMPC policies without relying on numerical solvers. The optimization problem of DMPC within each prediction interval is decomposed into several sequential subproblems and solved by policy learning. The control policies are composed of parameterized functions capable of online learning and deployment to scenarios with varying robot scales, enhancing scalability and transferability for large-scale MRS. ii) A computationally fast and efficient distributed policy learning algorithm is developed, integrating the receding horizon optimization strategy into policy updates. In each prediction interval, policy learning is executed forward in time rather than backward in time with a distributed incremental actor-critic implementation, enabling fast online policy updates. The control policies generated from each prediction interval are successively refined in subsequent intervals to improve learning efficiency, fundamentally different from the common independent problem-solving paradigm of DMPC in different prediction intervals. Compared with numerical DMPC, our approach significantly reduces the computational load through fast and efficient policy learning while maintaining closed-loop stability. iii) We further address the challenge of safe policy learning in MRS through a force field-inspired policy design, which has clear physical interpretations to balance the objective force and the constraint force acting in MRS, enabling online policy learning and policy deployment with safety and robustness guarantees. iv) We numerically and experimentally validate our method’s superior sim-to-real transferability and scalability in large-scale multirobot control. In particular, we have shown on different computing platforms, i.e., a laptop and a Raspberry PI 5, that our approach efficiently learns near-optimal formation policies for MRS with scales up to

Read the complete text