Scaling up Stability: Reinforcement Learning for Distributed Control of Networked Systems in the Space of Stabilizing Policies

John Cao, Luca Furieri · 2025

Read Scaling up Stability: Reinforcement Learning for Distributed Control of Networked Systems in the Space of Stabilizing Policies by John Cao, Luca Furieri (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-18540.txt.

Introduction

John Cao and Luca Furieri design controllers for networked systems, where many agents must coordinate through local information alone. Reinforcement learning can train such controllers, but learned policies often break stability, scale poorly, or both. Their paper, posted to arXiv in December 2025, builds stability into the policy's shape from the start.

They embed graph neural networks into a Youla-like split of magnitude and direction. The magnitude part is a stable operator: a graph network acting on disturbance feedback. The direction part is a second graph network acting on local observations. The result is a distributed stochastic controller with network-level stability guaranteed by design. They also prove the policy tolerates changes in graph shape and model parameters, and simulations confirm the approach works.

Catalog record

CREATORJohn Cao, Luca Furieri
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-18540

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Scaling up Stability: Reinforcement Learning for Distributed Control of Networked Systems in the Space of Stabilizing Policies John Cao, Luca Furieri Abstract We study distributed control of networked systems through reinforcement learning, where neural policies must be simultaneously scalable, expressive and stabilizing. We introduce a policy parameterization that embeds Graph Neural Networks (GNNs) into a Youla-like magnitude-direction parameterization, yielding distributed stochastic controllers that guarantee network-level closed-loop stability by design. The magnitude is implemented as a stable operator consisting of a GNN acting on disturbance feedback, while the direction is a GNN acting on local observations. We prove robustness of the closed loop to perturbations in both the graph topology and model parameters, and show how to integrate our parameterization with Proximal Policy Optimization. Experiments on a multi-agent navigation task show that policies trained on small networks transfer directly to larger ones and unseen network topologies, achieve higher returns and lower variance than a state-of-the-art MARL baseline while preserving stability. keywords Learning methods for optimal control; Reinforcement learning; Distributed robust controller synthesis; Stability of nonlinear systems; Graph neural networks remark 1 Introduction 1.1 Background The control of large-scale dynamical systems is a fundamental topic in control theory, with applications ranging from UAV coordination ( Qie et al. 2019 ) , search and rescue ( Kumar et al. 2004 ) to warehouse robotics ( Kattepur et al. 2018 ) . Learning-based controllers, such as those obtained via Reinforcement Learning (RL), often outperform traditional controllers on complex systems ( Kaufmann et al. 2023 ; Lee et al. 2020 ) . In distributed settings, they are typically implemented under the centralized training with decentralized execution (CTDE) paradigm, often using actor-critic methods ( Konda and Tsitsiklis 1999 ) with a centralized critic and decentralized actor. Despite strong empirical results in multi-agent reinforcement learning (MARL), CTDE methods face major challenges: lack of formal safety and stability guarantees, high training variance, and poor generalization when the number of agents change between training and deployment. Safety and stability are usually encouraged heuristically via reward shaping or by layering safety certificates and fall-back controllers on top of trained policies. Existing work largely emphasizes safety filtering, with Control Barrier Functions (CBF) ( Ames et al. 2017 ) and Hamilton-Jacobi (HJ) Reachability ( Bansal et al. 2017 ) as core tools, while methods that ensure both safety and stability while retaining expressivity remain underexplored despite their importance in safety-critical applications. Recent approaches often enforce stability using simple nominal controllers. Zhang et al. 2025 consider a multi-agent navigation problem where LQR- and PID-based controllers guide agents to target states with a learned safety filter added on top. However, convergence is not guaranteed, and the approach can suffer from deadlocks when scaled to large systems that simple controllers cannot resolve. Combined with safety filtering, basic nominal policies can also induce jittery, bang-bang inputs ( Borquez et al. 2024 ) . These limitations highlight the need for control policies that are simultaneously expressive , scalable and stable , capable of reasoning over complex feedback while retaining closed-loop guarantees. Designing such policies is a challenging task, as distributed optimal control is hard even for linear quadratic (LQ) cases, where optimization problems become impractical to solve due to the exponential growth of the search space as systems scale up. For nonlinear networked systems, deriving distributed stabilizing controllers becomes substantially more difficult, which motivates the need for neural policies to capture system complexity. However, direct neural parameterizations generally lack stability guarantees and can destabilize the system unless carefully tuned ( Fazel et al. 2018 ) . Several works aim to address this problem by incorporating stability into the design and training of neural policies. Lale et al. (2022) learn stabilizing policies using a Lyapunov constraint derived from Krasovskii’s method. An adjacent line of work ( Roberts et al. 2011 ; Furieri et al. 2022a ; Furieri et al. 2024 ) generalizes Youla and system-level parameterizations to nonlinear systems, characterizing all stabilizing policies through the lens of disturbance feedback. These methods, however, are either limited by restrictive assumptions on system and controller structures, or rely exclusively on disturbance feedback without exploiting state and graph-structured information, limiting their expressivity and applicability. More recently, Furieri et al. 2025 extended these policies to include explicit state feedback, enlarging the class of learnable controllers and making them compatible with RL frameworks. However, the method focuses on fully centralized setups, limiting its scalability and ability to transfer to systems of varying sizes. In parallel, distributed neural control has been studied for structured systems. Furieri et al. 2022b learn large-scale stabilizing distributed controllers for port-Hamiltonian networks, while Saccani et al. 2024 enforce stability during training of neural controllers in networked settings. These works demonstrate that stability-constrained neural architectures can be deployed in distributed control, but tailored assumptions on system structure limit their general applicability, and they do not address scalability in the sense of training on small systems and transferring to much larger ones with modified topologies. Graph Neural Networks (GNNs) naturally address some of these challenges by performing layer-wise local aggregations over a graph. Gama and Sojoudi 2022 use GNNs as deterministic controllers for LQR problems and show that GNN-based controllers are permutation equivariant, enabling training on small networks and generalization to larger ones. However, their stability and robustness guarantees are established only for linear systems and small GNN parameters, which limits expressivity and risks destabilization if parameters exceed certain thresholds during training. Despite the many advances in distributed control, stability-constrained RL and MARL, a unifying framework that simultaneously offers (1) distributed computation with guaranteed closed-loop stability, (2) robustness to topology changes with transfer from small to large systems, and (3) state-of-the-art performance remains a desideratum. 1.2 Contributions We make the following contributions: • Distributed and scalable GNN-based policies: We present a Youla-like distributed policy parametrization that guarantees closed-loop stability while being realized by GNNs, enabling training on small networked systems and transferring to larger ones with different topologies during deployment. • Integration with PPO: We provide a recipe for making our parameterization compatible with Proximal Policy Optimization ( Schulman et al. 2017 ) , a popular RL algorithm that has enabled training stochastic policies achieving state-of-the-art performance. This allows our method to enjoy the benefits of PPO and directly plug into existing RL frameworks. • Empirical validation of benefits: On a multi-agent navigation task, we show that our distributed GNN policies trained on a few agents generalize to larger systems with unseen network topologies, improve performance over a state-of-the-art MARL baseline, and exhibit more stable and sample efficient training. Notation We denote by \ell^{n} the set of all sequences \mathbf{x}=(x_{0},x_{1},\ldots) where \mathbf{x}_{t}\in\mathbb{R}^{n} . For p\in\mathbb{N} , we write \mathbf{x}\in\ell_{p}^{n} if \|\mathbf{x}\|_{p}=\left(\sum_{t=0}^{\infty}|\mathbf{x}_{t}|^{p}\right)^{1/p}<\infty , where |\cdot| denotes any vector norm. When clear, we omit the superscript from \ell^{n} (resp. \ell_{p}^{n} ) and write \ell (resp. \ell_{p} ). For \mathbf{x}\in\ell_{p} , we denote its truncation by \mathbf{x}_{j:i}=(x_{i},x_{i+1},\dots,x_{j}) . An operator \mathbf{A}:\ell^{n}_{p}\to\ell^{m}_{p} is causal if \mathbf{A}(\mathbf{x})=\big(A_{0}(x_{0}),A_{1}(x_{1:0}),\dots,A_{t}(x_{t:0}),\dots\big) , and strictly causal if A_{t}(x_{t:0})=A_{t}(0,x_{t-1:0}) . We also write A_{j:i}(x_{j:0})=\big(A_{i}(x_{i:0}),\dots,A_{j}(x_{j:0})\big) . An operator is \ell_{p} -stable if it is causal and \mathbf{A}(\mathbf{a})\in\ell_{p}^{m} for all \mathbf{a}\in\ell_{p}^{n} , and we denote this as \mathbf{A}\in\mathcal{L}_{p} . Finally, an \mathcal{L}_{p} operator \mathbf{A}:\mathbf{w}\mapsto\mathbf{u} has finite \mathcal{L}_{p} -gain \gamma(\mathbf{A})>0 if \|\mathbf{u}\|_{p}\leq\gamma(\mathbf{A})\,\|\mathbf{w}\|_{p} for all \mathbf{w}\in\ell_{p}^{n} .

Read the complete text