Hierarchical Reinforcement Learning with Low-Level MPC for Multi-Agent Control

Max Studt, Georg Schildbach · 2025

Read Hierarchical Reinforcement Learning with Low-Level MPC for Multi-Agent Control by Max Studt, Georg Schildbach (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-15799.txt.

Introduction

Max Studt and Georg Schildbach pair learning with model based control for teams of agents. Safe, coordinated acts in tight settings are hard for learning alone. Pure learning can be slow to train and shaky in use. Model based control is firm but leans on fixed targets and travels poorly.

Their frame stacks the two: a learning policy picks broad targets from marked regions of interest. A predictive controller then turns each target into safe, feasible motion. The split suits multi agent systems, where coordination matters most. On a predator and prey test, the frame beats pure learning and shield based rivals. It wins on reward, safety, and steadiness.

Catalog record

CREATORMax Studt, Georg Schildbach
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-15799

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Hierarchical Reinforcement Learning with Low-Level MPC for Multi-Agent Control Max Studt, Georg Schildbach Abstract Achieving safe and coordinated behavior in dynamic, constraint-rich environments remains a major challenge for learning-based control. Pure end-to-end learning often suffers from poor sample efficiency and limited reliability, while model-based methods depend on predefined references and struggle to generalize. We propose a hierarchical framework that combines tactical decision-making via reinforcement learning (RL) with low-level execution through Model Predictive Control (MPC). For the case of multi-agent systems this means that high-level policies select abstract targets from structured regions of interest (ROIs), while MPC ensures dynamically feasible and safe motion. Tested on a predator–prey benchmark, our approach outperforms end-to-end and shielding-based RL baselines in terms of reward, safety, and consistency, underscoring the benefits of combining structured learning with model-based control. I Introduction Reinforcement learning (RL) has emerged as a powerful tool for solving complex decision-making problems, offering the ability to learn control policies from interaction with an environment. However, deploying RL in safety-critical applications remains a significant challenge. Standard end-to-end learning approaches often lack formal safety guarantees and suffer from poor sample efficiency, especially when required to satisfy hard physical constraints such as collision avoidance, actuator saturation, or operational safety limits. Learning both task behavior and constraint satisfaction simultaneously can lead to unstable or slow convergence and, in real-world systems, may even render training infeasible [ 1 , 2 ] . At the same time, purely model-based control methods, such as MPC, while effective in enforcing constraints and guaranteeing safe execution, typically rely on externally defined reference trajectories. In static or well-structured settings, such references can be manually designed. However, in complex, dynamic, or partially known environments, designing meaningful reference trajectories a priori is challenging. As a result, model-based approaches struggle to adapt to changing goals, task objectives, or environment layouts without external guidance. This limits their autonomy and applicability in many real-world scenarios. These complementary limitations make a hybrid approach particularly attractive: RL can provide high-level, adaptive intent in the form of target selection or task decisions, while model-based control can ensure safety, constraint satisfaction, and physical feasibility during execution. Consider, for example, a fleet of autonomous delivery drones operating in a changing urban environment. While MPC can ensure collision-free trajectories and respect dynamic constraints such as battery limits or no-fly zones, it cannot decide where to go or how to prioritize deliveries without predefined references. Conversely, an RL-based policy can flexibly decide on routing and target selection based on learned objectives, but may fail to produce safe low-level motions if trained in an end-to-end fashion. The integration of both paradigms in a hierarchical structure allows strategic reasoning and adaptive behavior on top of a reliable, constraint-respecting execution layer, offering a promising path toward safe and generalizable learning-based control. An especially challenging application domain for RL arises in multi-agent systems, where agents must coordinate under coupled dynamics and safety constraints, often in complex environments. Multi-agent reinforcement learning (MARL) in such settings exacerbates the sample inefficiency and safety limitations of single-agent RL due to the increased dimensionality, non-stationarity, and the need for implicit coordination [ 3 ] . Moreover, safety violations by individual agents can propagate through the system, further increasing the difficulty of ensuring constraint satisfaction during both training and execution. This paper addresses precisely these challenges. We propose a hierarchical MARL framework that decouples high-level decision-making from low-level control, combining the flexibility of learned coordination strategies with the safety and reliability of model-based execution. Through structured target selection and decentralized constraint handling, our method aims to enable safe, efficient, and robust behavior in dynamic, multi-agent systems. II Related Work and Contribution The idea of combining RL with model-based control has been explored from multiple angles, particularly to mitigate safety risks and to improve the sample efficiency in complex decision-making tasks. Prior work can broadly be divided into single-agent and multi-agent approaches, each with different assumptions and integration strategies. In the single-agent setting, several methods aim to tune robust MPC schemes within a constrained RL framework, for example by learning cost weights and constraint tightenings from data, while keeping the overall structure hand-crafted and modular [ 4 , 5 ] . Other approaches incorporate uncertainty estimates through Gaussian Process models and leverage MPC for safe exploration under probabilistic guarantees [ 6 ] . Formal safety guarantees during policy updates have also been obtained using Lyapunov-based criteria [ 7 ] . Additional methods introduce safety filters or shielding mechanisms that override unsafe actions at execution time [ 8 , 9 ] . A broader taxonomy and classification of these approaches is available in a recent survey work [ 10 ] . Extending such ideas to multi-agent systems presents further challenges due to inter-agent coupling and the curse of dimensionality. One line of work uses learned policies to warm-start MPC optimizers during deployment, thereby enhancing safety without changing the learned decision structure [ 11 ] . Other methods employ distributed MPC as a safety shield that filters actions in cooperative navigation tasks and penalizes unsafe deviations [ 12 ] . Hierarchical architectures have also been proposed, in which a robust multi-agent policy handles high-level coordination under observation uncertainty, while a low-level MPC controller ensures safe execution through control barrier functions [ 13 ] . Our Contribution In contrast to previous approaches, we propose a new RL framework (shown in Fig. 1 ) that does not treat MPC as a passive safety layer. Instead, the two are combined in a hierarchical order, during learning and execution, attempting to decouple the strategic decision making via RL from the low-level system control via MPC. The key contributions are as follows: ( i ) A hierarchical RL-MPC control framework that uses task-specific prior knowledge to structure each agent’s action space into goal-relevant regions of interest (ROIs) around likely coordination targets. Here, the policy outputs continuous target points from these ROIs, which a decentralized MPC then tracks under constraints. This improves sample efficiency and stability, especially in sparse-reward settings. ( ii ) Application of this concept to cooperative MARL, demonstrated on a predator-prey benchmark with discrete-continuous hybrid actions, nonlinear polar coordinate transformations and minimal reward shaping. ( iii ) Experimental evaluation against end-to-end learning and shielding-based MPC baselines, demonstrating superior performance in reward, safety, and behavioral consistency. Fig. 1: Overview of our hierarchical decision-making and control framework. The high-level MARL policy selects discrete targets from a structured ROI, informed by task-specific prior knowledge. A low-level MPC controller then tracks the selected target under dynamics and safety constraints. III Background and Preliminaries In this section, we lay out the formal foundations for a hierarchical control architecture that couples learned decision-making with model-based execution. We start by reviewing cooperative MARL under the centralized-training, decentralized-execution (CTDE) paradigm, which can be cast as a decentralized partially observable Markov decision process (Dec-POMDP) [ 14 ] . We then introduce a generic continuous-state, continuous-action agent model and formulate the decentralized MPC problem used for low-level target tracking. III-A Cooperative Multi-Agent Reinforcement Learning We model cooperative decision making as a decentralized partially observable Markov decision process (Dec-POMDP) for N agents [ 14 ] . Formally, a Dec-POMDP can be expressed as a tupel: M=\bigl(\,\mathcal{S},\,\mathcal{A},\,\mathcal{O},\,P,\,\Omega,\,R,\,n,\,\gamma\bigr), where n is the number of agents

Read the complete text