MAESTRO: Multi-Agent Environment Shaping through Task and Reward Optimization
Boyuan Wu · 2025
Read MAESTRO: Multi-Agent Environment Shaping through Task and Reward Optimization by Boyuan Wu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-19253.txt.
Introduction
Boyuan Wu addresses two chores that slow cooperative multi-agent learning: writing good reward functions and building a training curriculum that avoids dead ends. Fixed rules are brittle, and putting a large language model inside the live control loop is costly. His framework, MAESTRO, moves the language model offline and uses it as a training architect instead.
One part generates varied traffic scenarios matched to current skill, while another writes reward functions as runnable Python code that adapts as tasks get harder. These guide a standard MADDPG learner without adding cost at run time. On large traffic signal control in Hangzhou, with sixteen intersections, the full system improves both performance and stability.
Catalog record
| CREATOR | Boyuan Wu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-19253
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MAESTRO: Multi-Agent Environment Shaping through Task and Reward Optimization Boyuan Wu Abstract Cooperative Multi-Agent Reinforcement Learning (MARL) is fundamentally constrained by two design bottlenecks: the manual engineering of dense reward functions and the difficulty of constructing curricula that prevent local optima in high-dimensional state spaces. Existing solutions typically rely on static, hand-crafted heuristics that struggle to generalize. While Large Language Models (LLMs) have recently been applied to decision-making, most approaches utilize the “LLM-as-Agent” paradigm—placing the model directly in the control loop—which incurs prohibitive inference latency and token costs unsuitable for real-time systems. In this work, we propose MAESTRO (Multi-Agent Environment Shaping through Task and Reward Optimization), a framework that shifts the role of the LLM from real-time agent to high-level architect . MAESTRO treats the training process as a dual-loop optimization problem where the LLM dynamically designs both the task and the solving guidance. The system integrates two generative components: (i) a semantic curriculum generator that synthesizes diverse, context-rich scenarios based on agent performance, and (ii) an automated reward synthesizer that writes executable Python reward functions to guide agents through the evolving curriculum. These components optimize the training of a standard off-policy MARL backbone (MADDPG) without requiring LLM inference during execution. We validate MAESTRO on the challenging domain of real-world Traffic Signal Control (Hangzhou network, 16 intersections), conducting a controlled ablation study comparing LLM-generated curriculum with template rewards (baseline) against our full system combining LLM curriculum and LLM-generated rewards. Our results demonstrate that using LLMs as architects for both task generation and reward shaping yields improved mean performance and risk-adjusted returns. Specifically, across four independent seeds, the full system achieves + 4.0 % +4.0\% improvement in mean performance (163.26 vs 156.93) and 2.2 × \times better risk-adjusted performance (Sharpe Ratio 1.53 vs 0.70) compared to the alternative curriculum baseline. These findings suggest that LLM-generated reward shaping with constrained generation provides measurable performance gains for production deployment in cooperative multi-agent systems. Our code is available at: https://github.com/BrianWu1010/MAESTRO.git . 1 Introduction Cooperative Multi-Agent Reinforcement Learning (MARL) holds immense promise for solving complex decentralized control problems, from warehouse logistics to sensor network coordination. However, scaling MARL to realistic, non-stationary environments is persistently hindered by two fundamental design bottlenecks: curriculum design and reward engineering . The first bottleneck, exploration, is often addressed by **Curriculum Learning (CL)**, which gradually increases task difficulty. However, constructing effective curricula requires deep domain expertise to ensure tasks are meaningfully diverse and effective [ 7 ] . The second bottleneck concerns reward specification: the efficacy of MARL is brittle with respect to reward specification. Standard dense rewards are often misaligned with global objectives [ 11 ] , while sparse global rewards fail to provide sufficient learning signal. Handcrafting reward functions that generalize across dynamic regimes remains a labor-intensive and error-prone process. Large Language Models (LLMs) offer a potential remedy by encoding rich semantic priors that can automate these design processes. However, the prevailing integration paradigm— LLM-as-Agent —typically inserts the model directly into the control loop to select actions [ 3 , 2 , 13 ] . While this yields strong zero-shot generalization, it incurs prohibitive inference latency and token costs, rendering it impractical for real-time control systems requiring sub-second decision making. Furthermore, this approach underutilizes the LLM’s capacity for high-level reasoning, relegating it to per-step tactical decisions. Our Approach: MAESTRO. In this work, we propose a paradigm shift from LLM-as-Agent to LLM-as-Architect . We introduce MAESTRO ( M ulti- A gent E nvironment S haping through T ask and R eward O ptimization), a generative meta-learning framework that uses the LLM offline to shape the training process of a conventional MARL backbone. We treat the training loop as a dual-optimization problem in which the LLM acts as a high-level designer. MAESTRO integrates two generative components: (i) a Semantic Curriculum Generator that synthesizes diverse, performance-driven task contexts, and (ii) an Automated Reward Synthesizer that produces executable Python reward functions dynamically tailored to the current curriculum stage. By distilling semantic knowledge into executable training scaffolds (tasks and rewards), MAESTRO guides a standard MADDPG policy, enabling the final agents to retain the lightweight, low-latency characteristics required for deployment while benefiting from the broad reasoning capabilities of foundation models. Figure 1: The MAESTRO framework workflow. The system operates as a dual-loop process: The LLM (Architect) designs the learning environment by generating curriculum contexts and synthesizing executable Python reward functions. The MARL agents (Learners) train within this shaped environment. Performance feedback triggers the Architect to adapt the curriculum and regenerate rewards, isolating the expensive LLM inference from the real-time execution loop. Contributions. We validate MAESTRO on the challenging domain of Urban Traffic Signal Control (TSC), a prominent benchmark for MARL coordination [ 10 , 12 ] . TSC specifically serves as a rigorous testbed for non-stationarity and coordination. Our contributions are: • A Generative Environment Shaping Framework. We formally propose MAESTRO, a method for contextual MARL that automates both task generation and reward synthesis, shifting the LLM’s role to an offline supervisor. • Controlled Ablation of Generative Components. We conduct a rigorous ablation comparing three configurations: a baseline LLM-driven curriculum with template rewards (A2), an alternative curriculum mode (A8), and the full MAESTRO system with LLM-generated reward shaping (A7). This design explicitly isolates the value of generative reward shaping from curriculum learning. • Demonstration of Performance and Efficiency. Empirical results across four independent seeds show that while curriculum learning alone improves performance, the full MAESTRO system achieves +4.0\% improvement in mean performance and superior risk-adjusted returns (Sharpe Ratio 2.2 \times higher than baselines). The system yields substantial improvements in domain metrics (20–28% delay reduction), confirming that LLM-architected training produces more effective policies than human-designed heuristics. 2 Related Work 2.1 MARL for Decentralized Control Multi-Agent Reinforcement Learning (MARL) is widely applied to cooperative control problems where agents share a global goal but make local decisions based on partial observations. TSC, a challenging instance of a Dec-POMDP, popularized the Centralized Training with Decentralized Execution (CTDE) paradigm [ 5 ] , where agents learn a joint value function during training but act independently in deployment. Early applications of CTDE to traffic, such as CoLight [ 10 ] and MPLight [ 1 ] , focused on modeling inter-agent dependencies using mechanisms like graph attention and phase-pressure signals. These approaches assume a static, manually designed reward function and a fixed environment setup. Our work maintains the stability and real-time capability of the CTDE backbone (MADDPG) but allows the reward function itself and the training environment to evolve dynamically via LLM-generated shaping signals, addressing a critical generalization gap. 2.2 Automated Curriculum and Environment Generation The difficulty of exploration in high-dimensional MARL motivates Automatic Curriculum Learning (ACL) , which autonomously sequences tasks to maximize agent learning progress [ 8 ] . Methods like POET [ 9 ] and ALP-GMM [ 8 ] demonstrate the power of generating diverse environments to force agents to acquire generalizable skills. In the context of TSC, curriculum generation has traditionally been limited to heuristic, numeric adjustments of traffic parameters, such as flow rates and phase durations [ 11 ] . These numeric methods lack the expressiveness to generate complex, semantically meaningful scenarios (e.g., “a peak-hour commuter pattern with a major bottleneck on the eastern artery”). Our approach