Scaling Multiagent Systems with Process Rewards

Ed Li, Junyu Ren, Cat Yan · 2026

Read Scaling Multiagent Systems with Process Rewards by Ed Li, Junyu Ren, Cat Yan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-23228.txt.

Introduction

Ed Li, Junyu Ren, and Cat Yan fine-tune multi-agent systems with per-action process rewards. Multi-agent setups promise strong results through specialization. Training several agents at once faces two hurdles: assigning credit across agents, and the cost of many multi-agent rollouts.

Their MAPPA method uses AI feedback to reward each action, not just final outcomes. Finer rewards attack the credit problem and stretch each expensive rollout further. Process feedback could make agent teams trainable at realistic budgets. Per-action rewards from AI feedback attack both named challenges at once: credit assignment and sample efficiency.

Catalog record

CREATOREd Li, Junyu Ren, Cat Yan
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE79 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-23228

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Scaling Multiagent Systems with Process Rewards Ed Li, Junyu Ren, Cat Yan Abstract While multiagent systems have shown promise for tackling complex tasks via specialization, finetuning multiple agents simultaneously faces two key challenges: (1) credit assignment across agents, and (2) sample efficiency of expensive multiagent rollouts. In this work, we propose finetuning multiagent systems with per-action process rewards from AI feedback (MAPPA) to address both. Through assigning credit to individual agent actions rather than only at task completion, MAPPA enables fine-grained supervision without ground truth labels while extracting maximal training signal from each rollout. We demonstrate our approach on competition math problems and tool-augmented data analysis tasks. On unseen math problems, MAPPA achieves +5.0–17.5pp on AIME and +7.8–17.2pp on AMC. For data analysis tasks, our method improves success rate by +16.7pp while quality metrics improve by up to 47%, validating that per-action supervision can lead to improvements across different multiagent systems on various domains. By addressing these challenges, our work takes a first step toward scaling multiagent systems for complex, long-horizon tasks with minimal human supervision. † † affiliationnotice: Equal contribution Code Blog 1 Introduction Figure 1 : Multiagent architectures with separate weights enable specialization through end-to-end training, sidestepping catastrophic forgetting that limits single-model scaling just like mixture-of-experts (MoE) architecture. Large reasoning models trained with reinforcement learning have emerged as the dominant paradigm for state-of-the-art AI systems ( 15 ; 2 ; 20 ) . Augmented with external tools such as code interpreters and search engines ( 18 ) , these models achieve remarkable performance on mathematical reasoning ( 19 ) , coding ( 5 ) , and scientific problems ( 14 ) . Building on these capabilities, multiagent systems—where multiple language model-based agents interact to solve complex tasks—offer an intuitive path forward as specializations and the diversity of perspectives can tackle problems better than any single agent. ( 24 ; 3 ) . The bulk of existing multiagent work implements specialization through prompt engineering: assigning different system prompts, personas, or tool access to agents within the system. While effective for orchestrating pre-trained models, the true strength of multiagent systems lies in the ability to modify the weights of individual agents, enabling specialization without interference. Recent evidence suggests such specialization naturally emerges through reinforcement learning: reasoning models trained solely for accuracy spontaneously develop diverse internal personas and expertise ( 7 ) . By separating weights across agents, improving one agent’s capabilities cannot degrade another’s performance, allowing scaling without catastrophic forgetting (Figure 1 ). Recent work has begun exploring this direction of fine-tuning agents in multiagent systems simultaneously ( 21 ; 11 ; 16 ; 13 ) . However, two significant challenges remain: 1. Credit assignment : How do we attribute the performance of the overall system to actions taken by individual agents? 2. Computational cost : Rollouts of the entire multiagent system can take minutes or even hours, yet yield only a single outcome reward signal. We address these challenges through finetuning M ulti A gent system with P er-action P rocess rewards from A I feedback ( MAPPA ). Rather than assigning a single sparse reward at trajectory’s end according to fixed instructions in LLM-as-a-Judge, we leverage language models as coaches to assess the quality of each agent’s action given its role, inputs, and environment feedback such as tool execution results. This yields dense learning signal throughout the trajectory, enabling effective training even when tasks fail entirely. Crucially, the coach performs implicit credit assignment: when a downstream agent encounters a file-not-found error for an artifact that an upstream agent should have produced, the coach assigns low scores to the upstream agent’s faulty actions rather than penalizing the downstream agent. Furthermore, the number of training signals now scales with the number of actions taken, dramatically improving sample efficiency over outcome-based methods. We validate this approach on two separate domains: mathematical reasoning with code execution (MathChat) and end-to-end data science pipelines (DSBench ( 6 ) ). On MathChat, our three-agent system achieves improvements of +5.0–17.5pp on AIME and +7.8–17.2pp on AMC across two model configurations. On DSBench, where multiagent pipelines must engineer features, train models, and generate predictions, training improves success rate by +16.7pp while quality metrics improve by up to 47% . We present a general framework for training multiagent systems on complex, tool-augmented tasks via coach-guided reinforcement learning. The key components—agent topology, reward structure, and training pipeline—are domain-agnostic and can be configured to support diverse applications. Our source code is available at https://github.com/ltjed/multiagent-coaching . Overall, our results suggest that scaling the number of specialized agents represents a promising new dimension for improving performance on complex, long-horizon tasks . 2 Methodology 2.1 Problem Setting MAPPA finetunes tool-augmented multiagent systems where LLM agents collaborate to solve tasks. Each agent is initialized from a pretrained LLM with independent policy parameters trained separately, enabling specialization. All agents live in a terminal environment and execute in a pre-defined topology (e.g., sequential pipelines, debate, mixture-of-agents). Within each agent’s turn, it generates one or more actions that may include tool calls. When tool calls are generated as part of an agent’s action, they are processed and executed in a sandboxed environment, the agent will then receive environment feedback (e.g., stdout, stderr, error messages). Depending on the task, the agent may also need to call tools to read, write files or other artifacts in the terminal environment. 2.2 Process Rewards from AI Feedback Multi-turn tool use with inter-agent dependencies creates challenging credit assignment—which agent’s actions led to success or failure? Unlike existing RL approaches that rely on sparse outcome rewards (success/failure for entire trajectories), we leverage a coach LLM to perform per-action credit assignment, providing dense process-quality rewards on a 0--10 scale. 1 1 1 We found the 0–10 scale empirically outperforms continuous 0–1 scores, likely due to the prevalence of 0–10 rating samples in the pre-training corpus, i.e., a “ 7 ” or a “ 4 ,” carries more semantic meaning than decimal values like 0.305 . We use the term “coach” rather than “judge” to emphasize that while LLM-as-a-judge evaluates final outputs against fixed rubrics, a coach provides context-aware feedback on each intermediate action to guide improvement throughout the trajectory. The coach evaluates each action holistically based on: (1) the agent’s role and responsibilities, (2) the input context the agent observed, (3) the agent’s action, and (4) tool execution results (stdout, stderr, error messages) when the output contains tool calls. This enables the coach to identify the responsible agent when failures occur: if a downstream agent encounters a missing file error, the coach assigns low scores to the upstream agent that should have produced it, not the downstream agent that correctly reported the issue. See Appendix D for example coach evaluations demonstrating this context-aware feedback. Crucially, MAPPA can work with or without ground truth/verifier that most other alternatives RL approaches such as RLVR ( 12 ; 2 ) require. When ground truth is available (e.g., ROC-AUC, F1, RMSE for DSBench), the coach incorporates it to inform process scores via intelligently synthesizing multi-dimensional metrics into a single reward; When ground truth is unavailable, the coach simply evaluates how sensible it is for the agent to take that action given its context to the best of the

Read the complete text