Agent-GSPO: Communication-Efficient Multi-Agent Systems via Group Sequence Policy Optimization

Yijia Fan; Jusheng Zhang; Jing Yang; Keze Wang · 2025

Read Agent-GSPO: Communication-Efficient Multi-Agent Systems via Group Sequence Policy Optimization by Yijia Fan; Jusheng Zhang; Jing Yang; Keze Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-22477.txt.

Introduction

Yijia Fan and three co-authors tackle the high message cost of multi agent systems where every agent talks to every other. Their framework, Agent-GSPO, trains agents to save tokens by using sequence level reinforcement learning. The reward directly punishes wordiness while still rewarding correct answers, so agents learn when speaking helps and when silence is smarter.

Across seven reasoning benchmarks, the trained systems set new best scores while using only a fraction of the tokens of rival methods. Habits such as strategic silence emerged on their own during training. The result is a practical plan for agent teams that stay affordable at scale.

Catalog record

CREATORYijia Fan; Jusheng Zhang; Jing Yang; Keze Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE16 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-22477

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Agent-GSPO: Communication-Efficient Multi-Agent Systems via Group Sequence Policy Optimization Yijia Fan; Jusheng Zhang; Jing Yang; Keze Wang Abstract To combat the prohibitive communication costs of “free-for-all” multi-agent systems (MAS), we introduce Agent-GSPO , a framework that directly optimizes for token economy using sequence-level reinforcement learning. Agent-GSPO leverages the stable and memory-efficient Group Sequence Policy Optimization (GSPO) algorithm to train agents on a communication-aware reward that explicitly penalizes verbosity. Across seven reasoning benchmarks, Agent-GSPO not only achieves new state-of-the-art performance but does so with a fraction of the token consumption of existing methods. By fostering emergent strategies like “strategic silence,” our approach provides a practical blueprint for developing scalable and economically viable multi-agent systems. † † address: Sun Yat-sen University 1 Introduction The research frontier for Large Language Models (LLMs) [ 1 , 2 ] is rapidly shifting from single-agent solvers to complex Multi-Agent Systems (MAS) [ 3 , 2 , 4 ] . This paradigm shift holds the promise of tackling problems far beyond the reach of any individual model. This collaborative approach has demonstrated significant potential in domains like complex reasoning and software development, where agents can pool distributed knowledge and refine solutions through iterative dialogue. However, this collaborative promise is frequently undermined by a critical bottleneck: communication inefficiency. Many advancements [ 5 , 6 ] rely on “free-for-all” communication protocols, permitting agents to broadcast information with little to no cost. This architectural choice often leads to an exponential increase in token consumption and a low signal-to-noise ratio, inundating agents with verbose, low-value exchanges. Existing methods [ 7 , 8 , 9 ] , despite their performance gains, often overlook this severe resource cost, resulting in prohibitive operational overhead that hinders practical, large-scale deployment. We argue that the core issue is not a lack of communication, but the absence of a principled mechanism to enforce communicative discipline and resource rationality. To address this challenge, we turn to sequence-level reinforcement learning (RL), a paradigm naturally suited for optimizing the generation of entire messages. Specifically, we leverage Group Sequence Policy Optimization (GSPO) [ 10 ] , a recent advancement that stabilizes policy updates by operating on groups of entire response sequences rather than individual tokens. Unlike token-level objectives common in Proximal Policy Optimization (PPO) [ 11 ] applications, GSPO’s sequence-level clipping mechanism is inherently more stable for optimizing long, variable-length action sequences. This makes it an ideal framework for multi-agent dialogue, where an agent’s “action” is a complete utterance. Furthermore, by directly optimizing the policy on sequence-level rewards, GSPO avoids the need for an auxiliary reward or value model for every token, leading to greater memory efficiency during training. Building on this foundation, we introduce Agent-GSPO , a novel framework designed to cultivate communication efficiency in LLM-based MAS. Agent-GSPO frames the generation of a message as a sequential action and directly optimizes a communication-aware reward function. This reward signal explicitly incentivizes task success while penalizing communication overhead, such as token count and conversational turns. By applying the GSPO objective, our framework encourages agents to learn a sophisticated trade-off: to “speak less but more precisely.” This process intrinsically prunes low-value chatter and fosters the emergent skill of “strategic silence,” where agents learn to withhold information that is redundant or unlikely to contribute to the team’s success. Extensive experiments demonstrate that Agent-GSPO achieves new state-of-the-art performance across several challenging reasoning benchmarks. Crucially, this is accomplished with remarkable efficiency, consuming only a fraction of the communication tokens used by current state-of-the-art methods. Our analysis reveals that Agent-GSPO effectively learns to adapt its communication strategies, shifting from verbosity to conciseness based on the implicit value of its messages, thereby establishing a more practical and scalable framework for multi-agent collaboration. Figure 1: Overview of the Agent-GSPO training pipeline. The Actor Network ( \pi_{\theta_{\text{old}}} ) samples a group of candidate responses. Each response receives a communication-aware reward ( r ), which balances task success ( r_{\text{task}} ) against a token-length penalty. The GSPO module processes these sequence-level rewards to update the Actor’s policy ( \pi_{\theta} ), training the agent to favor concise and effective communication. 2 Related Work Communication in Multi-Agent Systems. The challenge of efficient communication in Multi-Agent Systems (MAS) has evolved significantly with the advent of LLMs. Early research focused on structured protocols like Agent Communication Languages (ACLs) and negotiation frameworks like the Contract Net Protocol to ensure clarity and manage overhead. More recently, the focus has shifted to learning emergent communication strategies. However, the “free-for-all” nature of LLM-based dialogue often leads to prohibitive token costs. To address this, current methods have explored various pruning strategies, such as gating mechanisms that decide when an agent should speak, or topological pruning that removes communication links, as seen in AgentPrune-R [ 12 , 13 ] . Our work diverges from these approaches by not relying on external pruning or auction mechanisms. Instead, we formulate the problem as a direct policy optimization task, where agents intrinsically learn to be concise by optimizing a communication-aware reward signal. Reinforcement Learning for LLMs. Reinforcement Learning (RL) has become a cornerstone for aligning LLMs with desired behaviors [ 14 ] , with Proximal Policy Optimization (PPO) being the de-facto standard for both post-training alignment and tool use. However, PPO’s token-level objective can suffer from high variance and instability when optimizing long, sequential actions like natural language messages. Furthermore, its reliance on a separate critic or value network adds significant memory and computational overhead. To overcome these limitations, recent work has introduced sequence-level policy optimization methods. Group Sequence Policy Optimization (GSPO), the algorithm we employ, stabilizes training by computing a clipped importance sampling ratio over entire sequences and uses group-wise reward normalization to estimate advantages directly, obviating the need for a critic network. 3 Agent-GSPO 3.1 Communication-Aware Reward Function Given an input query x , the policy \pi_{\theta} generates a response sequence y=(y_{1},\dots,y_{|y|}) . We define a composite reward function that balances task accuracy with communication efficiency: \begin{split}r(x,y)=r_{\text{task}}(x,y)&-\lambda_{\text{tok}}\cdot\text{tokens}(y)\\ &-\lambda_{\text{turn}}\cdot\text{turns}(y)-\lambda_{\text{rep}}\cdot\text{repetition}(y),\end{split} (1) where r_{\text{task}} measures task-specific performance (e.g., exact match, pass@1). The remaining terms penalize token usage, conversational turns, and content repetition. The coefficient \lambda_{\text{tok}} is the primary hyperparameter controlling token efficiency. 3.2 Sequence-Level GSPO Objective To ensure optimization stability, GSPO operates at the sequence level. For each input x , we sample a group of G candidate responses \{y_{i}\}_{i=1}^{G} from a frozen, older policy \pi_{\theta_{\text{old}}} . We then compute group-wise normalized advantages: \hat{A}_{i}=\frac{r(x,y_{i})-\mu_{r}}{\sigma_{r}+\epsilon}, (2) where \mu_{r}=\frac{1}{G}\sum_{j=1}^{G}r(x,y_{j}) is the mean reward and \sigma_{r} is the standard deviation of rewards within the group, with \epsilon as a small constant for numerical stability. To mitigate variance arising from sequence length disparities, the importance sampling ratio is normalized by the sequence length: \begin{split}s_{i}(\theta)&=\Bigg(\frac{\pi_{\theta}(y_{i}|x)}{\pi_{\theta_{\text{old}}}(y_{i}|x)}\Bigg)^{\frac{1}{|y_{i}|}}\\ &=\exp\!\Bigg(\frac{1}{|y_{i}|}\sum_{t=1}^{|y_{i}|}\log\frac{\pi_{\theta}(y_{i,t}|x,y_{i,<t})}{\pi_{\theta_{\text{old}}}(y_{i,t}|x,y_{i,<t})}\Bigg).\end{split} (3) The final GSPO objective, analogous to PPO, incorporates a clipping mechanism at the sequence level: \displaystyle J_{\text{GSPO}}(\theta)=\mathbb{E}_{x,\{y_{i}\}\sim\pi_{\theta_{\text{old}}}} \displaystyle\Bigg[\frac{1}{G}\sum_{i=1}^{G}\min\Big(s_{i}(\theta)\hat{A}_{i}, (4) \displaystyle\text{clip}(s_{i}(\theta),1-\varepsilon,1+\varepsilon)\hat{A}_{i}\Big)\Bigg]. This sequence-level clipping is crucial for stabilizing training; it prevents the large-variance updates that can arise from token-level probability ratios, especially for long sequences. 3.3 Dual Budget Constraint To explicitly enforce an expected communication budget, B , we can treat \lambda_{\text{tok}} as a learnable dual variable, updated via projected gradient ascent: \lambda_{\text{tok}}\leftarrow\max\big(0,\lambda_{\text{tok}}+\eta\cdot(\overline{\text{tokens}}-B)\big), (5) where \overline{\text{tokens}} is the batch-average token consumption and \eta is a learning rate. This mechanism dynamically adjusts the penalty to

Read the complete text