Self-Compression of Chain-of-Thought via Multi-Agent Reinforcement Learning

Yiqun Chen, Jinyuan Feng, Wei Yang, Meizhi Zhong, Zhengliang Shi, Rui Li, Xiaochi Wei, Yan Gao, Yi Wu, Yao Hu, Zhiqiang Pu, Jiaxin Mao · 2026

Read Self-Compression of Chain-of-Thought via Multi-Agent Reinforcement Learning by Yiqun Chen, Jinyuan Feng, Wei Yang, Meizhi Zhong, Zhengliang Shi, Rui Li, Xiaochi Wei, Yan Gao, Yi Wu, Yao Hu, Zhiqiang Pu, Jiaxin Mao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-21919.txt.

Introduction

Yiqun Chen and co-authors try to make reasoning models briefer without making them dumber. Large reasoning models write long chains of thought, and the redundant parts slow responses and bottleneck deployment. Earlier reinforcement learning fixes coupled a length penalty with rewards for correct outcomes.

That simple weighting struggles to balance brevity against accuracy, since forcing short answers can damage critical reasoning steps. The team's 2026 preprint instead compresses chains of thought through multi-agent reinforcement learning, where several agents shape the training signal. If models can learn to skip only the waste, users get faster answers at lower cost. And long reasoning stays affordable for everyday use rather than special occasions only.

Catalog record

CREATORYiqun Chen, Jinyuan Feng, Wei Yang, Meizhi Zhong, Zhengliang Shi, Rui Li, Xiaochi Wei, Yan Gao, Yi Wu, Yao Hu, Zhiqiang Pu, Jiaxin Mao
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-21919

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Self-Compression of Chain-of-Thought via Multi-Agent Reinforcement Learning Yiqun Chen, Jinyuan Feng, Wei Yang, Meizhi Zhong, Zhengliang Shi, Rui Li, Xiaochi Wei, Yan Gao, Yi Wu, Yao Hu, Zhiqiang Pu, Jiaxin Mao Abstract The inference overhead induced by redundant reasoning undermines the interactive experience and severely bottlenecks the deployment of Large Reasoning Models. Existing reinforcement learning (RL)-based solutions tackle this problem by coupling a length penalty with outcome-based rewards. This simplistic reward weighting struggles to reconcile brevity with accuracy, as enforcing brevity may compromise critical reasoning logic. In this work, we address this limitation by proposing a multi-agent RL framework that selectively penalizes redundant chunks, while preserving essential reasoning logic. Our framework, Self-Compression via MARL (SCMA), instantiates redundancy detection and evaluation through two specialized agents: a Segmentation Agent for decomposing the reasoning process into logical chunks, and a Scoring Agent for quantifying the significance of each chunk. The Segmentation and Scoring agents collaboratively define an importance-weighted length penalty during training, incentivizing a Reasoning Agent to prioritize essential logic without introducing inference overhead during deployment. Empirical evaluations across model scales demonstrate that SCMA reduces response length by 11.1% to 39.0% while boosting accuracy by 4.33% to 10.02%. Furthermore, ablation studies and qualitative analysis validate that the synergistic optimization within the MARL framework fosters emergent behaviors, yielding more powerful LRMs compared to vanilla RL paradigms. † † affiliationnotice: Equal contribution 1 Introduction Recent advancements in Large Reasoning Models (LRMs) ( 4 ; 13 ) have demonstrated remarkable capabilities often characterized as ’thinking.’ Specifically, LRMs employ detailed Chain-of-Thought (CoT) sequences to facilitate complex problem-solving through self-reflection, backtracking, and verification. The deep thinking capacities are predominantly elicited through Reinforcement Learning (RL) ( 32 ; 30 ) , but this paradigm simultaneously introduces significant efficiency bottlenecks. This issue arises because existing RL optimization is primarily driven by sparse, outcome-based binary rewards. Lacking fine-grained guidance on the intermediate reasoning process, models are prone to ’over-thinking’ ( 5 ; 8 ) , where they generate prolonged paths cluttered with non-essential steps or repetitive verifications to maximize reward certainty. This redundancy imposes substantial inference latency for deployment, necessitating methods that guide models to ’think less’ by synthesizing concise, robust, and accurate reasoning chains. Current methods, which we discuss extensively in Appendix A , primarily address these limitations by incorporating length-based penalties to curb reasoning redundancy ( 6 ; 37 ; 10 ; 45 ; 16 ) . However, such solutions suffer from misaligned credit assignment: they indiscriminately penalize the total sequence length rather than intrinsic redundancy of intermediate reasoning steps ( 44 ) . This undifferentiated strategy poses a significant risk: models may inadvertently sacrifice critical reasoning steps essential for success just to satisfy length constraints, thereby compromising final accuracy. Therefore, a pivotal challenge remains: how to distinguish and preserve high-value reasoning steps at a fine-grained level, while precisely eliminating redundancy and meaningless repetition. The key to overcoming this bottleneck lies in two aspects: first, decomposing the continuous reasoning process into independent logical chunks; and second, accurately quantifying the substantive contribution of each chunk to the final derivation. Regrettably, existing single-agent RL paradigms struggle to achieve this, as they inherently lack the mechanisms for fine-grained structural modeling and decoupled value estimation. This necessitates a paradigm shift towards Multi-Agent Reinforcement Learning (MARL), where distinct agents can collaborate to achieve simultaneous optimization of reasoning generation and fine-grained process control. To bridge this gap, we propose Self-Compression via MARL (SCMA), an end-to-end multi-agent framework designed to achieve fine-grained compression of the thinking process. SCMA innovatively reformulates the compression task as a collaborative game, coordinating three specialized roles: the Reasoning Agent explore the solution space to generate reasoning paths; the Segmentation Agent structurally parses the reasoning paths into discrete logical chunks; and the Scoring Agent quantifies the substantive contribution of each chunk to the final derivation. In this setup, standard length penalties are replaced by an importance-weighted length penalty derived collaboratively by the Segmentation and Scoring agents. This feedback incentivizes the Reasoning Agent to selectively discard redundancy while preserving essential logic. Crucially, since the Reasoning Agent’s ability to balance correctness with conciseness hinges entirely on the fidelity of this dynamic feedback, the three roles are intrinsically coupled. Consequently, SCMA employs a shared reward to drive the co-evolution of all agents by multi-agent group relative policy optimization (GRPO), ensuring their objectives remain strictly aligned towards generating high-quality, efficient reasoning chains. In summary, our contributions can be concluded as follows: • Framework Proposal: We propose Self-Compression via MARL (SCMA), a framework that reformulates the Chain-of-Thought compression task by transitioning from the conventional single-agent RL to a MARL training paradigm, enabling fine-grained compression with no test-time overhead. • Mechanism Design: We design a collaborative optimization mechanism governed by a multi-agent GRPO objective, incorporating an importance-weighted length penalty as a unified reward signal to drive the co-evolution of all agents. • Experimental Verification: We empirically demonstrate that SCMA reduces reasoning length by 11.1%–39.0% while simultaneously improving accuracy by 4.33%–10.02% . Ablation studies further validate the superiority of MARL joint optimization, which significantly outperforms decoupled baselines relying on single-agent architectures. Figure 1: Overview of SCMA Compared to general RL with length penalty. (Left) The general RL calculates rewards by penalizing the length of the thinking process directly. (Right) The SCMA employs an importance-weighted length penalty within a multi-agent system. 2 Preliminary 2.1 General RL with Length Penalty In post-training based on RL, LLM is modeled as a policy \pi_{\theta} that autoregressively generates a response y given a prompt x . To compress the reasoning process, existing solutions typically incorporate a length penalty, formulating the reward function as R(y|x)=R_{\text{acc}}(y|x)-\alpha f(|y|) . R_{\text{acc}} denotes outcome-based rewards for correctness, \alpha is the penalty strength coefficient, and f(\cdot) applies group-wise normalization to the response length |y| , typically taking a linear or exponential form ( 6 ; 47 ) . Consequently, the RL objective is to optimize \theta for maximizing the expected reward: \mathcal{J}(\theta)=\mathbb{E}_{x\sim\mathcal{D},y\sim\pi_{\theta}(\cdot|x)}\left[R_{\text{acc}}(y|x)-\alpha f(|y|)\right] (1) 2.2 Group Relative Policy Optimization (GRPO) GRPO ( 32 ) streamlines the alignment of large language models by eliminating the critic network inherent to PPO ( 31 ) , thereby reducing memory overhead and avoiding value approximation instability. Specifically, GRPO estimates the advantages by sampling a group of responses \{y_{i}\}_{i=1}^{G} and normalizing the rewards within group. The advantage for each token t in the i -th response is computed as \hat{A}_{i,t}=\frac{r_{i}-\text{mean}(\mathbf{r})}{\text{std}(\mathbf{r})} , where \mathbf{r} is the reward vector for the group. The GRPO objective incorporates a clipping mechanism and a KL-divergence penalty to stabilize policy updates and prevent excessive deviation from the reference policy \pi_{ref} : \displaystyle\mathcal{J}_{\text{GRPO}}(\theta) \displaystyle=\mathbb{E}_{x\sim P(x),\{y_{i}\}_{i=1}^{G}\sim\pi_{\theta_{\text{old}}}(\cdot|x)}\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|y_{i}|}\sum_{t=1}^{|y_{i}|} (2) \displaystyle\Bigg\{\min\Bigl[r_{i,t}(\theta)\hat{A}_{i,t},\ \operatorname{clip}\bigl(r_{i,t}(\theta),1-\epsilon,1+\epsilon\bigr)\hat{A}_{i,t}\Bigr] \displaystyle-\beta D_{\text{KL}}\bigl[\pi_{\theta}\parallel\pi_{\text{ref}}\bigr]\Bigg\}, where the importance sampling ratio is defined as: r_{i,t}(\theta)=\frac{\pi_{\theta}(y_{i,t}\mid x,y_{i,<t})}{\pi_{\theta_{\text{old}}}(y_{i,t}\mid x,y_{i,<t})}. (3) Based on \{y_{i}\}_{i=1}^{G} sampled from the behavior policy \pi_{\theta_{\text{old}}} , the optimization is constrained by the clipping hyperparameter \epsilon and the KL-divergence penalty coefficient \beta . 3 Method 3.1 An Overview of SCMA As illustrated in Fig 1 , SCMA departs from the scalar penalties of general RL by instantiating three functionally distinct agents within a multi-agent system to achieve fine-grained compression. Crucially, all agents share parameters \theta from a single base LLM \pi_{\text{base}} to achieve internal reasoning compression, a process we term Self-Compression . Under this paradigm, the multi-agent

Read the complete text