Multi-Agent Evolve: LLM Self-Improve through Co-evolution

Yixing Chen, Yiding Wang, Siqi Zhu, Haofei Yu, Tao Feng, Muhan Zhang, Mostofa Patwary, Jiaxuan You · 2025

Read Multi-Agent Evolve: LLM Self-Improve through Co-evolution by Yixing Chen, Yiding Wang, Siqi Zhu, Haofei Yu, Tao Feng, Muhan Zhang, Mostofa Patwary, Jiaxuan You (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-23595.txt.

Introduction

Yixing Chen and seven co-authors propose Multi-Agent Evolve, a way for language models to improve their reasoning without human-made training sets. Reinforcement learning works well for such models, but it leans on curated data and checkable rewards, which limits how far it scales. Self-play methods drop the human data yet usually need a grounded environment, like a code interpreter, to judge results.

Their design splits one model into three roles that train together: a Proposer that invents questions, a Solver that answers them, and a Judge that grades both. Reinforcement learning tunes all three as they co-evolve, across math, reasoning, and general knowledge questions. On a three-billion-parameter Qwen model, the method lifted benchmark scores by about four and a half percent on average. The authors offer it as a data-efficient path to stronger general reasoning.

Catalog record

CREATORYixing Chen, Yiding Wang, Siqi Zhu, Haofei Yu, Tao Feng, Muhan Zhang, Mostofa Patwary, Jiaxuan You
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE67 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-23595

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Multi-Agent Evolve: LLM Self-Improve through Co-evolution Yixing Chen, Yiding Wang, Siqi Zhu, Haofei Yu, Tao Feng, Muhan Zhang, Mostofa Patwary, Jiaxuan You Abstract Reinforcement Learning (RL) has demonstrated significant potential in enhancing the reasoning capabilities of large language models (LLMs). However, the success of RL for LLMs heavily relies on human-curated datasets and verifiable rewards, which limit their scalability and generality. Recent Self-Play RL methods, inspired by the success of the paradigm in games and Go, aim to enhance LLM reasoning capabilities without human-annotated data. However, their methods primarily depend on a grounded environment for feedback ( e.g. , a Python interpreter or a game engine); extending them to general domains remains challenging. To address these challenges, we propose Multi-Agent Evolve (MAE) , a framework that enables LLMs to self-evolve in solving diverse tasks, including mathematics, reasoning, and general knowledge Q&A. The core design of MAE is based on a triplet of interacting agents ( Proposer , Solver , Judge ) that are instantiated from a single LLM, and applies reinforcement learning to optimize their behaviors. The Proposer generates questions, the Solver attempts solutions, and the Judge evaluates both while co-evolving. Experiments on Qwen2.5-3B-Instruct demonstrate that MAE achieves an average improvement of 4.54% on multiple benchmarks. These results highlight MAE as a scalable, data-efficient method for enhancing the general reasoning abilities of LLMs with minimal reliance on human-curated supervision. 1 Introduction Reinforcement Learning (RL) ( Kaelbling et al. 1996 ; Silver et al. 2014 ) has demonstrated substantial potential in training Large Language Models (LLMs), leading to notable improvements in tasks such as coding and reasoning ( Guo et al. 2025 ) . However, these successes rely heavily on human-curated datasets, where ground truth answers are available to provide verifiable rewards ( Shao et al. 2024 ) . Human-curated datasets are costly and limited in numbers, which raises concerns about their scalability. Moreover, if LLMs are to advance beyond human-level intelligence in general domains, they will likely require training signals that surpass the capacity of human curation. In this paper, we focus on the central research question: can we build an effective RL framework for LLM to self-improve without human annotation in general domains? Self-Play has long been a proven paradigm for achieving self-improvement in machine learning, particularly in environments with well-defined feedback such as Go, and other games ( OpenAI et al. 2019 ; Silver et al. 2017 ; Klein 2022 ) . By letting models compete with themselves, Self-Play enables the discovery of progressively stronger strategies without human supervision. Recent studies have extended this idea to LLMs, demonstrating success in tasks such as code reasoning and games ( Zhao et al. 2025 ; Liu et al. 2025a ) . However, existing approaches typically depend on grounded environments that can provide verifiable feedback ( e.g. , a Python interpreter or a game engine), making it challenging to generalize Self-Play to open-ended domains such as natural language reasoning or general knowledge. The key challenge lies in designing reward signals that can improve LLMs’ general capabilities without relying on such domain-specific grounding, as rewards in most real-world reasoning scenarios are inherently ambiguous and difficult to quantify. To address these challenges, we propose Multi-Agent Evolve (MAE) , a multi-agent self-evolving framework that extends the Self-Play paradigm to general domains. Multi-Agent Evolve instantiates three cooperative while competing roles (Proposer, Solver, and Judge) from a single base LLM. The Proposer generates questions, the Solver produces answers, and the Judge follows the LLM-as-a-Judge paradigm ( Gu et al. 2024 ) to evaluate their generation and provide reward signals. This design forms a self-rewarding loop where the model can assess and improve itself without external supervision or domain-specific ground truth, as shown in Figure 1 . The Proposer and Solver engage in an adversarial interaction : the Solver is rewarded by the Judge for accurate and well-reasoned answers, while the Proposer receives both a quality reward from the Judge and a difficulty reward that increases when the Solver fails, driving co-evolution toward more challenging and informative tasks. To further ensure stable and scalable training, MAE applies format rewards and quality filtering , filtering out low-quality questions based on the Judge’s evaluation scores. Experimental results validate the effectiveness of Multi-Agent Evolve. Even from a minimal setup without real-world data or verifiable rewards, MAE improves upon the base model across nearly all benchmark types and outperforms the strong AZR baseline. When initialized with a small seed of unlabeled reference questions, MAE’s performance is further amplified. This approach significantly outperforms standard Supervised Fine-Tuning (SFT) on the same dataset, even though SFT relies on ground-truth answers while MAE does not. Figure 1: Overview of the Multi-Agent Evolve Framework. Multi-Agent Evolve instantiates three interactive roles ( Proposer , Solver , and Judge ) from a single LLM to form a closed self-improving loop. The Proposer generates new questions, the Solver attempts to answer them, and the Judge evaluates both to provide general-domain reward signals. The Judge rewards the Solver for accurate reasoning, while the Proposer receives both a quality reward from the Judge and a difficulty reward that increases when the Solver fails, creating an adversarial co-evolution process that continuously enhances the model’s reasoning ability. In summary, our main contributions are as follows: • We introduce Multi-Agent Evolve(MAE) , a multi-agent self-evolving framework that instantiates three interactive roles— Proposer , Solver , and Judge —from a single base LLM and jointly trains them via reinforcement learning. The framework forms a closed-loop propose–solve–judge pipeline with synchronized updates, supports both with/without reference settings, and integrates quality filtering to stabilize self-evolution. • We design domain-agnostic self-rewarding mechanisms , including Judge-based evaluation, difficulty-aware rewards, and format rewards, which eliminate the reliance on human-labeled ground truth or external verifiers. • We empirically demonstrate the effectiveness and scalability of Multi-Agent Evolve on Qwen2.5-3B-Instruct ( Qwen et al. 2025 ) across mathematics, coding, reasoning, and general knowledge benchmarks, achieving improvements over both base and supervised fine-tuning baselines. We also give a thorough analysis of the training process and an ablation study to investigate the contribution of our components. 2 Related Works LLM-as-a-judge The paradigm of “LLM-as-a-judge” ( Gu et al. 2024 ) leverages the advanced capabilities of Large Language Models (LLMs) to perform evaluation tasks such as scoring, ranking, or selection, which have traditionally relied on costly human annotation ( Li et al. 2024 ) . This approach has been widely adopted across the LLM life-cycle for applications including evaluation, alignment, retrieval, and reasoning. Although it has some limitations, it has become a promising approach to provide self-training signals. Early prompting-based methods ( Raman et al. 2024 ; Weng et al. 2022 ) , utilizing LLMs themselves to provide generative feedback and refine their answers. Following work, like Pang et al. 2023 , increases answering accuracy for reasoning tasks through a teacher-student self-play. Yuan et al. 2025 use Self-Rewarding Language Models to iteratively train their instruction-following capability given the self-rated scores. In our paper, the LLM-as-a-judge paradigm is integrated into the system and participates in the model’s evolution. Self-Play for LLM Self-Play is a data-free approach that requires minimal human supervision. It primarily relies on the system’s capabilities and its self-interaction to enable higher intelligence to emerge. Self-Play

Read the complete text