Self-Play Only Evolves When Self-Synthetic Pipeline Ensures Learnable Information Gain

Wei Liu, Siya Qi, Yali Du, Yulan He · 2026

Read Self-Play Only Evolves When Self-Synthetic Pipeline Ensures Learnable Information Gain by Wei Liu, Siya Qi, Yali Du, Yulan He (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-02218.txt.

Introduction

Wei Liu, Siya Qi, Yali Du, and Yulan He ask when self-play can keep improving a language model in this 2026 paper. In self-play, a model makes tasks, tries them, and learns from the results. Many such loops stall fast, the authors note. The data they make adds volume but no new learnable content.

Experiments on a coding task pin down what a working loop needs. New data must carry more learnable content in each round than the last. The model plays three roles: proposer, solver, and checker of answers. Three design fixes target growth in learnable content from those roles. Roles can evolve in a loop, budgets can grow with content, and fresh outside tasks can prevent stalls.

Catalog record

CREATORWei Liu, Siya Qi, Yali Du, Yulan He
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52908
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.02218

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Self-Play Only Evolves When Self-Synthetic Pipeline Ensures Learnable Information Gain Wei Liu, Siya Qi, Yali Du, Yulan He Abstract Large language models (LLMs) make it plausible to build systems that improve through self-evolving loops, but many existing proposals are better understood as self-play and often plateau quickly. A central failure mode is that the loop synthesises more data without increasing learnable information for the next iteration. Through experiments on a self-play coding task, we reveal that sustainable self-evolution requires a self-synthesised data pipeline with learnable information that increases across iterations. We identify triadic roles that self-evolving LLMs play: the Proposer , which generates tasks; the Solver , which attempts solutions; and the Verifier , which provides training signals, and we identify three system designs that jointly target learnable information gain from this triadic roles perspective. Asymmetric co-evolution closes a weak-to-strong-to-weak loop across roles. Capacity growth expands parameter and inference-time budgets to match rising learnable information. Proactive information seeking introduces external context and new task sources that prevent saturation. Together, these modules provide a measurable, system-level path from brittle self-play dynamics to sustained self-evolution. 1 Introduction Figure 1 : A self-evolving LLM plays three roles as Proposer , Solver and Verifier . The whole self-evolving process can be seen as different synthetic operations (synthesis qa, solution and feedback) on the same information source, which is the LLM itself. Figure 2 : Overall framework of a triadic self-evolving loop. A self-evolving LLM plays three roles: the Proposer and Verifier form the internal environment, proactively interacting with the external environment to provide data and supervision for the Solver . The Solver and internal environment co-evolve asymmetrically, adaptively expanding capacity to capture more learnable information. From an information perspective, the system continually absorbs external information, and transform them into internal learnable information. The rapid progress of large language models (LLMs) has made self-evolving AI systems plausible ( 13 ; 11 ) . In such systems, a model plays different roles to generate data ( Proposer ), produce solutions ( Solver ), and provide feedback signals ( Verifier ), thereby forming self-training loops autonomously. Related research has evolved from early self-supervised training systems ( 52 ; 50 ; 6 ; 49 ; 10 ; 9 ; 33 ; 38 ) towards more recent self-play systems trained with reinforcement learning ( 56 ; 20 ; 51 ; 46 ; 26 ; 7 ; 28 ; 17 ; 39 ; 16 ) . Some focus on verifiable domains such as mathematics and coding, where a fixed Verifier is available, and the LLM performs self-play between the Proposer and Solver roles. Others emphasise free-form domains, such as preference learning and instruction following, where the Solver and Verifier are co-evolved on a fixed dataset. In most cases, these systems adopt multi-reward reinforcement learning to achieve self-evolution. Despite their promise, such systems are often fragile and quickly enter a plateau or collapse after only a few rounds of self-play. In self-play between Proposer and Solver , 56 report that the Proposer tends to generate trivial identity-like problems ( f(x)=x ). 20 and 51 observe an early peak and subsequent decline in overall model performance. 7 report that the Proposer requires carefully tuned prompts to ensure the proposed data remains within a reasonable regime. Some approaches benefit from periodically introducing ground truth data to recalibrate the Verifier ( 46 ; 28 ) . Without such intervention, the system risks falling into a state of self-delusion, resulting in a rapid decline in overall performance. To resolve this mode collapse and achieve genuine self-evolution, we argue that self-evolution should be viewed as a healthy self-synthesised data pipeline, where the Proposer , Solver , and Verifier operate cooperatively to ensure a monotonic increase in learnable information, rather than relying on fragile self-play dynamics. This position extends beyond a trivial composition of P+S and S+V into P+S+V ( 46 ; 8 ) . It treats the self-evolving loop as a self-synthetic data pipeline (as shown in Figure 1 ), whose success is characterised by an increase in learnable information over iterations. We make this requirement precise by formalising learnable information under bounded observers in § 2.2 . The information-theoretic view makes this question concrete. It separates learnable structure from unlearnable noise under bounded computation, which clarifies why reward shaping alone is insufficient for sustained improvement. Empirical analysis of existing self-play loops further validates this diagnosis. Motivated by this observation, we propose three system-level design principles for the self-evolution loop. 1. Asymmetric Co-evolution. We frame self-evolution as a co-evolutionary process between the Solver and its internal environment, consisting of the Proposer and Verifier . Although all roles share the same information source (pre-trained model weights), and their divergent synthetic directions (synthesis question, solution and feedback) induce an information gap that enables weak-to-strong supervision. This asymmetry needs to be explicitly preserved by synchronising improvements in the Solver back to the internal environment as strong-to-weak supervision, thereby sustaining continuous self-improvement. 2. Capacity Growth. The learnable information in self-synthesised data is determined not only by the data distribution but also by the capacity of the observer model. As self-evolution progresses, the model needs to continually expand its effective capacity budgets , encompassing both parameter capacity and inference-time computation, to keep pace with the increasing amount of learnable information produced within the loop. 3. Proactive Information Seeking. Self-evolution driven by zero-data or a fixed dataset is fundamentally bounded by finite information. We argue that a self-evolving system should proactively acquire external information sources aligned with its current capabilities. Such sources provide not only new contexts for synthetic data but also new synthetic directions, creating fresh asymmetries that can be exploited by the internal co-evolutionary process. 2 Background We model self-evolution as an iterative process that produces self-synthetic training data D^{(t)} and updates a single LLM with it. The goal is not merely to improve task accuracy within a fixed self-play game, but to ensure that the learnable information in D^{(t)} increases during iterations. 2.1 Background on Triadic Self-Evolution View of System Design. At iteration t , the LLM acts as a Proposer to generate tasks, optionally conditioned on external context. It then acts as a Solver to produce solutions. It finally acts as a Verifier to assess the solutions and produce feedback signals. The next update trains the base model on the resulting self-synthetic training instances D^{(t)} , which include tasks, solutions, and verification signals. In this loop, there is no external labelled dataset, teacher model, or reward model, so the loop is self-evolving. The Proposer and Verifier form the internal environment ( 16 ) of the system. They jointly shape what the Solver practises and what feedback becomes a learnable training signal. An external environment, when present, provides information sources such as documents or interactive worlds ( 26 ) . Usually, such information enters as context that conditions task and evaluation generation. This three-role design matters for scope. P+S with a rule-based Verifier constrains its application to reinforcement learning from verifiable rewards (RLVR). S+V focuses on free-form domains and trains a better Verifier by learning on fixed preference data instead of self-synthesised data. Triadic self-evolution targets both regimes by evolving all three roles, which share one base model. Improvements

Read the complete text