Scaling Self-Play with Self-Guidance
Luke Bailey, Kaiyue Wen, Kefan Dong, Tatsunori Hashimoto, Tengyu Ma · 2026
Read Scaling Self-Play with Self-Guidance by Luke Bailey, Kaiyue Wen, Kefan Dong, Tatsunori Hashimoto, Tengyu Ma (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-20209.txt.
Introduction
Bailey and four colleagues tackle self-play scaling failure, where a Conjecturer inventing problems for a Solver games its reward. The Conjecturer collapses into pointlessly hard problems that do not help the Solver, stalling learning. Self-Guided Self-Play adds a Guide scoring synthetic problems for relevance to unsolved targets and cleanliness, blocking that degeneracy. In Lean4 theorem proving it beats the strongest reinforcement learning baseline's asymptotic solve rate in under eighty rounds. After two hundred rounds, a 7B model out-solves a 671B model at pass four.
Catalog record
| CREATOR | Luke Bailey, Kaiyue Wen, Kefan Dong, Tatsunori Hashimoto, Tengyu Ma |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59292 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.20209
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Scaling Self-Play with Self-Guidance Luke Bailey, Kaiyue Wen, Kefan Dong, Tatsunori Hashimoto, Tengyu Ma Abstract LLM self-play algorithms are notable in that, in principle, nothing bounds their learning: a Conjecturer model creates problems for a Solver, and both improve together. However, in practice, existing LLM self-play methods do not scale well with large amounts of compute , instead hitting learning plateaus. We argue this is because over long training runs, the Conjecturer learns to hack its reward, collapsing to artificially complex problems that do not help the Solver improve. To overcome this, we introduce Self-Guided Self-Play (SGS), a self-play algorithm in which the language model itself guides the Conjecturer away from degeneracy. In SGS, the model takes on three roles: Solver, Conjecturer, and a Guide that scores synthetic problems by their relevance to unsolved target problems and how clean and natural they are, providing supervision against Conjecturer collapse. Our core hypothesis is that language models can assess whether a subproblem is useful for achieving a goal. We evaluate the scaling properties of SGS by running training for significantly longer than prior works and by fitting scaling laws to cumulative solve rate curves. Applying SGS to formal theorem proving in Lean4, we find that it surpasses the asymptotic solve rate of our strongest RL baseline in fewer than 80 rounds of self-play and enables a 7B parameter model, after 200 rounds of self-play, to solve more problems than a 671B parameter model pass@4. Stanford University † † footnotetext: Correspondence to [email protected] Code available at https://github.com/LukeBailey181/sgs Figure 1: Left: Intuition behind SGS. Bottom, the space of problems solvable over course of SGS, starting as a small area , expanding past asymptotic RL performance , directed towards unsolved target problems ( \blacklozenge ) by synthetic problems ( \bm{\times} ). Top, the Guide score for synthetic problems, which is high along the path to unsolved targets, and low for not related and inelegant problems (see Figure 2 ). Right: Results of running SGS on D_{\text{3k}} ( \approx 3000 Lean4 formal math problems) compared to our best performing RL baseline ( Section 4.3 ). At 6.3M generations, SGS applied to the 7B parameter DeepSeek-Prover-V2 model exceeds the pass@4 of the larger 671B counterpart (DSPv2 671B). 1 Introduction We would like AI systems that, given enough compute, can solve any problem we give them. Unfortunately, when a problem is sufficiently difficult, standard reinforcement learning techniques fail due to sparse or absent reward ( Prakash & Buvanesh 2025 ; Qu et al. 2026 ) . Asymmetric self-play offers a solution to the sparse reward problem. In this framework, a Conjecturer model proposes new learning problems that bridge the gap between the current Solver capability and the unsolved problems ( Sukhbaatar et al. 2017 ; Florensa et al. 2018 ) . Over time, the Solver and Conjecturer improve together. Self-play algorithms are unique in that, in principle, nothing bounds their learning ( Silver et al. 2017 ) . However, existing LLM self-play methods do not sustain learning over long periods ( Yu et al. 2025b ; Chae et al. 2025 ) . In general, these methods reward the synthetic problems only according to the Solver’s pass rate on the problem ( Zhao et al. ; Huang et al. 2025 ; Chen et al. 2025c ; Liu et al. 2025 ; Yang et al. 2026 ; Jana et al. 2026 ; Kuba et al. 2025 ) . Over time the Conjecturer hacks this reward function, collapsing to artificially complex problems that are not useful for learning ( Figure 2 ) ( Dong & Ma 2025 ) . That is, the quality of the synthetic problems degrades over the course of training , leading to a learning plateau. We can quantify this plateauing problem by fixing a set of hard training problems and seeing how many we can solve with self-play. We strive for an algorithm that, if you run it for long enough, leads to all the problems being solved. In the pursuit of this ideal, we ask the question: Given a fixed set of hard problems, how can we scale self-play to solve as many as possible? Towards this end, we propose Self-Guided Self-Play (SGS). SGS is an asymmetric self-play algorithm in which an LLM takes on three roles, Solver, Conjecturer, and Guide. It has two mechanisms to stop Conjecturer degradation over the course of training. First, for each unsolved problem, the Conjecturer is prompted to produce a synthetic problem that is useful for solving that problem. Secondly, the Guide is used to score each synthetic problem by how relevant it is to its corresponding unsolved problem, and how clearly it is formulated. Crucially, a synthetic problem that is superficially related to a target but is messy, overly complex, or inelegant, is downweighted by the Guide. The Solver is rewarded for every problem it solves, while the Conjecturer is rewarded for producing problems that the Solver can make progress on and that the Guide judges to be high quality. Our core hypothesis is that language models can assess whether a subproblem is useful for achieving a goal, and thus guide the learning process towards that end. To evaluate the scaling dynamics of SGS, we take two complementary approaches. First, we run training for much longer than prior works: our main experiment produces over 6 billion tokens during the self-play procedure, epoching the target data over 230 times ( Zhao et al. ; Huang et al. 2025 ; Chen et al. 2025c ) . Second, we fit scaling laws to the cumulative solve rate over training, allowing us to extrapolate long-run behavior and compare the asymptotic solve rates of different methods ( Khatri et al. 2025 ; Ruan et al. 2024 ) . We use this evaluation method to analyze the long-running dynamics of SGS applied to formal theorem proving in Lean4, where the goal is to generate formally verified proofs of mathematical statements ( De Moura et al. 2015 ; Moura & Ullrich 2021 ) . In this setting, the target problems are Lean theorems, and solutions are proofs that can be automatically checked by the Lean compiler. We provide a summary of our contributions below: • In Section 4.3 , we show that SGS outperforms RL and parallel sampling baselines, achieving a 7% higher asymptotic solve rate than RL alone. With sufficient compute, SGS enables DeepSeek-Prover-V2-7B to surpass the pass@4 of the much larger DeepSeek-Prover-V2-671B model ( Ren et al. 2025 ) . • In Section 4.4 , we study the design of the Conjecturer training algorithm. We ablate conditioning the Conjecturer on unsolved problems and the Guide component. In both cases we find the Conjecturer collapses to producing degenerate problems that do not help the Solver make progress on the training problems. We also show that freezing the Conjecturer is suboptimal, as the Solver quickly saturates the fixed problem distribution. • In Section 4.5 , we study the design of the Solver training algorithm. We show that Solver entropy collapse can starve the Conjecturer of training signal, making the choice of Solver RL objective critical for long-running self-play. Overall, our key findings are as follows: (1) scaling asymmetric self-play requires avoiding undesirable collapses in the Solver