PopuLoRA: Co-Evolving LLM Populations for Reasoning Self-Play
Roger Creus Castanyer, Geoffrey Bradway, Lorenz Wolf, Maxwill Lin, Augustine N. Mavor-Parker, Matthew James Sargent · 2026
Read PopuLoRA: Co-Evolving LLM Populations for Reasoning Self-Play by Roger Creus Castanyer, Geoffrey Bradway, Lorenz Wolf, Maxwill Lin, Augustine N. Mavor-Parker, Matthew James Sargent (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-16727.txt.
Introduction
Castanyer and colleagues replace single-agent self-calibration with groups cross-checking each other in PopuLoRA, a population-based asymmetric self-play system for reinforcement learning with verifiable rewards in language models. Teachers and students are specialist LoRA adapters sharing one frozen base; teachers set problems matched students solve under a programmatic verifier. Weight-space mutations and crossovers evolve the population at 7B scale. The population mean reportedly beats a compute-matched single-agent baseline on code and math tests. Even the weakest member also beats that baseline overall.
Catalog record
| CREATOR | Roger Creus Castanyer, Geoffrey Bradway, Lorenz Wolf, Maxwill Lin, Augustine N. Mavor-Parker, Matthew James Sargent |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58645 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.16727
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PopuLoRA: Co-Evolving LLM Populations for Reasoning Self-Play Roger Creus Castanyer, Geoffrey Bradway, Lorenz Wolf, Maxwill Lin, Augustine N. Mavor-Parker, Matthew James Sargent Abstract We introduce PopuLoRA , a population-based asymmetric self-play framework for reinforcement learning with verifiable rewards (RLVR) post-training of LLMs. Teachers and students are specialised LoRA adapters on a shared frozen base: teachers propose problems, matched students solve them under a programmatic verifier, and cross-evaluation between sub-populations replaces the self-calibration that limits single-agent self-play. A family of LoRA weight-space evolution operators (mutations and crossovers that produce same-rank population members in seconds) serves as the replacement step of a population-based training loop at 7B scale. We instantiate PopuLoRA on top of Absolute Zero Reasoner and compare it against a per-adapter compute-matched single-agent baseline. Where the single agent self-calibrates to generating easy problems it can reliably solve, the population enters a co-evolutionary arms race: teachers produce increasingly complex problems, student solve rates oscillate, and problem-space coverage keeps expanding throughout training. Despite lower training-time reward, the population mean outperforms the baseline on three code benchmarks (HumanEval+, MBPP+, LiveCodeBench) and seven math benchmarks (AIME 24/25, AMC 23, MATH-500, Minerva, GSM8K, OlympiadBench), and even the weakest member of the population beats the baseline on aggregate. \correspondence \emails roger, augustine, matthew 1 Introduction RL post-training has become a dominant regime for specialising large language models, from RLHF ( 41 ) and DPO on human preference data through self-play fine-tuning ( 7 ) and large-scale reinforcement learning with verifiable rewards (RLVR) ( 16 ; 34 ) . The optimisation machinery has matured, but the supply of problems has not: most current systems depend on a hand-curated task distribution whose scope, difficulty, and coverage must be chosen in advance. The open question we study in this paper is how to generate the curriculum itself , the stream of problems the policy is trained on, without relying on human-authored datasets, using only a programmatic verifier as the external signal. The most direct approach lets a single model propose its own problems and grade itself through the verifier, as in Absolute Zero Reasoner ( 62 ) or, more broadly, single-model self-play ( 7 ; 32 ) . The shared flaw is that the same network generates problems and (implicitly, through its solve rate or judgement) estimates their difficulty. In this paper we provide empirical evidence that such single-agent self-generation self-calibrates : the proposer converges to generating problems it can consistently produce in valid format and consistently solve, and the training distribution collapses onto a narrow band long before the base model’s capability is exhausted. The fix is structural: make the judge a different agent from the proposer. A growing line of work explores this asymmetry ( 5 ; 30 ; 38 ; 49 ; 11 ; 50 ; 57 ; 24 ) , but all are scalar in agent count ( \leq 3 agents total), whether the roles share parameters or train separate networks. We take the asymmetry further, building on the teacher-proposes / student-solves structure introduced by 48 , and replace the single agent with co-evolving populations of specialised teachers and students . Teachers are rewarded for producing problems that are hard for the particular student they face, students are rewarded by the verifier, and matchmaking across the two sub-populations turns difficulty into a population-level signal rather than a self-estimate. Population dynamics add a second layer of exploration on top of gradient updates: lineages diverge, members specialise, and a population-based training (PBT) ( 27 ) replacement step recombines what the gradient path has already discovered. Running many independent full-parameter 7B models on a single node is expensive and memory-constrained, particularly when each member must support both rollout inference and gradient updates in a shared training loop. We therefore instantiate every population member as a LoRA adapter ( 21 ) over a shared frozen base, which collapses the population’s memory footprint to the sum of its adapter weights rather than a full copy of the base per member. Classical PBT ( 27 ) mutates members by copying one agent’s full weights onto another and perturbing; at 7B scale that copy is itself expensive. Our second contribution is a set of LoRA weight-space evolution operators (SVD-structured, layer-selective, and component-masking mutations together with DARE ( 58 ) , TIES-inspired ( 56 ) , and task-arithmetic ( 26 ) crossovers, in the spirit of evolutionary model merging ( 1 ) ) that produce same-rank children in seconds without any retraining. They serve as the replacement step of an online PBT loop, making a population-of-adapters regime that is inaccessible to prior LoRA composition work ( 23 ; 4 ; 13 ; 61 ) . We evaluate in the Absolute Zero code-reasoning setting, with a sandboxed Python executor as the verifier. CONTRIBUTIONS 1. Population-based asymmetric self-play for RLVR. Prior work ( 5 ; 30 ; 38 ; 49 ; 11 ; 50 ; 57 ; 24 ; 32 ) is scalar in agent count ( \leq 3 agents). PopuLoRA replaces these with populations of teacher and student LoRA adapters coupled by TrueSkill-weighted cross-evaluation. 2. LoRA weight-space evolution operators as the PBT replacement step. Mutations and crossovers produce rank-matched children in seconds; closest is evolutionary model merging ( 1 ) , applied offline rather than inside an online PBT loop. 3. Empirical validation on held-out code and math benchmarks. The population outperforms a per-adapter compute-matched single-agent baseline; diagnostics confirm it avoids the mode-collapse we observe in the baseline. 2 Background and Related Work RLVR and self-generated curricula. RLVR ( 16 ) replaces learned preference models with programmatic checkers. AZR ( 62 ) takes this to its logical extreme: a single code LLM both proposes and solves its own problems, rewarded only by a sandboxed executor, with no external dataset. Adjacent methods (STaR ( 60 ) , rStar-Math ( 15 ) , Self-Rewarding ( 59 ) ) also train on self-generated data but rely on fixed problem sets or learned reward signals; only AZR treats the model as both proposer and verifier-checked solver. We evaluate PopuLoRA on the three code-reasoning task types AZR defines: code_i (infer-input), code_o (infer-output), and code_f (infer-function), all verified mechanically by the executor. Self-play and asymmetric roles. Asymmetric self-play ( 48 ) , where one agent proposes tasks and another solves them, is the structural ancestor of PopuLoRA’s teacher–student loop. In the unsupervised environment design (UED) literature, PAIRED ( 10 ) formalises regret-based adversarial curriculum generation, POET ( 54 ) co-evolves environments and agents, and ACCEL ( 42 ) adds mutation operators on level structure; emergent autocurricula arise in multi-agent competition ( 3 ) . In the LLM setting, a growing line of work separates proposer and solver: SPIN ( 7 ) , Language Self-Play ( 32 ) , SOAR ( 49 ) , R-Zero ( 24 ) , ALIVE ( 11 ) , TriPlay-RL ( 50 ) , and others (Appendix 6 ). All are scalar in agent count ( \leq 3 agents total). PopuLoRA differs in three ways: (i) we train populations of teachers and students rather than a single pair, (ii) updates are joint and on-policy rather than alternating, and (iii) the difficulty signal comes from cross-evaluation across the population rather than from a fixed target solve-rate band.