Learning to Self-Evolve
Xiaoyin Chen, Canwen Xu, Yite Wang, Boyi Liu, Zhewei Yao, Yuxiong He · 2026
Read Learning to Self-Evolve by Xiaoyin Chen, Canwen Xu, Yite Wang, Boyi Liu, Zhewei Yao, Yuxiong He (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-18620.txt.
Introduction
Chen and co-authors teach language models to improve their own context at test time. Their Learning to Self-Evolve framework uses reinforcement learning for this skill. Each context edit earns reward from the gain in later task scores. That trick turns a multi-step process into a single-step training goal. A tree-guided loop steers edits during use. A 4B model trained this way beats self-evolving setups run by GPT-5 and Claude Sonnet 4.5. It can also guide other models with no extra training.
Catalog record
| CREATOR | Xiaoyin Chen, Canwen Xu, Yite Wang, Boyi Liu, Zhewei Yao, Yuxiong He |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43372 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.18620
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Learning to Self-Evolve Xiaoyin Chen, Canwen Xu, Yite Wang, Boyi Liu, Zhewei Yao, Yuxiong He Abstract We introduce Learning to Self-Evolve (LSE), a reinforcement learning framework that trains large language models (LLMs) to improve their own contexts at test time. We situate LSE in the setting of test-time self-evolution, where a model iteratively refines its context from feedback on seen problems to perform better on new ones. Existing approaches rely entirely on the inherent reasoning ability of the model and never explicitly train it for this task. LSE reduces the multi-step evolution problem to a single-step RL objective, where each context edit is rewarded by the improvement in downstream performance. We pair this objective with a tree-guided evolution loop. On Text-to-SQL generation (BIRD) and general question answering (MMLU-Redux), a 4B-parameter model trained with LSE outperforms self-evolving policies powered by GPT-5 and Claude Sonnet 4.5, as well as prompt optimization methods including GEPA and TextGrad, and transfers to guide other models without additional training. Our results highlight the effectiveness of treating self-evolution as a learnable skill. 1 1 1 Code is available at https://github.com/chenyn66/learning-to-self-evolve . 1 Introduction The ability to adapt and evolve in response to environmental feedback has long been considered central to human intelligence ( 15 ; 18 ) . A chess player improves by analyzing past games; a software engineer grows more proficient with a codebase through months of daily work. In both cases, experience accumulates and the person adjusts their approach accordingly. Current large language model (LLM) training pipelines exhibit a similar dynamic, particularly at the post-training stage, where reinforcement learning (RL) refines the behavior of the model on its own generated data ( 12 ; 13 ; 5 ) . However, this learning stops once training ends. At deployment, an LLM applies the same policy regardless of how many problems it has solved in a domain, and discards all accumulated experience once the context resets. This gap between static deployment and dynamic adaptation motivates the study of test-time self-evolving systems: systems that continuously update themselves in response to new observations at test time. Figure 1: Overview of Learning to Self-Evolve (LSE). Left: Tree-guided self-evolution at test time. Upper Confidence Bound (UCB) selection chooses a context from the evolution tree; the action model generates outputs for a new batch of problems; the self-evolving policy receives the performance summary and proposes a revised context. Right: LSE trains the self-evolving policy via RL with an improvement-based reward computed as the difference between post-edit and pre-edit performance. Test-time self-evolution can be characterized along at least two dimensions: how the policy is updated and when . On one end of the first dimension, gradient-based methods modify model parameters directly; on the other, prompt-based methods rewrite the model context while keeping parameters frozen. Along the second dimension, intra-episode evolution updates the policy within a single episode: the model revisits its own attempts and refines its answer to a particular problem, trading additional compute for instance-level gains ( 17 ; 10 ; 27 ) . Inter-episode evolution updates the policy after one or more completed episodes and applies the result to new problems, extracting transferable knowledge that generalizes across tasks ( 24 ; 8 ; 30 ) . We focus on inter-episode, prompt-based self-evolution: an LLM observes its performance on a batch of problems and rewrites its own context to improve on the next batch. Several recent works explore this direction through automatic prompt optimization ( 9 ; 1 ; 26 ) , self-referential updates ( 6 ; 32 ; 30 ; 8 ; 31 ) , and agentic memory systems ( 29 ; 31 ; 4 ) . These methods, however, rely entirely on the inherent ability of the LLM to analyze feedback and propose better context. The model is never explicitly trained for this self-improvement task. We argue that self-evolution poses a reasoning challenge distinct from other reasoning domains. The process, in essence, shares the structure of an RL problem. An RL optimizer relies on dedicated algorithms to assign credit, estimate gradients, and balance exploration against exploitation. In self-evolution, the model must perform all three implicitly, through natural language reasoning alone. It must judge which parts of the current context help and which hurt, anticipate how a revision will change downstream behavior, and decide whether to refine what works or try something new. These demands motivate explicit optimization for self-evolution. We propose Learning to Self-Evolve (LSE) , an RL framework that explicitly trains an LLM to be an effective self-evolving policy. Rather than optimizing over the full multi-step evolution trajectory, LSE simplifies training to a single step: the model receives the current context and a performance summary, and produces a better context. Each edit is rewarded by the improvement in downstream performance, instead of the absolute post-edit score. At test time, we leverage a tree-guided evolution loop that allows the system to explore and backtrack across possible contexts. We evaluate LSE on Text-to-SQL generation and general question answering. Despite using only a 4B-parameter model, the LSE-trained policy outperforms self-evolving policies powered by frontier models such as GPT-5 and Claude Sonnet 4.5, as well as prompt optimization methods such as GEPA and TextGrad. Our contributions are as follows: • We formalize test-time inter-episode self-evolution and operationalize it through prompt-based updates with tree-guided search (§ 3.1 , § 3.2 ). • We propose LSE, an RL framework that explicitly trains the self-evolving policy with an improvement-based reward (§ 3.3 ). • We show that a 4B-parameter model trained with LSE outperforms larger untrained models and prompt optimization methods, and transfers to guide other models without additional training (§ 4 ). 2 Related Work The term self-evolution has been used to refer to many different concepts in recent LLM research. We organize the landscape into two broad categories. Training-time self-evolution focuses on using LLMs to generate their own training data and learning signals during training. Test-time self-evolution enables a policy to continue updating itself after training, adapting dynamically based on experience accumulated during deployment. Training-time self-evolution. A growing body of work leverages LLMs to generate their own data and learning signals during training. RL-based post-training has the model produce reasoning traces and optimizes them against verifiable rewards ( 12 ; 13 ; 5 ) . Bootstrapping methods such as STaR ( 28 ) iteratively generate candidate rationales and fine-tune on the correct ones. Self-rewarding approaches ( 25 ; 34 ) extend this by using the model itself as the reward signal. Absolute Zero ( 33 ) takes this to its extreme: a single model both proposes and solves tasks with no external data, using a code executor as the sole source of verifiable reward. While these methods produce stronger models, the resulting policy remains static once training ends. Our work addresses a complementary problem: enabling the policy to continue improving at test time. Test-time self-evolution. A static policy cannot accommodate distribution shifts encountered at test time. Test-time self-evolution addresses this by enabling the model to self-update based on its own experience after deployment. This capability spans two temporal scales. Intra-episode methods improve on a single problem instance by allocating additional compute. Reflexion ( 17 ) prompts the model to reflect on failed attempts and