Training LLM Agents for Spontaneous, Reward-Free Self-Evolution via World Knowledge Exploration
Qifan Zhang, Dongyang Ma, Tianqing Fang, Jia Li, Jing Tang, Nuo Chen, Haitao Mi, Yan Wang · 2026
Read Training LLM Agents for Spontaneous, Reward-Free Self-Evolution via World Knowledge Exploration by Qifan Zhang, Dongyang Ma, Tianqing Fang, Jia Li, Jing Tang, Nuo Chen, Haitao Mi, Yan Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-18131.txt.
Introduction
Zhang and seven colleagues train language-model agents to explore unfamiliar digital environments on their own. No human rewards or rules guide the agent at run time. During training only, a reward measures how much the agent's self-written world knowledge improves later task success. The trained agent gains 20 percent on WebVoyager and WebWalker tests, and a compact 14-billion-parameter model beats an unassisted Gemini-2.5-Flash.
Catalog record
| CREATOR | Qifan Zhang, Dongyang Ma, Tianqing Fang, Jia Li, Jing Tang, Nuo Chen, Haitao Mi, Yan Wang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 71484 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.18131
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Training LLM Agents for Spontaneous, Reward-Free Self-Evolution via World Knowledge Exploration Qifan Zhang, Dongyang Ma, Tianqing Fang, Jia Li, Jing Tang, Nuo Chen, Haitao Mi, Yan Wang Abstract Most agents today “self-evolve” by following rewards and rules defined by humans. However, this process remains fundamentally dependent on external supervision; without human guidance, the evolution stops. In this work, we train agents to possess an intrinsic meta-evolution capability to spontaneously learn about unseen environments prior to task execution. To instill this ability, we design an outcome-based reward mechanism that measures how much an agent’s self-generated world knowledge improves its success rate on downstream tasks. This reward signal is used exclusively during the training phase to teach the model how to explore and summarize effectively. At inference time, the agent requires no external rewards or human instructions. It spontaneously performs native self-evolution to adapt to unknown environments using its internal parameters. When applied to Qwen3-30B and Seed-OSS-36B, this shift to native evolution yields a 20% performance increase on WebVoyager and WebWalker. Most strikingly, the generated world knowledge even enables a compact 14B Qwen3 model to outperform the unassisted Gemini-2.5-Flash, establishing a new paradigm for truly evolving agents. 1]Tencent 2]The Hong Kong University of Science and Technology (Guangzhou) \contribution [*]Equal contribution \contribution [†]Project Lead \headercontent Code Models Data Figure 1 : Progression of self-evolution agent paradigms. Left: Experience-Driven Evolution updates agents through predefined tasks and external rewards, requiring extensive human effort to design these components. Center: Adversarial Evolution employs a challenger-solver dynamic where one proposes harder tasks and the other improves to solve them; while tasks and rewards are agent-generated, the cooperative pipeline still requires human setup. Right: Our Meta-Learning-Driven Evolution enables agents to autonomously explore and compress environments into reusable world knowledge for adaptation, achieving a task- and reward-free paradigm with minimal human intervention. 1 Introduction Current research on “self-evolving” agents is largely an illusion. Most existing methods do not allow an agent to evolve on its own; instead, they depend on human-defined workflows and verified reward signals to guide every step of improvement. If these external rewards or instructions are removed, the evolution stops. We argue that such agents are not truly autonomous—they are merely being instructed by humans within predefined guidance. They lack the fundamental ability to decide their own direction of growth when facing a completely new environment. This paradigm is fundamentally different from human intuition. Human intelligence is naturally curious and proactive. When we enter a new city or start using a new software, we spontaneously learn the layout and the underlying logic, even without a specific task or a verified reward. This learning process is entirely workflow-free and reward-free. We build an internal map of the world simply because understanding the environment is a prerequisite for intelligence. Today’s agents, however, are passive; they wait for instructions and rewards before they begin to “evolve.” In this work, we bridge this gap by granting agents Native Agency . Our goal is to move beyond task-specific optimization and achieve a truly Workflow-free and Reward-free self-evolution. We train the model to possess an intrinsic meta-evolution capability: the ability to explore a novel environment and distill its observations into structured “World Knowledge” entirely on its own. As shown in Figure 1 , our agent does not follow a predefined script; it defines its own path of discovery. The core challenge is: how can we train an agent to explore and summarize effectively without human-provided rewards? To solve this, we propose an outcome-based reward mechanism used exclusively during the training phase. We measure the quality of self-generated knowledge by its “utility”— specifically, how much it improves the success rate on downstream tasks. This allows the model to learn how to evolve during training, using a multistage pipeline that includes teacher-model bootstrapping (SFT) and on-policy reinforcement rejection sampling (RFT). Once trained, the agent no longer requires any external guidance or reward signals to adapt to an unseen world at inference time. We evaluate our approach on two major web-based benchmarks: WebVoyager [ 11 ] and WebWalker [ 38 ] . Our experiments show that our method, when applied to Qwen3-30B [ 41 ] and Seed-OSS-36B [ 27 ] , achieves a significant absolute performance increase of approximately 20% over standard baselines. Strikingly, the generated world knowledge even enables a compact Qwen3-14B [ 30 ] model to outperform the unassisted Gemini-2.5-Flash [ 6 ] . These results prove that agents can be trained to possess the innate ability to understand and adapt to the unknown entirely on their own, without any human intervention or inference-time rewards. 2 Related Works 2.1 Self-Evolving Agents Self-evolving agents are designed to autonomously explore novel environments and continuously improve their capabilities without direct human intervention [ 10 ] . However, the current paradigm of “self-evolution” is often an illusion. In reality, these agents still heavily rely on meticulously human-defined workflows and verified, environment-specific reward signals to guide every incremental step of their improvement. Broadly, existing approaches can be categorized into the following two paradigms: Experience-Driven Evolution. As depicted in the left panel of Figure 1 , this paradigm fundamentally relies on human-crafted tasks and predefined reward functions tailored to a target environment. The agent iteratively attempts to solve these tasks, generating execution trajectories that are subsequently evaluated by the reward signals. These trajectory-score pairs—collectively termed as experience —serve as the primary learning signal. By leveraging this accumulated experience, the agent optimizes its future performance through various updating mechanisms, such as refining its system prompts [ 47 , 34 , 39 , 22 , 42 ] , expanding external memory databases [ 20 , 44 , 48 , 9 , 40 , 5 ] , augmenting tool and skill libraries [ 45 , 49 , 21 , 32 ] , or directly fine-tuning its internal model parameters [ 46 , 7 , 35 , 37 , 24 , 33 , 31 ] . However, this paradigm is fundamentally bottlenecked by the massive human labor required to engineer these tasks and rewards. Rather than genuinely exploring, the agent merely passively adapts to the environment by studying from these human-provided “textbooks”. Adversarial Evolution. To alleviate manual design efforts, an alternative paradigm employs heavily engineered adversarial workflows . As illustrated in Figure 1 (middle), a challenger agent synthesizes environment-specific tasks for a solver to execute. Through this zero-sum game, the solver refines its capabilities while the challenger generates increasingly difficult tasks to push its boundaries [ 15 , 13 , 50 , 23 , 43 ] . Although this paradigm substantially reduces human labor by bypassing manual task and reward design, it merely shifts the engineering burden to orchestrating complex agent workflows. Furthermore, the agent remains trapped solving synthesized “exercise books”, failing to break free and engage in genuine, unguided exploration within the environment. Meta-Learning-Driven Evolution (Ours). To overcome the limitations of previous paradigms and empower agents to achieve workflow-free and task- and reward-free self-evolution, we propose a novel meta-evolution paradigm . As illustrated in the right panel of Figure 1 , under this paradigm, the agent spontaneously explores the environment and compresses raw environmental observations into structured world knowledge . This knowledge acts as a ”mental map” that significantly enhances downstream performance, eliminating