WebSeer: Training Deeper Search Agents through Reinforcement Learning with Self-Reflection
Guanzhong He, Zhen Yang, Jinxin Liu, Bin Xu, Lei Hou, Juanzi Li · 2025
Read WebSeer: Training Deeper Search Agents through Reinforcement Learning with Self-Reflection by Guanzhong He, Zhen Yang, Jinxin Liu, Bin Xu, Lei Hou, Juanzi Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-18798.txt.
Introduction
Guanzhong He and his five co-authors train a web search agent. The agent reflects on its own work. Search agents gather data through repeated tool use. Errors pile up over long chains. Most systems keep their chains shallow for that reason. Their agent is called WebSeer. It trains with reinforcement learning. A self-reflection mechanism is added. A large data set supports the training. It is annotated with reflection patterns.
Training runs in two stages. A cold start is united with reinforcement learning. Both sit inside the self-reflection framework. The agent learns longer, more reflective tool-call chains. It works in real web environments. One model of moderate size reached leading results. It scored about seventy-two percent on a hard multi-hop benchmark. It scored ninety percent on a simpler one. It also generalized well to outside data sets.
Catalog record
| CREATOR | Guanzhong He, Zhen Yang, Jinxin Liu, Bin Xu, Lei Hou, Juanzi Li |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 69 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-18798
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
WebSeer: Training Deeper Search Agents through Reinforcement Learning with Self-Reflection Guanzhong He, Zhen Yang, Jinxin Liu, Bin Xu, Lei Hou, Juanzi Li Abstract Search agents have achieved significant advancements in enabling intelligent information retrieval and decision-making within interactive environments. Although reinforcement learning has been employed to train agentic models capable of more dynamic interactive retrieval, existing methods are limited by shallow tool-use depth and the accumulation of errors over multiple iterative interactions. In this paper, we present WebSeer, a more intelligent search agent trained via reinforcement learning enhanced with a self-reflection mechanism. Specifically, we construct a large dataset annotated with reflection patterns and design a two-stage training framework that unifies cold start and reinforcement learning within the self-reflection paradigm for real-world web-based environments, which enables the model to generate longer and more reflective tool-use trajectories. Our approach substantially extends tool-use chains and improves answer accuracy. Using a single 14B model, we achieve state-of-the-art results on HotpotQA and SimpleQA, with accuracies of 72.3% and 90.0%, respectively, and demonstrate strong generalization to out-of-distribution datasets. The code is available at https://github.com/99hgz/WebSeer . 1 Introduction Large language models (LLMs) have demonstrated remarkable performance across a wide range of natural language processing tasks, including question answering, summarization, and dialogue generation ( Hendrycks et al. 2021 ; Rein et al. 2024 ) . However, relying solely on the parametric knowledge of language models poses fundamental limitations: it is static, often outdated, and prone to hallucinations ( Sardana 2025 ) . To overcome these challenges, retrieval-augmented generation (RAG) ( Lewis et al. 2020 ) approaches have been developed to enable models to access and retrieve external documents dynamically. With the continuous advancement of model capabilities, agentic RAG ( Trivedi et al. 2023 ; Li et al. 2025 ; Jin et al. 2025 ) has emerged as a powerful paradigm. This design empowers models to follow more complex reasoning trajectories. Unlike traditional RAG systems ( Asai et al. 2023 ; Trivedi et al. 2023 ; Yu et al. 2024 ) , agentic RAG can freely browse vast knowledge sources available on the internet and leverage tools such as code execution to extend their skills, enabling them to tackle a more diverse range of tasks. While agentic RAG greatly extends the scope of tool use and demonstrates strong potential, existing approaches exhibits several notable limitations in practice. In particular, when faced with complex or open-domain tasks, current systems often struggle to maintain coherent reasoning chains and robust retrieval. Errors introduced at intermediate steps can easily accumulate, while the lack of effective coordination across components makes it difficult to achieve reliable end-to-end performance. Thus, despite its promise, agentic RAG still confronts a set of fundamental challenges that must be addressed. Figure 1: Comparasion between different search agent: Search-r1 ( Jin et al. 2025 ) faces shallow search and error buildup, while WebSeer trained using our self-reflection paradigm significantly improve tool calls and maintain rigorous reasoning. 1) Insufficient Search Calls. The most common issue when it comes to model invocation tools to solve problems is Insufficient API Calls ( Kokane et al. 2025 ) , which is also often seen in the RAG scenario, as shown in the Appendix B . This may be due to models being biased toward synthesizing the currently available information into a plausible answer rather than actively seeking new or complementary knowledge. Existing work typically exhibits short tool-use chains ( Jin et al. 2025 ; Song et al. 2025 ) , suggesting that models may fall into suboptimal behaviors—prematurely producing an answer instead of verifying its correctness or exploring further evidence. 2) Lack of Spontaneous Self-Reflection Mechanisms. Current search agents lack spontaneous reflection steps in RAG scenario: models neither actively cross-verify information nor autonomously rewrite queries or backtrack retrievals when uncertain. As a result, if initial retrieval is incomplete, the generation phase expands answers based on flawed or partial context, amplifying early errors. 3) Neglect of Real-World Web Scenarios. Most existing work focuses on retrieval from local vector databases, with limited attention to more complex and open-ended web agent scenarios. In this paper, we introduce Webseer, a novel search agent designed to tackle complex real-world multi-hop question answering tasks. Different from prior approaches ( Jin et al. 2025 ; Zheng et al. 2025 ) , WebSeer explicitly encourages deeper exploration and integrates a build-in self-reflection mechanism, enabling the model to backtrack, reformulate queries, and iteratively refine its reasoning process. As shown in Figure 1 , previous agentic RAG primarily rely on short tool-use chains and often terminate once a superficially plausible answer is formed. In contrast, WebSeer actively prolongs the search trajectory and incorporates reflection steps, and revises its queries when uncertainty is detected. This design enables Webseer to gather more comprehensive evidence, mitigate the accumulation of errors, and improve robustness in open-domian multi-hop reasoning. Specifically, we design a two-stage training framework that unifies cold start and reinforcement learning within the self-reflection paradigm. A central component of this framework is Self-Reflective Reinforcement Learning (SRRL), which leverages answer correctness signals during multi-turn interactions to more effectively encourage reflective behavior. To ground the model in realistic web scenarios, we equip it with three complementary tools: a web search API for external knowledge acquisition, a webpage reader for lightweight comprehension of web content, and a code executor for precise computation. To support this framework, we construct a high-quality dataset of long-horizon reasoning trajectories through rejection sampling. These trajectories contain multiple rounds of answer refinement and substantially longer tool-use chains compared to conventional dialogue datasets. This training framework significantly increases the average length of tool invocation chains while maintaining rigorous reasoning quality. Compared to prior work ( Zheng et al. 2025 ) , all decisions and tool interactions are handled by a single model, eliminating the need for auxiliary agent controllers or stronger backbone models. Overall, our contributions are three-fold: • We propose the first unified two-stage training framework with a self-reflection paradigm, enabling a more intelligent search agent that improves search depth, breadth, and accuracy. • We propose a novel SFT data synthesis method that encourages proactive tool invocation, resulting in significantly longer and more complex tool-use chains; • Through extensive experiments, we demonstrate notable improvements in answer accuracy, achieving SOTA on HotpotQA with a 72.3% accuracy and on simpleQA with a 90.0% accuracy. Our proposed training framework shows explicit self-verification behaviors and strong generalization to OOD datasets. 2 WebSeer We introduce WebSeer, a search agent for multi-hop question answering in real-world web settings. The model is trained with our proposed unified two-stage framework based on self-reflection as shown in Fig 2 . To obtain diverse high-quality data, we propose multi-turn rejection sampling for collecting positive trajectories used in supervised fine-tuning, enabling the model to learn reflective reasoning patterns. Building on this, we develop Self-Reflective Reinforcement Learning (SRRL), which incorporates feedback and preserves reflective context during RL, guided by effective reward design to achieve robust and optimal performance. Figure 2: Overview of our two-stage training framework. In the first stage, we construct self-reflective, tool-augmented reasoning trajectories. In the second stage, we apply SRRL, allowing the model to iteratively refine and resubmit answers with F 1 -based feedback. 2.1 Task Formulation For a given problem, our objective is to construct