OffSeeker : Online Reinforcement Learning Is Not All You Need for Deep Research Agents
Yuhang Zhou 1,2, Kai Zheng 2,*, Qiguang Chen 1, Mengkang Hu 3, Qingfeng Sun 2, Can Xu 2, † \dagger Jingjing Chen 1, † \dagger · 2026
Read OffSeeker : Online Reinforcement Learning Is Not All You Need for Deep Research Agents by Yuhang Zhou 1,2, Kai Zheng 2,*, Qiguang Chen 1, Mengkang Hu 3, Qingfeng Sun 2, Can Xu 2, † \dagger Jingjing Chen 1, † \dagger (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-18467.txt.
Description
A 2026 arXiv research paper by Yuhang Zhou 1,2, Kai Zheng 2,*, Qiguang Chen 1, Mengkang Hu 3, Qingfeng Sun 2, Can Xu 2, † \dagger Jingjing Chen 1, † \dagger. 1] Fudan University 2] Tencent Hunyuan 3] The University of Hong Kong \correspondence [email protected], {kevinezheng,leocaxu}@tencent.com, [email protected] \checkdata [Website] https://github.com/Ralph-Zhou/OffSeeker † † footnotetext: ∗ Project Lead. † Corresponding authors.
Catalog record
| CREATOR | Yuhang Zhou 1,2, Kai Zheng 2,*, Qiguang Chen 1, Mengkang Hu 3, Qingfeng Sun 2, Can Xu 2, † \dagger Jingjing Chen 1, † \dagger |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-18467
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
OffSeeker : Online Reinforcement Learning Is Not All You Need for Deep Research Agents Yuhang Zhou 1,2, Kai Zheng 2,*, Qiguang Chen 1, Mengkang Hu 3, Qingfeng Sun 2, Can Xu 2, † \dagger Jingjing Chen 1, † \dagger Abstract 1] Fudan University 2] Tencent Hunyuan 3] The University of Hong Kong \correspondence [email protected], {kevinezheng,leocaxu}@tencent.com, [email protected] \checkdata [Website] https://github.com/Ralph-Zhou/OffSeeker † † footnotetext: ∗ Project Lead. † Corresponding authors. 1 Introduction Large language model (LLM)-based agents have been widely applied to complex tasks involving extended reasoning, multi-step decision-making, and tool use [ 5 , 9 , 24 , 4 , 2 , 8 ] . Among them, deep research agents specialize in long-horizon web information gathering, iterative hypothesis refinement, and structured synthesis, and have achieved strong performance on challenging benchmarks such as GAIA, BrowseComp, Xbench, and HLE [ 15 , 27 , 1 , 17 ] . Despite this progress, developing deep research agents remains difficult in practice due to (1) inefficient open-source data synthesis frameworks, (2) limited availability of high-quality training trajectories, and (3) the high cost of online reinforcement learning, which together hinder scalability, accessibility, and reproducibility in academic research. Specifically, as summarized in Table 1 , most systems employ online RL with live web interactions. This causes instability in non-stationary environments and incurs high API costs (over $350 for 50 GRPO steps), undermining research priorities such as cost control, reproducibility, and accessibility, and confining large-scale experiments to well-resourced industry labs. In contrast, offline training is constrained by inadequate data synthesis frameworks and datasets. A key bottleneck lies in obtaining diverse seed entities, as most methods depend on well-cleaned and well-structured Wikipedia resources [ 13 , 6 , 11 , 14 ] , which demand heavy human effort for preprocessing and yield limited timeliness and diversity. Constrained by these frameworks, high-quality trajectories and preference data remain scarce. Existing deep-research datasets lack the complexity and diversity required for advanced reasoning [ 9 ] . Table 1 : Comparison among recent deep research works. “Training Regime” indicates whether model training follows an online (“On”) or offline (“Off”) procedure. “API Cost” denotes the estimated monetary cost (USD) arising from additional search API calls during the complete training pipeline due to interactions with external environments. More details about our estimation method can be found in Appendix A.1 . Current Works Open Source Components Training Regime API Cost Data Synthesis Framework Training Script Model Weight Deep Search QA Data High-Qulity Trajectories Kimi-K2 [ 24 ] ✗ ✗ ✓ - - Off+On - WebDancer [ 28 ] ✗ ✗ ✓ 200 200 Off+On - WebSailor [ 11 ] ✗ ✗ ✓ 20 - Off+On - WebShaper [ 23 ] ✗ ✗ ✓ 500 - Off 0 WebExplorer [ 13 ] ✗ ✗ ✓ 100 - Off+On > $2,918 Tongyi DeepResearch [ 26 ] ✗ ✗ ✓ - - Off+On - DeepDive [ 14 ] ✓ ✗ ✗ 3.2k 858 Off+On > $4,300 Asearcher [ 6 ] ✓ ✓ ✓ 35k - On > $9,216 Ours ✓ ✓ ✓ 66k 33k Off 0 To solve this, our work systematically addresses the three key challenges described above by providing a comprehensive, fully open-source deep research agent training solution. First, we introduce DeepForge , an end-to-end pipeline for synthesizing deep research tasks through real-world web interaction. It uses LLM-based prompting and iterative expansion to construct large collections of initial entities and URLs, together with entity graph. This lightweight design eliminates the need for heavy preprocessing and enables the synthesis of dynamic and diverse tasks beyond static resources such as Wikipedia. Second, to address the scarcity of academic resources, we release the first comprehensive large-scale deep research dataset to the public, including 66k challenging QA pairs generated by DeepForge , 33k SFT trajectories, and 21k DPO preference pairs. This dataset substantially lowers the entry barrier for research in this area. Third, we demonstrate that online reinforcement learning is not all you need to develop enterprise-level offline-trained deep research agents. We present OffSeeker , an 8B-parameter deep research agent based on Qwen3-8B [ 30 ] , trained entirely using offline supervised fine-tuning and DPO. Experimental results show that our offline approach matches or even exceeds the performance of prominent online RL systems, for example, 26.6 on BrowseComp-zh, compared with 25.5 for WebSailor-32B, while significantly reducing training costs and providing better reproducibility. Our contributions are as follows: • We open-source DeepForge , an end-to-end framework for synthesizing large-scale and diverse deep research tasks that require extensive multi-step reasoning and exploration. The framework is ready to use out of the box and does not require heavy preprocessing. • We release the first fully open deep-research resource suite, including the full codebase and dataset: 66k challenging deep research tasks, 33k high-quality SFT trajectories, and 21k DPO preference data, substantially lowering barriers to academic research. • We release our offline-trained 8B model, OffSeeker , via a fully offline training recipe. It achieves strong deep-research capabilities without online RL, reducing API costs to near zero. Experiments show OffSeeker ranks top-tier among comparable-scale agents and rivals 30B systems trained with online RL. 2 DeepForge : Large Scale Deep-Research Data Synthesis Figure 1 : Overview of our DeepForge data synthesis pipeline. DeepForge comprises two main stages: (a) Scalable Entity Expansion and (b) Complex Question Generation. After synthesizing complex deep search tasks, we further deploy an agent framework to collect high-quality trajectories. Existing open-source data synthesis frameworks require extensive preprocessing, such as downloading full Wikipedia dumps or querying the Wikipedia SPARQL API [ 13 , 6 , 14 , 23 , 11 ] . These heavy requirements increase implementation complexity and constrain dataset diversity. Moreover, frameworks from ASearcher and DeepDive rely primarily on fully offline Wikipedia datasets, which cover only a small fraction of real-world web sources. Consequently, these approaches cannot generate dynamic, up-to-date high-quality data. To address these limitations, we introduce DeepForge , a simple yet highly effective pipeline that synthesizes large-scale deep research tasks and is ready to use immediately. We then construct DeepForge-QA , a 66k challenging deep search dataset for model training. 2.1 Scalable Entity Expansion In contrast to existing approaches that rely on extensive preprocessing infrastructures, DeepForge initiates the process with a lightweight and scalable entity expansion stage. As illustrated in Figure 1 , we first prompt an LLM to generate a broad pool of initial entities, typically simple nouns that collectively span a wide range of domains. We then iteratively refine this pool into a comprehensive seed entity set through a two-step process: (1) For each initial entity, we query a web search API to collect a set of relevant URLs, thereby constructing a large-scale seed URL corpus; (2) For each seed URL, we extract clean textual content using html2text , and prompt an LLM to identify additional entities mentioned within the retrieved text. To ensure that the generated questions encourage genuine research, the LLM is instructed to prioritize the extraction of long-tail and low-frequency entities. Following deduplication and noise filtering, this procedure results in a large, diverse, and high-quality entity pool suitable for constructing multi-hop knowledge graphs. 2.2 Complex Question Generation Given the expanded entity set, DeepForge constructs an entity-centric knowledge graph through multi-step traversal. Starting from each seed entity, a