Reinforcement Learning Foundations for Deep Research Systems: A Survey

Wenjun Li, Zhi Chen, Jingru Lin, Hannan Cao, Wei Han, Sheng Liang, Zhi Zhang · 2025

Read Reinforcement Learning Foundations for Deep Research Systems: A Survey by Wenjun Li, Zhi Chen, Jingru Lin, Hannan Cao, Wei Han, Sheng Liang, Zhi Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-06733.txt.

Introduction

Training AI research agents with reinforcement learning is surveyed by Wenjun Li and ten co-authors in a 2025 review. Deep research systems chain reasoning, web search, and tool use to solve long tasks, often with planner, coordinator, and executor roles. Training a whole stack at once is impractical, so most work trains one planner with core tools.

Supervised fine-tuning teaches the format but copies flaws and ignores feedback from the environment. Preference methods depend on schemas and human-defined decision points, and they handle long-horizon credit poorly. Reinforcement learning, by contrast, optimizes whole trajectories and rewards recovery from mistakes. The survey organizes the field along data curation, learning methods, and training systems, then covers architectures, coordination, and benchmarks. It closes with recurring patterns, infrastructure bottlenecks, and practical guidance for building robust research agents.

Catalog record

CREATORWenjun Li, Zhi Chen, Jingru Lin, Hannan Cao, Wei Han, Sheng Liang, Zhi Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE109 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-06733

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Reinforcement Learning Foundations for Deep Research Systems: A Survey Wenjun Li, Zhi Chen, Jingru Lin, Hannan Cao, Wei Han, Sheng Liang, Zhi Zhang Abstract Deep research systems, agentic AI that solve complex, multi-step tasks by coordinating reasoning, search across the open web and user files, and tool use, are moving toward hierarchical deployments with a Planner, Coordinator, and Executors. In practice, training entire stacks end-to-end remains impractical, so most work trains a single planner connected to core tools such as search, browsing, and code. While SFT imparts protocol fidelity, it suffers from imitation and exposure biases and underuses environment feedback. Preference alignment methods such as DPO are schema and proxy-dependent, off-policy, and weak for long-horizon credit assignment and multi-objective trade-offs. A further limitation of SFT and DPO is their reliance on human defined decision points and subskills through schema design and labeled comparisons. Reinforcement learning aligns with closed-loop, tool-interaction research by optimizing trajectory-level policies, enabling exploration, recovery behaviors, and principled credit assignment, and it reduces dependence on such human priors and rater biases. This survey is, to our knowledge, the first dedicated to the RL foundations of deep research systems. It systematizes work after DeepSeek-R1 along three axes: (i) data synthesis and curation; (ii) RL methods for agentic research covering stability, sample efficiency, long context handling, reward and credit design, multi-objective optimization, and multimodal integration; and (iii) agentic RL training systems and frameworks. We also cover agent architecture and coordination, as well as evaluation and benchmarks, including recent QA, VQA, long-form synthesis, and domain-grounded, tool-interaction tasks. We distill recurring patterns, surface infrastructure bottlenecks, and offer practical guidance for training robust, transparent deep research agents with RL. A curated paper list is available at github.com/wenjunli-0/deepresearch-survey . 1 Introduction The rapid emergence of deep research systems (e.g., OpenAI 2025 ; Google 2025 ; Perplexity Team 2025 ), agentic AI models capable of tackling complex, multi-step information-seeking tasks, marks a significant shift in how AI approaches reasoning, execution, and synthesis. In this survey, we focus on information-seeking use cases, as most existing research and products center on this application. We define deep research systems as agentic AI researchers that autonomously plan and carry out multi-step investigations across the open web and user-provided files, iteratively searching, reading, and reasoning as new evidence appears, and ultimately producing either a concise answer for an objective question or a well-structured, citation-backed report for a subjective open question. A trend in both academia ( Li et al. 2025f ; Jin et al. 2025b ; Wan et al. 2025 ) and industry ( ByteDance & contributors 2025 ; LangChain & contributors 2025 ; MiroMindAI & contributors 2025 ) is to move from monolithic agents to hierarchical agent architectures for deep research. Figure 1 mirrors this architecture: a Planner performs step-by-step decomposition and reflection; a Coordinator handles assignment, delegation, aggregation, and verification; and a pool of Executors (i.e., specialized agents and tools) executes grounded actions over the web and files. This separation of concerns decouples strategic planning from execution details, enabling parallelization, plug-and-play expertise (e.g., swapping in better searchers or code runners and scaling to additional tools), and tighter instrumentation for process logging, credit assignment, and auditability. It also keeps the Planner’s global state clean and coherent over long horizons while the Coordinator and Executors handle delegation and grounded actions. While the hierarchical architecture is attractive for deployment, it is not yet practical to train the entire workflow end-to-end. As a result, most research targets a single model (typically the Planner) wired directly to a small set of fundamental tools (search/browse/code), which collapses rollout length and variance, fits existing RL/SFT/DPO infrastructure, and yields cleaner signals. The training objective is to strengthen long-horizon capabilities in one place (i.e., reasoning, decomposition, tool use, reflection, and synthesis) in an end-to-end manner so that the resulting Planner can later slot into the full hierarchy as a stronger “brain,” while coordination and execution remain modular and swappable. Hence, in this survey, we primarily focus on the training of the planner model and will cover the agent architecture and coordination design in § 5 . Figure 1: Illustration of the hierarchical deep research system architecture. Supervised fine-tuning (SFT; Ouyang et al. 2022 ; Wei et al. 2022 ) is an effective way to initialize deep research agents: it is stable, data-efficient, and good at teaching protocol fidelity (e.g., tool-call schemas, response formats), and basic stepwise reasoning patterns. Because SFT optimizes against gold (x,y) pairs, it excels at imparting local behaviors like query rewriting templates, citation style, argument wrapping, and reduces variance early on. The same properties, however, limit performance on multi-turn research tasks. Reference traces are long, composite, and human-authored; imitation induces imitation bias (copying a particular decomposition) and exposure bias (teacher-forced steps hide compounding errors at inference). SFT also leaves environment feedback underused: it cannot directly learn from tool failures, stochastic retrieval, or non-stationary states (e.g., prices, availability). In short, SFT is valuable scaffolding for competencies and interfaces, but not a vehicle for optimizing end-to-end decision quality. Preference-based methods (e.g., DPO; Rafailov et al. 2023 ) can be pushed beyond single-turn outputs by decomposing agent workflows into labeled steps (e.g., query generation, retrieval selection, synthesis) and learning local preferences at each stage. However, although there are several research works exploring training deep research agents via DPO-based methodologies ( Zhang et al. 2025c ; Zhao et al. 2025a ; Asai et al. 2023 ) , we think several structural mismatches remain in such approahces. First, DPO optimizes textual alternatives rather than state–action returns: pairwise losses are applied to strings conditioned on prior text, without explicit grounding in environment state (tool results, cache, budgets) or action semantics. This makes credit assignment inherently myopic—it judges which snippet is preferable at that step but cannot attribute downstream success/failure to earlier retrieval or tool-use decisions, nor can it trade off depth of search against cost/latency under partial observability. Second, stepwise DPO inherits schema and proxy dependence: one must hand-design the process decomposition and generate preferences (often with heuristics or another LLM), which introduces label noise, and brittleness when the unseen task requires a different decomposition. Third, DPO is largely off-policy and offline: it improves on fixed comparisons but does not explore the closed-loop space of actions and tool outcomes, so it struggles to learn recovery behaviors (e.g., when a query returns junk, a site blocks access, or prices shift) and to adapt to non-stationary environments. Finally, multi-objective desiderata (accuracy, calibration, cost, safety) enter only implicitly through rater preferences; DPO provides no principled mechanism to aggregate vector rewards over long horizons. Given the limitations of SFT/DPO-based approaches, we regard reinforcement learning (RL) as a promising pathway towards training deep research agents in an end-to-end manner. Deep research ultimately demands trajectory-level learning in a closed-loop, tool-rich environment: deciding how to decompose problems, when and how to invoke tools, which evidence to trust, when to stop, and how to trade off accuracy, cost, and latency as the state evolves. RL treats the system as a policy over states and actions, enabling end-to-end improvement from environment signals, credit assignment across multi-step traces, and exploration of alternative strategies for search, tool orchestration, recovery, and synthesis. Motivated by the shift toward RL for training deep research

Read the complete text