Supervising the search process produces reliable and generalizable information-seeking agents

Guangzhi Xiong, Qiao Jin, Xiao Wang, Yin Fang, Haolin Liu, Yifan Yang, Fangyuan Chen, Zhixing Song, Dengyu Wang, Minjia Zhang, Zhiyong Lu, Aidong Zhang · 2025

Read Supervising the search process produces reliable and generalizable information-seeking agents by Guangzhi Xiong, Qiao Jin, Xiao Wang, Yin Fang, Haolin Liu, Yifan Yang, Fangyuan Chen, Zhixing Song, Dengyu Wang, Minjia Zhang, Zhiyong Lu, Aidong Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-13957.txt.

Introduction

Xiong and co-authors argue in 2025 that search agents should be graded on their search steps, not just their final answers. Language models now hunt across sources and build replies step by step. When only the last answer earns a reward, agents learn to cheat or lean on stored facts, and fail on new topics.

Their test frame grades each search step and rewards clear thought about what was found. A new agent built on that rule scored much better on hard, multi step questions. It also held up on topics outside its training. The work points to fairer ways to train search tools.

Catalog record

CREATORGuangzhi Xiong, Qiao Jin, Xiao Wang, Yin Fang, Haolin Liu, Yifan Yang, Fangyuan Chen, Zhixing Song, Dengyu Wang, Minjia Zhang, Zhiyong Lu, Aidong Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-13957

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Supervising the search process produces reliable and generalizable information-seeking agents Guangzhi Xiong, Qiao Jin, Xiao Wang, Yin Fang, Haolin Liu, Yifan Yang, Fangyuan Chen, Zhixing Song, Dengyu Wang, Minjia Zhang, Zhiyong Lu, Aidong Zhang Abstract Large language models (LLMs) are transforming web search by shifting from document ranking to synthesizing answers, and are increasingly deployed as autonomous agentic search systems that iteratively interact with external knowledge sources. Despite this progress, building effective search agents remains challenging because high-quality intermediate search steps are difficult to generate. Previous approaches have primarily relied on outcome supervision, rewarding agents only for producing correct final answers. This often leads to reward hacking and excessive dependence on parametric memory, limiting generalization to out-of-domain tasks. To address these limitations, we introduce RAG-Gym, a framework that shifts supervision from final answers to the search process itself. With RAG-Gym, we systematically investigate architecture design, parameter optimization, and action evaluation, identifying reasoning reflection as a critical capability for search agents. Building on this insight, we propose Re 2 Search++, a process-supervised agent that achieves substantial improvements on multi-hop information-seeking benchmarks, especially in out-of-domain settings. Performance gains are driven primarily by higher-quality search queries rather than answer optimization alone, and the learned search critics transfer across models, including proprietary LLMs. These findings show that supervising the search process produces more reliable and generalizable information-seeking agents. 1 Introduction Large language models (LLMs) are fundamentally transforming how humans seek information from the Internet. We are witnessing a paradigm shift from traditional search engines, which return lists of documents for users to synthesize, to generative answer engines that synthesize knowledge directly [ 44 , 46 , 28 , 11 , 43 ] . This transition is largely powered by Retrieval-Augmented Generation (RAG), which grounds model outputs in external documents to ensure factuality [ 26 , 8 ] . Standard RAG systems typically operate in a single pass, retrieving once and generating immediately. This approach inherently limits their ability to solve complex, multi-hop problems that require gathering and linking multiple pieces of evidence. To overcome this limitation, the field is advancing toward autonomous information-seeking agents [ 27 , 39 , 56 , 3 ] , where LLMs iteratively interact with external knowledge sources to complete search tasks [ 40 , 45 , 18 , 34 , 6 ] . This shift marks a move from simple retrieval to autonomous research, enabling systems to navigate complex information environments in a human-like manner [ 32 , 19 , 1 , 2 , 57 ] . Despite recent advances, building effective autonomous search agents remains a significant challenge. The core obstacle lies in the complexity of decision making required to navigate information environments [ 30 , 15 ] . To resolve multi-hop queries, an agent must break down high-level problems into verifiable intermediate steps, formulate precise queries, and synthesize discrete pieces of evidence. The search space for these intermediate actions is both vast and sparse. Because multi-step retrieval involves a combinatorial explosion of possible action sequences, most paths lead to dead ends or irrelevant information [ 54 , 25 , 31 , 17 ] . Without explicit guidance, agents struggle to learn effective search strategies and often fall into unproductive search loops or rely excessively on parametric knowledge [ 42 , 24 ] . Building on the success of reinforcement learning in enhancing LLMs for mathematical reasoning [ 38 , 10 , 50 ] , current methods attempt to address this challenge through outcome supervision, rewarding agents based on the correctness of their final response [ 41 , 20 , 5 ] . While this approach can align model outputs with ground truth answers, it is susceptible to reward hacking and may incorrectly assign credit for intermediate actions, especially when answers are derived from parametric memory rather than intermediate evidence [ 48 , 29 , 22 ] . As a result, these systems may fail to learn effective search strategies and often struggle to generalize, with performance degrading on out-of-domain tasks where internal knowledge is insufficient (Fig. 5 ), which reduces their practical utility. In this work, we introduce RAG-Gym, a framework that moves the training paradigm from outcome-based signals to process-level supervision. We formulate agentic search as a Markov Decision Process (MDP) in which search queries act as macro-actions. This approach enables fine-grained supervision of intermediate search steps rather than focusing solely on final outcomes. Agents can therefore learn coherent and verifiable search trajectories that logically support their conclusions. Using this framework, we systematically decompose and optimize search agents along three key dimensions: architecture design, parameter optimization, and action evaluation. This allows us to identify the specific design choices that drive optimal performance. We identify reasoning reflection as a powerful but underexplored capability for search agents and introduce a new agent architecture called Re 2 Search (Reason, Reflection, Search) that embodies this approach. At each step, Re 2 Search reasons toward the final answer and explicitly reflects on its reasoning process to uncover knowledge gaps. By integrating Re 2 Search with the optimal strategies discovered through RAG-Gym, we develop Re 2 Search++, a process-supervised agent that achieves strong performance on multi-hop and medical information-seeking benchmarks. Re 2 Search++ surpasses robust outcome-supervised baselines [ 41 , 20 ] , especially in out-of-domain evaluations. Our analysis shows that these improvements arise from a greater ability to generate effective search queries, not just from optimizing answers. This is supported by substantial increases in the retrieval recall of ground-truth documents. We also find that the search heuristics learned through process supervision transfer well, enabling critics trained on open-source trajectories to guide proprietary models effectively. Supervising the evidence-gathering process, rather than focusing only on outcomes, provides a principled path to building reliable and generalizable information-seeking agents. 2 Results 2.1 A framework for supervising the search process We conceptualize agentic search as a high-level Markov Decision Process (MDP) rather than a standard token-by-token generation task. As illustrated in Fig. 1 a, the agent functions as a policy interacting with an information-retrieval environment. At each step t , the state s_{t} comprises the original question, the accumulated reasoning history, and the set of documents retrieved up to that point. The agent selects a macro-action a_{t} by either issuing a targeted search query to gather additional evidence or terminating the process by generating a final answer. This MDP formulation is crucial because it exposes intermediate reasoning steps to direct optimization. In contrast, standard approaches rely on outcome-only supervision (Fig. 1 b), which suffers from a severe credit assignment problem. A correct final answer may result from unsupported or hallucinated knowledge and can incorrectly reinforce a flawed search trajectory. Conversely, a rigorous and necessary search step might be unfairly penalized if the subsequent answer generation is unsuccessful. Building on the MDP formulation, we introduce RAG-Gym, a framework that decomposes the agentic design space into three coupled dimensions that enable process-level supervision (Fig. 1 c). The first dimension is architecture design, which defines the agent’s internal cognitive flow and verification loops. The second is parameter optimization, which trains the actor using supervised, preference, or reinforcement learning. The third is action evaluation, which trains an external critic to guide decision-making at inference. This modularity allows us to isolate

Read the complete text