InfoFlow: Reinforcing Search Agent Via Reward Density Optimization
Kun Luo 1,2 Hongjin Qian 1 Zheng Liu 1 Ziyi Xia 1 Shitao Xiao 1 Siqi Bao 3 Jun Zhao 2 Kang Liu 2 1 Beijing Academy of Artificial Intelligence 2 Institute of Automation, Chinese Academy of Sciences 3 Baidu Inc. {luokun695, chienqhj, ziyixia85, zhengliu1026}@gmail.com [email protected] · 2025
Read InfoFlow: Reinforcing Search Agent Via Reward Density Optimization by Kun Luo 1,2 Hongjin Qian 1 Zheng Liu 1 Ziyi Xia 1 Shitao Xiao 1 Siqi Bao 3 Jun Zhao 2 Kang Liu 2 1 Beijing Academy of Artificial Intelligence 2 Institute of Automation, Chinese Academy of Sciences 3 Baidu Inc. {luokun695, chienqhj, ziyixia85, zhengliu1026}@gmail.com [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-26575.txt.
Introduction
Kun Luo, with seven co-authors, strengthens AI agents that search for information. Such agents are often trained with rewards only at the end of a long search, so good learning signals are rare and costly to earn. The team names this the reward density problem: how much reward an agent gets for each unit of effort spent exploring.
Luo and the team present InfoFlow, which attacks the problem from three sides. It splits long tasks into subproblems with their own rewards. It feeds corrective hints into stalled runs. And it uses a second agent to compress the search history, cutting the cost the main agent feels. On several search benchmarks, InfoFlow beat strong baselines by clear margins. It also let lightweight models match the performance of advanced proprietary ones.
Catalog record
| CREATOR | Kun Luo 1,2 Hongjin Qian 1 Zheng Liu 1 Ziyi Xia 1 Shitao Xiao 1 Siqi Bao 3 Jun Zhao 2 Kang Liu 2 1 Beijing Academy of Artificial Intelligence 2 Institute of Automation, Chinese Academy of Sciences 3 Baidu Inc. {luokun695, chienqhj, ziyixia85, zhengliu1026}@gmail.com [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-26575
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
InfoFlow: Reinforcing Search Agent Via Reward Density Optimization Kun Luo 1,2 Hongjin Qian 1 Zheng Liu 1 Ziyi Xia 1 Shitao Xiao 1 Siqi Bao 3 Jun Zhao 2 Kang Liu 2 1 Beijing Academy of Artificial Intelligence 2 Institute of Automation, Chinese Academy of Sciences 3 Baidu Inc. {luokun695, chienqhj, ziyixia85, zhengliu1026}@gmail.com [email protected] Abstract Reinforcement Learning with Verifiable Rewards (RLVR) is a promising approach for enhancing agentic deep search. However, its application is often hindered by low Reward Density in deep search scenarios, where agents expend significant exploratory costs for infrequent and often null final rewards. In this paper, we formalize this challenge as the Reward Density Optimization problem, which aims to improve the reward obtained per unit of exploration cost. This paper introduce InfoFlow , a systematic framework that tackles this problem from three aspects. 1) Subproblem decomposition : breaking down long-range tasks to assign process rewards, thereby providing denser learning signals. 2) Failure-guided hints : injecting corrective guidance into stalled trajectories to increase the probability of successful outcomes. 3) Dual-agent refinement : employing a dual-agent architecture to offload the cognitive burden of deep exploration. A refiner agent synthesizes the search history, which effectively compresses the researcher’s perceived trajectory, thereby reducing exploration cost and increasing the overall reward density. We evaluate InfoFlow on multiple agentic search benchmarks, where it significantly outperforms strong baselines, enabling lightweight LLMs to achieve performance comparable to advanced proprietary LLMs. 1 Introduction Large language models (LLMs) have become essential tools for information seeking in daily life ( Zhao et al. 2023 ; Gao et al. 2023 ) . As their applications expand, users increasingly expect LLMs to handle not only factual queries but also complex, multi-step tasks requiring knowledge discovery and synthesis. However, because an LLM’s internal knowledge is limited and quickly outdated, relying solely on parametric memory is insufficient for knowledge-intensive tasks ( Vu et al. 2023 ) . Addressing such challenges requires integrating external knowledge sources and moving beyond surface-level retrieval toward deeper reasoning and information synthesis ( Shi et al. 2023 ) . Most existing approaches follow the retrieval-augmented generation (RAG) paradigm ( Gao et al. 2023 ) , which treats the input as a query and retrieves relevant documents for generation. While effective for factual questions, RAG struggles with hierarchical or implicit information needs ( Asai et al. 2023 ; Qian et al. 2025 ) . Extensions such as query rewriting, iterative retrieval, and self-refinement ( Ma et al. 2023 ; Jiang et al. 2023 ; Madaan et al. 2023 ) improve flexibility but remain bound to a pre-inference design that retrieves information before reasoning begins, limiting adaptability in dynamic, multi-step tasks. Inspired by reasoning-centric models ( OpenAI 2024 ; DeepSeek-AI 2025 ) , recent studies adopt the search-integrated reasoning (SIR) paradigm ( Yao et al. 2023 ; Chen et al. 2025a ; Xue et al. 2025 ; Huang et al. 2025 ) , which interleaves reasoning and search to adaptively incorporate external knowledge at each step ( Li et al. 2025c ; Jin et al. 2025b ; Li et al. 2025b ) . However, current LLMs lack native mechanisms to invoke external search tools. Early SIR implementations relied on manually crafted prompts and exhibited limited generalization ( Li et al. 2025a ) . To overcome this, Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as an effective approach for training LLMs to conduct agentic deep search. RLVR enables models to learn search-integrated reasoning policies via trajectory rollouts and final reward-driven optimization ( Jin et al. 2025b ; Jin et al. 2025a ; Qian & Liu 2025 ) . Despite its promise, RLVR for deep search suffers from low reward density , which we define as total reward per unit exploration cost (i.e., per trajectory length). Deep search tasks typically require multiple turns of reasoning-searching exploration before producing a final answer. As trajectory length increases , success rates decline rapidly, since a single reasoning error can accumulate to invalidate the entire trajectory. Moreover, recent work highlights that training robust search agents requires more complex, reasoning-intensive tasks ( Xia et al. 2025 ; Tao et al. 2025 ; Bae et al. 2025 ; Yan et al. 2025 ) . However, our preliminary experiments (Fig. 2 ) show that on difficult tasks successful rollouts become rare (often less than 10% of initial accuracy), further reducing reward density and increasing computational inefficiency. Figure 1: The framework of InfoFlow and example of DSQA task. Researcher agent focuses on reasoning and planning, refiner agent synthesizes massive searched content into condensed info. To address these issues, we formulate Reward Density Optimization and propose InfoFlow , a reinforcement learning framework that improves reward accessibility and stabilizes learning in search-integrated reasoning. InfoFlow increases reward density and learning efficiency via three core components: (1) Sub-goal Scaffolding. To make deep search more tractable for agents with limited initial capabilities, InfoFlow decomposes complex search queries into sub-goals and awards intermediate rewards for solving them. Deep search tasks naturally exhibit hierarchical structure: reaching the final answer typically requires identifying intermediate key facts or anchor entities. Rather than assigning rewards only for full task success, InfoFlow grants partial rewards for resolved sub-goals, providing denser feedback for policy updates. This scaffolding yields a denser learning signal and mitigates the sparsity of final rewards. (2) Pathfinding Hints. To guide agents toward full solutions, InfoFlow incorporates explicit guidance during RL exploration in the form of pathfinding hints. We employ an expert model (DeepSeek-V3.1 ( DeepSeek-AI 2024 ) ) to enrich training data (§ A.1 ) by generating search queries that guide the agent toward reaching key sub-goals. When the agent struggles to reach final answers within a predefined turns during on-policy rollouts, InfoFlow inserts guiding queries into the next turn to suggest more informative search directions. These pathfinding hints make intermediate key facts and anchor entities easier to discover, increasing sub-goal rewards and the likelihood of a correct final answer. They also help the agent learn improved search strategies via learning from expert demonstrations. (3) Trajectory Refinement. To reduce the cognitive burden associated with long trajectories, InfoFlow adopts a dual-agent design for deep search. A research agent performs reasoning and search, while a refiner agent condenses retrieved information into concise, structured summaries that are fed back to the research agent. This collaboration improves efficiency and accuracy: we observe up to 59.5% higher initial rewards, 16.4% reduced inference time, and 44.8% shorter trajectories (§ 3.1 ), substantially increasing reward density. Together, these techniques enable InfoFlow to overcome the reward sparsity bottleneck in RL training, making complex information-seeking tasks more tractable for agents while fostering deeper reasoning and evidence synthesis. We evaluate InfoFlow on a suite of knowledge-intensive agentic search benchmarks as well as the challenging complex benchmark BrowseComp-Plus ( Chen et al. 2025b ) . Experimental results demonstrate that our method consistently outperforms strong baselines. Notably, on BrowseComp-Plus, our optimized small-scale model achieves performance comparable to much larger LLMs. Our main contributions are summarized as follows: (1) We propose InfoFlow, a dual-agent framework for agentic deep search, where a researcher agent is responsible for central reasoning and planning, while a refiner agent synthesizes retrieved evidence into coherent knowledge. (2) We introduce a well-tailored reward density optimization strategy,