CoSearch: Joint Training of Reasoning and Document Ranking via Reinforcement Learning for Agentic Search
Hansi Zeng, Liam Collins, Bhuvesh Kumar, Neil Shah, Hamed Zamani · 2026
Read CoSearch: Joint Training of Reasoning and Document Ranking via Reinforcement Learning for Agentic Search by Hansi Zeng, Liam Collins, Bhuvesh Kumar, Neil Shah, Hamed Zamani (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-17555.txt.
Introduction
Zeng and four colleagues argue that agentic search stalls because its retrieval system stays frozen while only the reasoning agent learns. Their CoSearch framework trains a multi-step reasoning agent together with a generative document ranker through Group Relative Policy Optimization. A preliminary test put the gap between an oracle retriever and a fixed one at up to 26.8 percent relative F1 across seven question-answering benchmarks. Joint training with a semantic grouping strategy steadies ranker learning across shifting reasoning paths, aiming to close that gap.
Catalog record
| CREATOR | Hansi Zeng, Liam Collins, Bhuvesh Kumar, Neil Shah, Hamed Zamani |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57117 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.17555
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CoSearch: Joint Training of Reasoning and Document Ranking via Reinforcement Learning for Agentic Search Hansi Zeng, Liam Collins, Bhuvesh Kumar, Neil Shah, Hamed Zamani Abstract Agentic search – the task of training agents that iteratively reason, issue queries, and synthesize retrieved information to answer complex questions – has achieved remarkable progress through reinforcement learning (RL). However, existing approaches such as Search-R1 treat the retrieval system as a fixed tool, optimizing only the reasoning agent while the retrieval component remains unchanged. A preliminary experiment reveals that the gap between an oracle and a fixed retrieval system reaches up to +26.8% relative F1 improvement across seven QA benchmarks, suggesting that the retrieval system is a key bottleneck in scaling agentic search performance. Motivated by this finding, we propose CoSearch , a framework that jointly trains a multi-step reasoning agent and a generative document ranking model via Group Relative Policy Optimization (GRPO). To enable effective GRPO training for the ranker—whose inputs vary across reasoning trajectories—we introduce a semantic grouping strategy that clusters sub-queries by token-level similarity, forming valid optimization groups without additional rollouts. We further design a composite reward combining ranking quality signals with trajectory-level outcome feedback, providing the ranker with both immediate and long-term learning signals. Experiments on seven single-hop and multi-hop QA benchmarks demonstrate consistent improvements over strong baselines, with ablation studies validating each design choice. Our results show that joint training of the reasoning agent and retrieval system is both feasible and strongly performant, pointing to a key ingredient for future search agents. Code is available at https://github.com/snap-research/CoSearch 1 Introduction Large language models (LLMs) have demonstrated strong reasoning capabilities, yet they face fundamental challenges when questions require knowledge beyond their parametric training data. Retrieval-augmented generation (RAG) ( 22 ; 53 ) addresses this limitation by allowing models to access external knowledge through retrieval. While early approaches relied on single-turn retrieval followed by answer generation, recent work has shown that multi-step agentic search—where models iteratively reason, generate search queries, and incorporate retrieved information—yields substantially stronger performance on knowledge-intensive tasks ( 44 ; 19 ; 38 ) . In these agentic search systems, an agent interacts with a retrieval system iteratively: the agent reasons about what information is needed, issues a search query, observes the retrieved documents, and repeats this process until sufficient evidence is gathered to produce a final answer. Training such agents through reinforcement learning (RL), where the reward is based on final answer correctness, has proven effective at teaching them when and what to search ( 14 ; 24 ; 54 ) . Despite this progress, a common assumption across these systems is that the retrieval model or search engine remains unchanged. This means that the agent treats the retrieval model as a tool and learns to use it and the tool does not adapt. 7B Main Agent 3B Main Agent Single Multi Avg Single Multi Avg Standard 0.600 0.506 0.546 0.531 0.371 0.440 Oracle Retrieval 0.709 0.564 0.626 0.636 0.499 0.558 Rel. Gain +18.2% +11.5% +14.7% +19.8% +34.5% +26.8% Table 1: Oracle retrieval gap (F1 score). Performance averaged over single-hop (PopQA, NQ, TQA) and multi-hop (HotpotQA, 2Wiki, Musique, Bamboogle) benchmarks. We argue that this assumption introduces a significant bottleneck. Through analysis of RL-trained search agents, we observe that most errors originate not from the reasoning process but from the retrieval component: the agent issues queries that look reasonable, yet the retrieval system fails to return relevant documents, forcing the agent to either retry with reformulated queries or draw incorrect conclusions from noisy information. To quantify this gap, we conduct an oracle retrieval experiment: at each retrieval step, we promote all documents matching the gold (ground-truth) answer to the top positions; for sub-queries where no retrieved document matches the gold answer, the ranking is left unchanged (see Appendix F for details). As shown in Table 1 , this simple intervention consistently improves average F1 by +14.7% for a 7B-parameter agent and by +26.8% for a 3B agent across seven QA benchmarks, demonstrating that the retrieval bottleneck holds broadly regardless of the agent’s reasoning capacity. These findings motivate the following question: can we jointly optimize the retrieval system alongside the multi-turn reasoning agent, so that they learn to complement each other for providing accurate responses? In this work, we propose CoSearch , a reinforcement learning (RL) framework that jointly trains a main reasoning agent and a generative ranker for agentic search tasks. The main agent operates in a standard ReAct-style loop ( 52 ) , generating thoughts and search queries. The generative ranker receives the main agent’s sub-queries along with candidate documents from a fixed first-stage dense retriever, and selects and reranks a subset for the main agent to observe; together, the retriever and ranker form the retrieval system. The main agent and the ranker are trained simultaneously using Group Relative Policy Optimization (GRPO) ( 4 ) , with rewards derived from the correctness of the final answer. Applying GRPO to the ranker introduces two key technical challenges. First, GRPO computes advantages over groups of responses generated from the same input prompt . While this is naturally satisfied for the main agent—where multiple trajectories are sampled from the same user query—the ranker receives different sub-queries across trajectories, preventing direct grouping. We address this through a semantic grouping strategy that clusters sub-queries by token-level F1 similarity, forming valid GRPO groups without additional rollouts. Second, the ranker’s effect on the final answer is indirect: good retrieval may still yield an incorrect answer if reasoning fails, and vice versa. We design a composite reward combining a relevance reward based on a ranking quality metric (Hit@ k ) with a trajectory-level reward reflecting the final answer correctness, providing both immediate feedback on retrieval precision and long-term signal on downstream utility. Both reward components are computed using rule-based metrics, requiring no expensive LLM annotations. Following prior work ( 14 ) , we evaluate CoSearch on seven QA benchmarks spanning single-hop (PopQA, NQ, TriviaQA) and multi-hop (HotpotQA, 2WikiMultiHopQA, Musique, Bamboogle) settings. Using Qwen2.5-7B-Instruct as the backbone for both the main and ranking agents, our method consistently outperforms strong baselines including Search-R1 ( 14 ) and fixed ranker variants. We further demonstrate effectiveness with a 3B main agent paired with a 7B ranker. Our main contributions are as follows: • We identify retrieval quality as a critical bottleneck in RL-trained agentic search systems: an oracle retrieval experiment reveals relative F1 gains of +14.7% for a 7B reasoning agent and +26.8% for a 3B agent, demonstrating a substantial and broadly held performance ceiling imposed by the fixed retrieval system. • We propose CoSearch , a framework that jointly trains a reasoning agent and a generative document ranker via RL. To enable this, we introduce a semantic grouping strategy for applying GRPO to the ranker despite varying inputs across trajectories, and a composite reward that combines ranking quality with trajectory-level answer correctness. • Experiments on seven QA benchmarks show consistent improvements over strong baselines, achieving +6.6% relative F1 over Search-R1 with a 7B agent and +10.8% with a 3B agent. Our results show that jointly training the retrieval system and reasoning agent is not only feasible but strongly performant, and is likely a key ingredient for future search agents.