Enhancing LLM-based Search Agents via Contribution Weighted Group Relative Policy Optimization

Junzhe Wang 1 ∗† , Zhiheng Xi 1 ∗ , Yajie Yang 1 , Hao Luo 1 , Shihan Dou 1 , Tao Gui 1 , Qi Zhang 1,2,3 † 1 Fudan University 2 Shanghai Artificial Intelligence Laboratory 3 Shanghai Key Laboratory of Intelligent Information Processing [email protected], [email protected] · 2026

Read Enhancing LLM-based Search Agents via Contribution Weighted Group Relative Policy Optimization by Junzhe Wang 1 ∗† , Zhiheng Xi 1 ∗ , Yajie Yang 1 , Hao Luo 1 , Shihan Dou 1 , Tao Gui 1 , Qi Zhang 1,2,3 † 1 Fudan University 2 Shanghai Artificial Intelligence Laboratory 3 Shanghai Key Laboratory of Intelligent Information Processing [email protected], [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-14267.txt.

Introduction

Wang and six colleagues improve search agents with Contribution-Weighted GRPO, which folds process supervision into group relative policy optimization. Rather than optimizing process rewards outright, an LLM judge scores retrieval utility and reasoning correctness each round, and those per-round scores rescale outcome-based advantages along the trajectory. The approach gives fine-grained credit without sacrificing optimization stability, beating standard GRPO by about 5 percent on Qwen3-8B and about 6 percent on Qwen3-1.7B. Winning trajectories show contributions bunched in specific rounds, a clue for future search agent design.

Catalog record

CREATORJunzhe Wang 1 ∗† , Zhiheng Xi 1 ∗ , Yajie Yang 1 , Hao Luo 1 , Shihan Dou 1 , Tao Gui 1 , Qi Zhang 1,2,3 † 1 Fudan University 2 Shanghai Artificial Intelligence Laboratory 3 Shanghai Key Laboratory of Intelligent Information Processing [email protected], [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41124
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.14267

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Enhancing LLM-based Search Agents via Contribution Weighted Group Relative Policy Optimization Junzhe Wang 1 ∗† , Zhiheng Xi 1 ∗ , Yajie Yang 1 , Hao Luo 1 , Shihan Dou 1 , Tao Gui 1 , Qi Zhang 1,2,3 † 1 Fudan University 2 Shanghai Artificial Intelligence Laboratory 3 Shanghai Key Laboratory of Intelligent Information Processing [email protected], [email protected] Abstract Search agents extend Large Language Models (LLMs) beyond static parametric knowledge by enabling access to up-to-date and long-tail information unavailable during pretraining. While reinforcement learning has been widely adopted for training such agents, existing approaches face key limitations: process supervision often suffers from unstable value estimation, whereas outcome supervision struggles with credit assignment due to sparse, trajectory-level rewards. To bridge this gap, we propose Contribution-Weighted GRPO (CW-GRPO), a framework that integrates process supervision into group relative policy optimization. Instead of directly optimizing process rewards, CW-GRPO employs an LLM judge to assess the retrieval utility and reasoning correctness at each search round, producing per-round contribution scores. These scores are used to rescale outcome-based advantages along the trajectory, enabling fine-grained credit assignment without sacrificing optimization stability. Experiments on multiple knowledge-intensive benchmarks show that CW-GRPO outperforms standard GRPO by 5.0% on Qwen3-8B and 6.3% on Qwen3-1.7B, leading to more effective search behaviors. Additional analysis reveals that successful trajectories exhibit concentrated contributions in specific rounds, providing empirical insight into search agent tasks. † † * Equal contribution. † Corresponding authors. † † 1 Our code is available at https://github.com/zsxmwjz/CW-GRPO . Figure 1 : CW-GRPO assesses the quality of each search round within successful trajectories and to concentrate the learning advantage on high-quality search rounds, thereby enabling round-level credit assignment. 1 Introduction Search agents empower large language models (LLMs) to move beyond fixed parametric knowledge by providing access to real-time information and specialized facts absent from pre-training ( Li et al. 2025a ; Zhang et al. 2025a ) . Through iterative retrieval of external evidence and integration into the reasoning process, search agents ground model predictions in verifiable sources and substantially enhance the factual reliability of LLM-based systems ( Zhao et al. 2026 ; Li et al. 2025b ; Zhou et al. 2024 ; Wang et al. 2025a ) . Consequently, they have become a key role in enabling LLMs to perform knowledge-intensive and fact-sensitive tasks ( Glass et al. 2022 ; Sawarkar et al. 2024 ) . While reinforcement learning (RL) offers a significant paradigm for training such agents, current RL approaches suffer from fundamental limitations ( Jin et al. 2025a ) . Depending on the level of supervision, they can be broadly categorized into process supervision and outcome supervision ( Uesato et al. 2022 ) . Process supervision ( Lightman et al. 2023 ; Zhu et al. 2025 ; Wang et al. 2025b ) assigns round-level rewards and is commonly optimized with actor–critic methods such as Proximal Policy Optimization ( Schulman et al. 2017 ) , but learning reliable critics over diverse intermediate states is unstable and frequently leads to biased advantages and brittle training ( Liu et al. 2024 ; Kazemnejad et al. 2025 ) . Outcome supervision, by contrast, relies solely on final answer correctness, which makes it difficult to attribute task success to individual search rounds, leading to a credit assignment problem and obscuring the uneven contributions of intermediate decisions. Recent advances, such as Group Relative Policy Optimization ( Shao et al. 2024 ) , improve training stability and memory efficiency, making outcome-supervised optimization more practical; however, they ( Jin et al. 2025b ; Wang et al. 2025a ) do not alter the sparsity of the reward signal, and the credit assignment issue remains unresolved. To address the aforementioned challenges, we propose C ontribution- W eighted GRPO (CW-GRPO) , which reformulates process supervision as a problem of modulating outcome-derived advantages, rather than directly optimizing process rewards. Specifically, CW-GRPO retains the stability of the standard GRPO by computing outcome advantages through group-relative comparisons. We use an LLM judge to assess each search round based on retrieval utility and reasoning correctness. These metrics are synthesized into a contribution score, based on which serves as a scaling factor to redistribute the outcome advantage across the trajectory. By avoiding direct comparison of process rewards, CW-GRPO incorporates process supervision into the GRPO framework, amplifying learning signals for informative rounds while suppressing those for redundant or unproductive ones, as illustrated in Figure 1 . Empirical results across multiple knowledge-intensive benchmarks demonstrate that CW-GRPO consistently outperforms both outcome-supervised and process-supervised baselines, enabling more effective search behaviors. Specifically, CW-GRPO achieves significant performance gains over standard GRPO, with relative improvements of 5.0\% on Qwen3-8B and 6.3\% on Qwen3-1.7B. The contributions of this work are summarized as follows: • A Reframing of Process Supervision : We introduce a new perspective that treats process supervision for search agents as advantage reallocation guided by round contribution, rather than explicit process reward estimation. • Contribution-Weighted GRPO : We propose a new optimization framework that brings process-level credit assignment into GRPO without requiring unstable value functions. • Empirical Characterization of Search Contribution : We provide empirical evidence showing a structural characteristic in search agent tasks: the contribution to task success is highly concentrated in informative rounds rather than being uniformly distributed. 2 Related Work Agentic Search. Large language models encode knowledge in static parameters, limiting access to up-to-date and long-tail knowledge. Retrieval-augmented generation (RAG) addresses this by enabling models to access external knowledge sources ( Zhang et al. 2026 ; Zhao et al. 2026 ; Li et al. 2025b ; Zhou et al. 2024 ) . Pioneering retrieval-augmented reasoning works such as IRCoT ( Trivedi et al. 2023 ) and FLARE ( Jiang et al. 2023 ) use prompts to guide iterative reasoning and reduce model uncertainty, and Self-RAG ( Asai et al. 2024 ) adopts supervised fine-tuning with reflection tokens for adaptive retrieval. More recently, agentic search enables LLMs to interact with real-world search engines for dynamic information access. WebGPT ( Nakano et al. 2022 ) pioneers this direction using RLHF. Search-o1 ( Li et al. 2025a ) integrates agentic search into large reasoning models via prompting, while Search-R1 ( Jin et al. 2025b ) , R1-Searcher ( Song et al. 2025 ) and MM-Doc-R1 ( Lin et al. 2026 ) adopt RL for end-to-end training with outcome-based rewards. However, outcome supervision assigns uniform credit across search rounds, failing to distinguish pivotal searches from redundant ones. Outcome Supervision and Process Supervision. A key challenge in training search agents lies in credit assignment across multi-round search trajectories. Outcome supervision , widely adopted in RL methods like PPO ( Schulman et al. 2017 ) and GRPO ( Shao et al. 2024 ) , assigns rewards only based on the final answer, treating all immediate rounds equally and ignoring their unequal contributions. Process supervision addresses this via round-level feedback for finer-grained credit assignment. Early approaches to process supervision rely on learned Process Reward Models (PRMs), but often suffer from costly round-level annotation requirements and limited out-of-domain generalizability ( Lightman et al. 2023 ; Huang et al. 2025 ) . To avoid training explicit reward models, recent works increasingly leverage frozen LLMs as external evaluators to assess intermediate rounds. These methods use LLM-based judges to assess reasoning

Read the complete text