Negative Advantage Is a Double-Edged Sword: Calibrating Advantage in GRPO for Deep Search

Jiayi Wu, Ruobing Xie, Zeqian Huang, Lei Jiang, Can Xu, Kangyang Luo, Ming Gao, Xiang Li · 2026

Read Negative Advantage Is a Double-Edged Sword: Calibrating Advantage in GRPO for Deep Search by Jiayi Wu, Ruobing Xie, Zeqian Huang, Lei Jiang, Can Xu, Kangyang Luo, Ming Gao, Xiang Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-18235.txt.

Introduction

Wu and eight colleagues fix a flaw in GRPO training for search agents that answer questions over several search rounds. Correct middle steps get punished when the final answer is wrong, and training often turns unstable or collapses. Their CalibAdv method scales down outsized negative advantages using middle-step correctness, then rebalances positive and negative advantages. Tests across three models and seven benchmarks show better performance and steadier training, with a lightweight design built from standard rollout signals.

Catalog record

CREATORJiayi Wu, Ruobing Xie, Zeqian Huang, Lei Jiang, Can Xu, Kangyang Luo, Ming Gao, Xiang Li
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44580
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.18235

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Negative Advantage Is a Double-Edged Sword: Calibrating Advantage in GRPO for Deep Search Jiayi Wu, Ruobing Xie, Zeqian Huang, Lei Jiang, Can Xu, Kangyang Luo, Ming Gao, Xiang Li Abstract Deep search agents can autonomously initiate multi-turn interactions with search engines, thereby exhibiting strong question-answering capabilities. Such performance critically relies on Group Relative Policy Optimization (GRPO) as its core training algorithm. However, GRPO still faces several challenges in deep search settings. First, there exists a substantial mismatch between the correctness of intermediate steps and the reward signal, causing numerous correct intermediate steps to be incorrectly penalized when the final answer is wrong. Second, training is highly unstable, often resulting in degradation of natural language ability or even catastrophic training collapse. Our analysis attributes these issues to coarse-grained advantage assignment and an imbalance between positive and negative advantages. To address these problems, we propose CalibAdv, an advantage calibration method specifically designed for deep search tasks. Specifically, CalibAdv leverages the correctness of intermediate steps to downscale excessive negative advantages at a fine-grained level. It then rebalances positive and negative advantages in the answer component. Extensive experiments across three models and seven benchmarks demonstrate that CalibAdv improves both model performance and training stability. Our code is available at https://github.com/wujwyi/CalibAdv . [email protected] 1 Introduction Figure 1 : CalibAdv improves model performance and stabilizes training by scaling advantages. Deep search agents demonstrate impressive question-answering capabilities through iterative interactions with search engines ( 11 ; 3 ; 13 ) . Compared with standard retrieval-augmented generation (RAG), deep search agents can flexibly reformulate search queries based on contextual information and adaptively determine the number of search iterations, enabling more effective resolution of multi-hop question-answering tasks. The success of deep search agents critically relies on Reinforcement Learning with Verifiable Rewards (RLVR) such as GRPO ( 10 ) . Compared to conventional supervised fine-tuning (SFT), RLVR does not require constructing complete ground-truth answers in the training data. Instead, it provides supervision by directly verifying the correctness of the final answers in rollouts sampled from the policy model. Furthermore, rather than relying mainly on positive training examples as SFT does, RLVR can provide explicit penalty signals for undesired outputs or behaviors. However, despite being one of the most widely used RLVR methods, GRPO-like training methods still face several challenges when applied to training deep search agents. First, there exists a non-negligible gap between process correctness and the reward signal. Deep search agents interact with search engines over multiple iterations, yielding responses that consist of multiple steps. When determining whether output tokens should be encouraged or penalized, GRPO treats the intermediate steps and the outcome uniformly. This can lead to a substantial mismatch between the correctness of intermediate reasoning steps and the actually assigned advantage values, which may affect the performance of the search agent. Second, training instability occurs with high probability . ( 3 ; 13 ; 18 ; 1 ) As training progresses, the frequency of incoherent or nonsensical outputs gradually increases, eventually evolving into a complete loss of natural language generation capability. At this stage, the agent fails to initiate searches or answer questions correctly, resulting in a catastrophic training collapse that severely undermines the training process. We observe that the negative signals , as a key mechanism in reinforcement learning, are closely related to the issues described above. Regarding process-level reward signals, correct intermediate steps are often penalized alongside the incorrect final answers. Our analysis indicates that this misattribution of penalties to correct steps occurs frequently. Therefore, advantages should not be assigned coarsely to all tokens in a rollout. They should be fine-grainedly adjusted based on the correctness of intermediate steps. In terms of training collapse, we observe that collapses are often accompanied by the gradual domination of negative advantages. As training progresses, the imbalance between positive and negative advantages leads to increased model entropy and perplexity, resulting in the complete loss of natural language capability. These observations inspire us to think: When training deep search agents with GRPO-like RL, although the penalty signal is crucial for success, negative advantages must be carefully calibrated based on the correctness of intermediate steps and the balance between positive and negative advantages. To this end, we propose CalibAdv , a GRPO advantage calibration method tailored for deep search tasks. As illustrated in Figure 1 , CalibAdv improves model performance and stabilizes training by scaling advantages at each interaction turn. CalibAdv adjusts the advantage from three aspects. First, for intermediate steps, we mitigate excessive penalties by attenuating the magnitude of incorrectly assigned negative signals. Specifically, by comparing the documents retrieved by incorrect and correct rollouts within the same group, we identify intermediate steps that are correct within incorrect rollouts and downscale their negative advantages accordingly. Second, for the final answer step, we rebalance positive and negative advantages. This stage is particularly prone to imbalance, and scaling the positive advantage helps mitigate the loss of natural language generation capability. Finally, <think> is prepended to the prompt without being considered a shared prefix across rollouts, and no advantage is assigned to it, thereby preventing format collapse caused by advantage imbalance. We conduct extensive experiments on three models and seven benchmarks, demonstrating that CalibAdv substantially improves both model performance and training stability. On average, CalibAdv yields a 11.80% relative improvement in F1 score, while avoiding degradation of the model’s natural language capabilities and the catastrophic training collapse. Our contributions can be summarized as follows: • We mitigate the mismatch between intermediate step correctness and reward signals in GRPO-like training for multi-turn interactive tasks by introducing fine-grained advantage calibration at intermediate steps. • We identify the root cause of training collapse in GRPO for deep search tasks and address this long-standing issue in the research community through rebalancing positive and negative advantages. • Our experiments show that calibrating GRPO advantages is necessary, highlighting that while negative advantages provide crucial supervisory signals, they also carry the risk of performance degradation and training instability. 2 Preliminaries Figure 2 : The proportion of erroneously penalized steps Figure 3 : Training signals associated with language collapse. In this section, we briefly introduce the pipeline of the deep search task, the reward design and implementation details commonly used when training deep search agents with GRPO, and the challenges that arise when applying GRPO to deep search tasks. 2.1 The Deep Search Task Pipeline A deep-search agent answers questions via multi-turn interaction with a retriever ( 3 ; 12 ; 13 ) . Given an input question or retrieved documents, the agent performs a reasoning process delimited by the <think> and </think> tags. After the reasoning process, if external information is required, the agent generates a retrieval query enclosed within the <search> and </search> tags. The retriever then returns relevant documents based on the generated query, wraps them with <information> and </information> tags, and appends them to the agent’s input context. When sufficient information has been accumulated, the agent outputs the final answer following the reasoning process, enclosed within the <answer> and </answer> tags. Formally, given a question x , the agent \pi generates the response at the i -th turn based on the context from the previous i-1 turns, and then retrieves a document d_{i} or produces the final answer.

Read the complete text