IG-Search: Step-Level Information Gain Rewards for Search-Augmented Reasoning

Zihan Liang, Yufei Ma, Ben Chen, Zhipeng Qian, Huangyu Dai, Xuxin Zhang, Chenyi Lei, Wenwu Ou · 2026

Read IG-Search: Step-Level Information Gain Rewards for Search-Augmented Reasoning by Zihan Liang, Yufei Ma, Ben Chen, Zhipeng Qian, Huangyu Dai, Xuxin Zhang, Chenyi Lei, Wenwu Ou (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-15148.txt.

Introduction

Liang and colleagues sharpen search-augmented reasoning with IG-Search, a framework that rewards each search step by information gain. For every step, the gain measures how much retrieved pages raise confidence in the gold answer versus random pages, judging the query itself. That signal flows back to query tokens through per-token advantage modulation in GRPO. Across seven question-answering benchmarks the method beats the strongest trajectory-level baseline by 1.6 points, and it still learns when every sampled trajectory fails.

Catalog record

CREATORZihan Liang, Yufei Ma, Ben Chen, Zhipeng Qian, Huangyu Dai, Xuxin Zhang, Chenyi Lei, Wenwu Ou
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE72109
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.15148

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

IG-Search: Step-Level Information Gain Rewards for Search-Augmented Reasoning Zihan Liang, Yufei Ma, Ben Chen, Zhipeng Qian, Huangyu Dai, Xuxin Zhang, Chenyi Lei, Wenwu Ou Abstract Reinforcement learning has emerged as an effective paradigm for training large language models to perform search-augmented reasoning. However, existing approaches rely on trajectory-level rewards that cannot distinguish precise search queries from vague or redundant ones within a rollout group, and collapse to a near-zero gradient signal whenever every sampled trajectory fails. In this paper, we propose IG-Search , a reinforcement learning framework that introduces a step-level reward based on Information Gain (IG). For each search step, IG measures how much the retrieved documents improve the model’s confidence in the gold answer relative to a counterfactual baseline of random documents, thereby reflecting the effectiveness of the underlying search query. This signal is fed back to the corresponding search-query tokens via per-token advantage modulation in GRPO, enabling fine-grained, step-level credit assignment within a rollout. Unlike prior step-level methods that require either externally annotated intermediate supervision or shared environment states across trajectories, IG-Search derives its signals from the policy’s own generation probabilities, requiring no intermediate annotations beyond standard question-answer pairs. Experiments on seven single-hop and multi-hop QA benchmarks demonstrate that IG-Search achieves an average EM of 0.430 with Qwen2.5-3B, outperforming the strongest trajectory-level baseline (MR-Search) by 1.6 points and the step-level method GiGPO by 0.9 points on average across benchmarks, with particularly pronounced gains on multi-hop reasoning tasks. Despite introducing a dense step-level signal, IG-Search adds only ∼ 6.4 % {\sim}6.4\% to per-step training wall-clock time over the trajectory-level baseline and leaves inference latency unchanged, while still providing a meaningful gradient signal even when every sampled trajectory answers incorrectly. 1 Introduction Large language models (LLMs) have shown impressive abilities in language understanding, planning, and problem solving [ 18 , 31 , 21 ] . Recent advances demonstrate that reinforcement learning (RL) further enhances LLMs’ reasoning capabilities [ 6 , 19 ] , especially in complex tasks such as mathematical problem solving [ 26 ] and code generation [ 5 ] . However, the knowledge encoded in LLMs is inherently bounded by their training corpora, limiting their reasoning performance on tasks that require up-to-date or specialized factual information [ 9 , 25 ] . A common strategy to address this limitation is retrieval-augmented generation (RAG), which equips LLMs with retrieval tools to access external knowledge bases during reasoning [ 14 , 10 , 23 ] . While early RAG approaches rely on supervised fine-tuning (SFT) to teach models when and how to search [ 1 , 28 , 38 ] , recent work draws inspiration from RL-based post-training [ 6 ] and explores RL for retrieval-augmented reasoning, forming a search-during-think paradigm where the model autonomously invokes retrieval tools within its reasoning trajectories [ 11 , 2 , 30 ] . Building on this paradigm, AutoRefine [ 29 ] introduces an explicit knowledge refinement step after each retrieval call and incorporates a retrieval-specific reward alongside the answer correctness reward via Group Relative Policy Optimization (GRPO) [ 26 ] , achieving strong performance on both single-hop and multi-hop question answering scenarios. MR-Search [ 36 ] further introduces cross-episode self-reflection, but assigns credit at the episode level rather than to individual search steps. Despite these promising results, we observe that reward signals in existing retrieval-augmented reasoning methods operate at trajectory level. In GRPO, a group of trajectories is sampled for each question, and each trajectory’s advantage is determined by its overall reward relative to the group. This design introduces two structural issues that limit fine-grained learning of search behaviors. Figure 1 : Trajectory-level rewards obscure step-level query quality. Despite the left trajectory executing a precise search and the right one retrieving irrelevant contents, both receive identical rewards for producing the same final answer. Inability to distinguish query quality within a rollout group. As illustrated in Figure 1 , for a multi-hop question, one trajectory may generate a precise query that directly retrieves the answer-bearing document, while another may issue a vague query that returns mostly irrelevant content. If both trajectories produce the same final answer, they receive identical reward signals, leaving the model unable to learn which query formulation is more effective. This problem is particularly severe in multi-hop scenarios, where a suboptimal query at an early step can derail the entire reasoning chain. Vanishing signal in the all-failure scenario. When all trajectories fail to produce a correct answer, the trajectory-level advantages become approximately zero, yielding nearly no gradient signal. This scenario is common during early training and on difficult multi-hop questions, exactly when effective supervision is most needed. To address these limitations, we propose IG-Search , an RL framework that rewards each search step according to the Information Gain (IG) of its retrieval. The core idea is simple: given a fixed retriever, the quality of a search query is directly reflected in the documents it retrieves, so we can evaluate each query by measuring the informational value of those documents. Concretely, for each search step, IG quantifies how much the actual retrieved documents improve the model’s confidence in the gold answer relative to a counterfactual baseline of random documents. This signal is then propagated back to the corresponding query tokens through per-token advantage modulation within GRPO, so that different search steps within the same trajectory receive gradients proportional to their informational contribution, rather than sharing a single trajectory-level advantage. Crucially, because IG is computed independently of whether the final answer is correct, the step-level signal remains informative even when all trajectories fail. We further develop stabilization mechanisms to ensure robust training, detailed in § 2.4 . Prior works have also explored step-level supervision. IGPO [ 34 ] defines a turn-level reward as the temporal difference in gold-answer log-probability between consecutive turns, but this signal conflates reasoning, querying, and retrieval within each turn rather than isolating query quality. StepSearch [ 41 ] and GiGPO [ 4 ] rely respectively on GPT-4o-generated sub-question annotations and on state recurrence across trajectories. IG-Search requires none of these: it derives a query-grounded signal via counterfactual comparison, using only standard QA pairs; we discuss these methods in detail in Appendix A . We evaluate IG-Search on seven QA benchmarks spanning single-hop (NQ, TriviaQA, PopQA) and multi-hop (HotpotQA, 2WikiMultihopQA, Musique, Bamboogle) settings. With Qwen2.5-3B as the base model, IG-Search achieves an average Exact Match of 0.430, outperforming the strongest trajectory-level baseline (MR-Search) by 1.6 points and step-level method GiGPO by 0.9 points on average across benchmarks, with particularly pronounced gains on multi-hop reasoning tasks. Experiments with Qwen2.5-7B further confirm the generalizability of our method, yielding consistent improvements across all benchmarks. Our contributions are summarized as follows: • We propose IG-Search, which introduces a step-level IG reward that measures the informational contribution of each search step. IG-Search can be viewed as a retrieval-grounded process reward : it derives signal from the policy’s own generation probabilities conditioned on the gold answer, requiring neither a separately trained reward model nor intermediate annotations beyond standard question-answer pairs. • We develop per-token advantage modulation within GRPO so that different search steps receive gradient signals proportional to their informational contribution, together with stabilization mechanisms that prevent degenerate behaviors. Notably, the resulting signal remains informative

Read the complete text