From Sufficiency to Reflection: Reinforcement-Guided Thinking Quality in Retrieval-Augmented Reasoning for LLMs
Jie He, Víctor Gutiérrez Basulto, and Jeff Z. Pan · 2025
Read From Sufficiency to Reflection: Reinforcement-Guided Thinking Quality in Retrieval-Augmented Reasoning for LLMs by Jie He, Víctor Gutiérrez Basulto, and Jeff Z. Pan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-22716.txt.
Introduction
Jie He, Víctor Gutiérrez Basulto, and Jeff Pan improve the thinking of retrieval-augmented language models in a 2025 paper. Such models fetch facts before answering, but most training rewards only the final answer.
The authors name three failure modes: fetching too little, reasoning badly despite good material, and answering against one's own reasoning. Their framework, TIRESRAG-R1, runs a think, retrieve, reflect loop. It rewards enough retrieval, sound reasoning, and self-correction separately. Hard questions get extra weight in training, and weak samples are filtered out. On four multi-hop question sets the method beat earlier systems. It also carried over to simpler one-hop questions.
Catalog record
| CREATOR | Jie He, Víctor Gutiérrez Basulto, and Jeff Z. Pan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 74 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-22716
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
From Sufficiency to Reflection: Reinforcement-Guided Thinking Quality in Retrieval-Augmented Reasoning for LLMs Jie He, Víctor Gutiérrez Basulto, and Jeff Z. Pan Abstract Reinforcement learning-based retrieval-augmented generation (RAG) methods enhance the reasoning abilities of large language models (LLMs). However, most rely only on final-answer rewards, overlooking intermediate reasoning quality. This paper analyzes existing RAG reasoning models and identifies three main failure patterns: (1) information insufficiency: failure to retrieve adequate support; (2) faulty reasoning: logical or content-level flaws despite sufficient information; (3) answer–reasoning inconsistency: a valid reasoning chain with a mismatched final answer. We propose TIRESRAG-R1 , a novel framework using a think–retrieve–reflect process and a multi-dimensional reward system to improve reasoning and stability. TIRESRAG-R1 introduces: (1) a sufficiency reward to encourage thorough retrieval; (2) a reasoning quality reward to assess rationality and accuracy of the reasoning chain; (3) a reflection reward to detect and revise errors. It also employs a difficulty-aware reweighting strategy and training sample filtering to boost performance on complex tasks. Experiments on four multi-hop QA datasets show TIRESRAG-R1 outperforms prior RAG methods and generalizes well to single-hop tasks 1 1 1 The code and data can be found at https://github.com/probe2/TIRESRAG-R1 . . 1 Introduction Large language models (LLMs) Grattafiori et al. 2024 ; Yang et al. 2025 ; OpenAI et al. 2024a have achieved remarkable performance across a wide range of downstream tasks, such as mathematical reasoning Ahn et al. 2024 , code generation Jiang et al. 2025 , open-domain question answering Kamalloo et al. 2023 . A key factor behind this success is chain-of-thought prompting Wei et al. 2022 , which enables LLMs to generate intermediate reasoning steps before arriving at a final answer, significantly enhancing performance on reasoning tasks Wang et al. 2023 ; Pan et al. 2023 ; Snell et al. 2025 . Despite these advances, the internal knowledge of LLMs is not always reliable. For instance, when faced with time-sensitive queries Mousavi et al. 2024 or conflicting evidence within their internal representations Xu et al. 2024 , LLMs often produce hallucinations or factual inaccuracies due to outdated or ambiguous information Li et al. 2024a ; Wang et al. 2024b . Figure 1: An example showing different reasoning trajectories for answering a multi-hop query. It compares insufficient information, incorrect predictions, and fully correct reasoning. The retrieval-augmented generation (RAG) paradigm Gao et al. 2024a improves factual accuracy and robustness by enabling LLMs to access up - to - date external knowledge. However, standard RAG often treats retrieval and generation as loosely coupled, lacking mechanisms for multi - step reasoning such as query decomposition or identifying knowledge gaps, which limits performance on tasks requiring deeper reasoning. To overcome this, a growing body of research has turned to reinforcement learning (RL) (a paradigm that has demonstrated strong performance in mathematical reasoning and code generation Shao et al. 2024 ; Wang et al. 2024a ) to train LLMs for retrieval-augmented reasoning. In this setting, the model learns when and how to retrieve, as well as how to integrate retrieved information into coherent reasoning chains, guided by outcome-level rewards (e.g., answer correctness) Jin et al. 2025a ; Song et al. 2025 ; Sun et al. 2025a . Despite recent progress, outcome-based RL methods still face two significant limitations: (1) they often neglect the quality and validity of intermediate reasoning steps, and (2) they lack fine-grained feedback to guide the reasoning process during multi-step tasks. As a result, models may learn incorrect reasoning paths that degrade answer quality, or they may arrive at correct answers through flawed reasoning, thereby compromising interpretability. In this paper, to address these challenges, we begin by evaluating the performance of RAG-based reasoning models trained with outcome-based rewards. Through human analysis on examples from multi-hop reasoning datasets (e.g., MuSiQue Trivedi et al. 2022 ), as shown in Fig. 1 , we identify three primary bottlenecks limiting model accuracy: (1) the model retrieves sufficient information but still fails to produce the correct answer; (2) reasoning is prematurely interrupted, resulting in inadequate retrieval; and (3) the reasoning chain is correct, yet the final answer is incorrect. Based on these insights, we propose TIRESRAG-R1 , an approach that enhances reasoning chain quality by incorporating rewards focused on sufficiency and reasoning coherence. Additionally, we introduce a thinking–retrieval–reflection framework, motivated by the principle that models should first ensure the integrity of their reasoning chains. If the final answer is incorrect despite a valid chain, the model engages in reflection to revise its response. TIRESRAG-R1 enhances RAG-based reasoning by introducing a thinking–retrieval–reflection framework, where the model can decide post-answer whether to reflect and revise its output through an additional round of reasoning and retrieval. To guide learning, TIRESRAG-R1 assigns multi-dimensional reward signals to each reasoning trajectory, including: an answer reward, a sufficiency reward to encourage retrieval of adequate evidence, a reasoning quality reward evaluating logical coherence, alignment, error awareness, and conciseness, and a reflection reward that promotes correction of wrong answers while discouraging unnecessary changes. To address the varying difficulty of retrieval tasks, we propose a difficulty-aware advantage reweighting strategy that adjusts reward weights based on the ease of retrieving sufficient evidence, and an advantage filtering mechanism that excludes trivial examples (where all responses are correct or incorrect) to stabilize RL training. Together, these components enable more robust and interpretable reasoning, significantly outperforming traditional outcome-based methods across multiple QA benchmarks. Our contributions can be summarized as follows: • We are the first to define overthinking and underthinking in RAG and conduct comprehensive analysis to reveal the main bottlenecks faced by current RL-trained RAG reasoning models. • We propose TIRESRAG-R1 , which enhances reasoning by introducing a reflection mechanism and fine-grained multi-dimensional rewards to improve both reasoning chains and answer accuracy. We also propose a difficulty-aware advantage reweighting strategy with no additional computational cost to guide the model toward optimizing on harder questions. • Our experiments show that TIRESRAG-R1 outperforms multiple state-of-the-art RAG methods on several open-ended QA datasets, with an average accuracy improvement of 5.8%. Further ablation studies and analysis confirm the effectiveness of our fine-grained rewards. 2 Related Work Retrieval-Augmented Generation. RAG has been widely adopted to mitigate hallucination, domain knowledge gaps, and temporal staleness Ayala and Bechard 2024 ; Siriwardhana et al. 2023 ; Gade and Jetcheva 2024 . Traditional RAG typically follows a static retrieve - then - generate pipeline Lewis et al. 2020 ; Guu et al. 2020 , which is effective for open - domain QA but often struggles with multi - hop reasoning, latent constraints, and ambiguous queries Tang and Yang 2024 ; Li et al. 2024b ; Chan et al. 2024 . Recent work has proposed more cognitively inspired architectures, such as AdaptiveRAG Jeong et al. 2024 using query classification to choose retrieval strategies. In parallel, modular and hybrid frameworks Gao et al. 2024b ; Zhang et al. 2024 ; Zhou et al. 2024 integrate evidence aggregation, verification, and query rewriting, reflecting a deeper interaction between retrieval and reasoning. Reinforcement Learning for LLM Reasoning. Reinforcement learning (RL) has recently been explored as a way to improve multi - step reasoning in LLMs. Models such as GPT - o1 and DeepSeek - R1 OpenAI et al. 2024b ; DeepSeek-AI et al. 2025 demonstrate RL - based training