DeepRAG: Integrating Hierarchical Reasoning and Process Supervision for Biomedical Multi-Hop QA
Yuelyu Ji, Hang Zhang, Shiven Verma, Hui Ji, Chun Li, Yushui Han, Yanshan Wang · 2025
Read DeepRAG: Integrating Hierarchical Reasoning and Process Supervision for Biomedical Multi-Hop QA by Yuelyu Ji, Hang Zhang, Shiven Verma, Hui Ji, Chun Li, Yushui Han, Yanshan Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-00671.txt.
Introduction
Yuelyu Ji and co-authors propose this 2025 framework for medical questions that need several hops of reasoning. DeepRAG splits a hard question into sub-questions with a strong reasoning model, then retrieves evidence for each step. A training setup tunes the pipeline with rewards at the process level, graded against a medical concept ontology. On the MedHopQA benchmark it beat the standalone reasoning model and the untuned retrieval setup on exact-match and concept accuracy.
Catalog record
| CREATOR | Yuelyu Ji, Hang Zhang, Shiven Verma, Hui Ji, Chun Li, Yushui Han, Yanshan Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 13 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-00671
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DeepRAG: Integrating Hierarchical Reasoning and Process Supervision for Biomedical Multi-Hop QA Yuelyu Ji, Hang Zhang, Shiven Verma, Hui Ji, Chun Li, Yushui Han, Yanshan Wang Abstract We propose DeepRAG, a novel framework that integrates DeepSeek’s hierarchical question decomposition capabilities with RAG-Gym’s unified retrieval-augmented generation (RAG) optimization using process-level supervision. Targeting the challenging MedHopQA biomedical question-answering task, DeepRAG systematically decomposes complex queries into precise sub-queries and employs concept-level reward signals informed by the UMLS ontology to enhance biomedical accuracy. Preliminary evaluations on the MedHopQA dataset indicate that DeepRAG significantly outperforms baseline models, including standalone DeepSeek and RAG-Gym, achieving notable improvements in both Exact Match (EM) and concept-level accuracy. [[email protected]] [[email protected]] [[email protected]] [[email protected]] [[email protected]] [[email protected]] [[email protected]] keywords retrieval-augmented generation, biomedical question answering, process supervision † † copyrightyear: 2025 † † copyright: Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). † † venue: Challenge and Workshop (BC9): Large Language Models for Clinical and Biomedical NLP, International Joint Conference on Artificial Intelligence (IJCAI), August 16–22, 2025, Montreal, Canada † † address: Department of Information Science, University of Pittsburgh, Pittsburgh, PA, USA † † address: Department of Biomedical Informatics, University of Pittsburgh, Pittsburgh, PA, USA † † address: Department of Health Information Management and ISP, University of Pittsburgh, Pittsburgh, PA, USA † † corresp: Corresponding author. † † These authors contributed equally. 1 Introduction Large language models (LLMs) have achieved remarkable success in knowledge-intensive tasks, yet they often struggle with complex, multi-hop reasoning required in biomedical question answering Lewis et al. 2020 ; Gao et al. 2023 ; Xiang et al. 2024 ; Su et al. 2022 ; Zhao et al. 2025 ; Zhang et al. 2025 ; Yang et al. 2025 ; Liang et al. 2025 ; Jin et al. 2024 ; Jin and Yang 2025 ; Li et al. 2024 ; Ding et al. 2024 ; Deng et al. 2024 ; Ji et al. 2024a ; Ji et al. 2024b . The MedHopQA track of BioCreative IX demands the sequential integration of heterogeneous sources, including genetic, disease, and treatment information derived from Wikipedia. DeepSeek Guo et al. 2025 introduced a hierarchical, multi-stage reasoning agent that dynamically decomposes questions into sub-queries. Separately, RAG-Gym Xiong et al. 2025 provides a unified framework for optimizing retrieval-augmented generation with process-level rewards, enabling robust tuning of both reasoning and search steps. In this work, we propose DeepRAG , a novel integration framework that leverages DeepSeek R1’s hierarchical question decomposition and RAG-Gym’s process supervision approach. Our key contributions include: (1) proposing a novel two-stage hierarchical reasoning pipeline with clearly defined Reasoning and Query Modules; (2) introducing hierarchical indicators to explicitly track and manage nested dependencies within complex biomedical queries; and (3) enhancing biomedical accuracy by replacing the original LLaMA backbone in RAG-Gym with DeepSeek R1, a distilled model optimized for hierarchical reasoning. We hypothesize that hierarchical reasoning can generate more precise search actions, while reward-guided fine-tuning ensures sub-queries and answers remain aligned with task objectives. 2 Method 2.1 Hierarchical Reasoning with DeepSeek Our framework employs DeepSeek R1, a distilled version of LLaMA optimized for hierarchical reasoning, particularly suited for complex biomedical multi-hop queries. Hierarchical reasoning decomposes the overall task into manageable subtasks through a two-stage pipeline. Our proposed hierarchical reasoning pipeline consists of two explicitly defined modules:1) Reasoning Module: This novel module generates a structured answer outline, explicitly identifying claims that cannot be directly validated internally and therefore require external retrieval.2)Query Module: Another innovative component of our framework that formulates targeted and precise sub-queries based on claims identified by the Reasoning Module, optimizing the retrieval process. Furthermore, we introduce novel hierarchical indicators, which are explicitly designed to track nested reasoning dependencies, enabling clear delineation and efficient retrieval of complex biomedical knowledge. These contributions represent enhancements beyond the capabilities provided by the standalone DeepSeek model, specifically tailored towards the complex, multi-hop nature of biomedical question-answering tasks. To effectively manage complex multi-step reasoning, we introduce hierarchical indicators that explicitly track and represent nested dependencies and levels of queries. This structured decomposition allows the system to clearly delineate reasoning pathways and optimizes retrieval processes by focusing on precise information needs. Our framework utilizes DeepSeek R1DeepSeek R1DeepSeek R1 as a foundational model optimized for hierarchical reasoning. Building upon this, we introduce our novel two-stage pipeline comprising the Reasoning Module and the Query Module. Specifically, we propose hierarchical indicators and structured decomposition, distinct from DeepSeek’s original capabilities, to explicitly track dependencies and optimize retrieval effectiveness. 2.2 Process Supervision via RAG-Gym The integration of hierarchical reasoning outputs from DeepSeek into RAG-Gym transforms the question-answering process into a sequential Markov Decision Process (MDP). Each sub-query generated by DeepSeek is treated as a discrete action within this MDP framework. RAG-Gym then guides the retrieval and answer generation processes through fine-grained, reward-based supervision. Specifically, RAG-Gym applies three distinct reward signals: 1. Sufficiency Reward : Rewards are provided based on the completeness and adequacy of the retrieved evidence relative to the sub-query, promoting comprehensive information gathering. 2. Utility Reward : Rewards directly assess the relevance and effectiveness of retrieved documents in formulating accurate final answers, encouraging optimal document selection. 3. Redundancy Penalty : Penalties discourage the retrieval of redundant or highly similar documents, thereby promoting diversity in the retrieved information and reducing computational overhead. Additionally, we introduce Concept-Level Rewards using Unified Medical Language System (UMLS) semantic matching. These rewards explicitly reinforce semantic precision and biomedical accuracy, ensuring that the system’s retrieved and generated information aligns closely with precise biomedical concepts and terminologies. 2.3 Training Procedure Training involves generating approximately 1,000 labeled query trajectories using ChatGPT-4o as an oracle for process supervision. These labeled trajectories include sequences of intermediate reasoning steps, generated sub-queries, retrieved documents, and the corresponding final answers. Leveraging these labeled examples, we apply reinforcement learning, specifically Direct Preference Optimization (DPO), to fine-tune the DeepRAG model. DPO specifically adjusts model parameters based on explicit preference signals derived from contrasting selected and rejected sub-queries, thereby aligning the model’s retrieval and reasoning behaviors closely with expert human judgments. 3 Experiments 3.1 Dataset We conducted experiments using the MedHopQA dataset provided by the BioCreative IX MedHopQA track. This dataset consists of 1,000 carefully curated multi-hop biomedical questions and answers related to rare diseases, genetics, and treatments. To rigorously evaluate our model and obscure test examples, the MedHopQA challenge provides approximately 10,000 questions, embedding 1,000 hidden test questions within. 3.2 Experimental Setup We set up experiments as follows: Baseline Models : We evaluated against zero-shot ChatGPT-4o, the standalone DeepSeek R1 model, and the vanilla RAG-Gym without hierarchical reasoning. DeepRAG Model Configuration : We implemented DeepRAG by modifying the original RAG-Gym framework in two significant ways: Model Backbone Replacement: Replaced RAG-Gym’s original LLaMA model with DeepSeek R1, a distilled variant optimized specifically for hierarchical reasoning in biomedical domains. This change significantly improved hierarchical decomposition accuracy for biomedical queries. Hierarchical Indicator Introduction: Implemented our proposed hierarchical indicators, enhancing the precision and clarity of sub-query generation and retrieval actions. These modifications aim to optimize both retrieval precision and reasoning efficiency specifically for the multi-hop biomedical tasks defined in the MedHopQA challenge. Evaluation Metrics : We adopted the evaluation metrics specified by the MedHopQA challenge, Exact Match (EM) for answer accuracy, and Concept Accuracy, measured