UR 2 : Unify RAG and Reasoning through Reinforcement Learning

Weitao Li [email protected], Boran Xiang, Xiaolong Wang, Zhinan Gou, Weizhi Ma [email protected], Yang Liu [email protected] · 2025

Read UR 2 : Unify RAG and Reasoning through Reinforcement Learning by Weitao Li [email protected], Boran Xiang, Xiaolong Wang, Zhinan Gou, Weizhi Ma [email protected], Yang Liu [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-06165.txt.

Introduction

Language models have two great helpers: looking things up, and being trained by reward to reason. Weitao Li and co-authors fuse the two in one training framework called UR2. Earlier fusions worked only on open question answering with fixed lookup rules. This one learns when to look things up and when to just think.

A difficulty-aware schedule calls for retrieval only on hard problems, sparing the easy ones. Knowledge comes from two wells: stored domain collections kept offline, and summaries the model writes on the fly. Tests spanned open questions, a hard general benchmark, medicine, and mathematics. The unified models beat both pure lookup and pure reasoning baselines. On several tests they matched strong commercial mini models, and everything is released openly.

Catalog record

CREATORWeitao Li [email protected], Boran Xiang, Xiaolong Wang, Zhinan Gou, Weizhi Ma [email protected], Yang Liu [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE86 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-06165

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

UR 2 : Unify RAG and Reasoning through Reinforcement Learning Weitao Li [email protected], Boran Xiang, Xiaolong Wang, Zhinan Gou, Weizhi Ma [email protected], Yang Liu [email protected] Abstract Large Language Models (LLMs) have shown remarkable capabilities through two complementary paradigms: Retrieval-Augmented Generation (RAG), which enhances knowledge grounding, and Reinforcement Learning from Verifiable Rewards (RLVR), which optimizes complex reasoning abilities. However, these two capabilities are often developed in isolation, and existing efforts to unify them remain narrow in scope—typically limited to open-domain QA with fixed retrieval settings and task-specific assumptions. This lack of integration constrains generalization and limits the applicability of RAG-RL methods to broader domains. To bridge this gap, we propose UR 2 ( U nified R AG and R easoning), a general framework that unifies retrieval and reasoning through reinforcement learning. UR 2 introduces two key contributions: a difficulty-aware curriculum training that selectively invokes retrieval only for challenging problems, and a hybrid knowledge access strategy combining domain-specific offline corpora with LLM-generated summaries. These components are designed to enable dynamic coordination between retrieval and reasoning, improving adaptability across a diverse range of tasks. Experiments across open-domain QA, MMLU-Pro, medical, and mathematical reasoning tasks demonstrate that UR 2 (built on Qwen2.5-3/7B and LLaMA-3.1-8B) significantly outperforms existing RAG and RL methods, achieving comparable performance to GPT-4o-mini and GPT-4.1-mini on several benchmarks. We have released all code, models, and data at https://github.com/Tsinghua-dhy/UR2 . † † footnotetext: \dagger Weizhi Ma ([email protected]) and Yang Liu ([email protected]) are corresponding authors. 1 Introduction Large Language Models (LLMs) have achieved remarkable performance across diverse tasks by incorporating external knowledge (Retrieval-Augmented Generation, RAG) [ 19 , 1 , 14 ] and by optimizing reasoning through reinforcement learning with verifiable rewards (RLVR) [ 7 ] . RAG methods enable LLMs to access external knowledge, while RLVR has shown strong gains on mathematical and logical reasoning [ 43 , 2 ] . Motivated by these successes, recent work has begun to integrate retrieval and reasoning: for example, Search - o1 [ 22 ] embeds an agentic RAG workflow into the LLM’s chain - of - thought, and RAG - Gym [ 39 ] proposes a unified RL - based training framework for RAG agents. Similarly, RAG-RL methods—which learn to invoke retrieval through RL—such as R1-Searcher [ 31 ] and Search-R1 [ 15 ] use RLVR to train models on when and what to retrieve during reasoning, improving answer accuracy in open-domain QA. Despite recent progress, RAG-RL frameworks remain limited in scope. Most methods focus narrowly on open-domain QA, with retrieval tied to fixed reasoning steps or static knowledge sources like Wikipedia. For instance, R1-Searcher and Search-R1 assume access to broad knowledge bases, ill-suited for tasks requiring specialized or Emergent Planning Behaviors Figure 1: UR 2 can formulate and dynamically adjust plans during reasoning. Left: UR 2 develops a two-step plan: search for knowledge, then check each option. Right: UR 2 decomposes the diagnostic task into sequential steps, identifies knowledge gaps, and queries external information. More emergent cognitive behaviors can be found in E . real-time information. While methods like DeepResearcher attempt training in real web environments, they face inefficiencies due to the noisy and unstructured nature of online data [ 45 ] . Other methods like ZeroSearch [ 34 ] , use LLM-generated corpora to simulate retrieval, avoiding API costs but risking hallucination and loss of real-world complexity. These limitations highlight the need for a more versatile framework that can support broad-domain reasoning and flexibly integrate dynamic knowledge into reasoning. To address the limitations of existing RAG-RL approaches—such as static retrieval, limited domain generalization, and poor robustness in noisy environments—we propose a general and adaptive framework, UR 2 ( U nified R AG and R easoning), which uses RL to dynamically coordinate retrieval and reasoning. Unlike prior methods that rely solely on static corpora (e.g., Wikipedia) or simulate retrieval with synthetic content, UR 2 combines both: it leverages task-specific offline corpora for accurate grounding, augmented with LLM-generated summaries for efficiency and generalization. To address the imbalance between retrieval and reasoning in prior methods, we design a difficulty-aware curriculum that adaptively controls when to trigger retrieval during training. Specifically, retrieval is used only for harder instances, encouraging the model to rely on internal reasoning when possible and to learn retrieval strategies only when necessary. This reduces retrieval overhead, improves query quality on challenging questions, and preserves reasoning capability across tasks. We train UR² on Qwen-2.5-3B/7B-Instruct [ 40 ] and LLaMA-3.1-8B-Instruct [ 4 ] across MMLU-Pro, Medicine, Math, and open-domain QA. During training, the model spontaneously develops key cognitive behaviors: self-verification through retrieval, intermediate reasoning validation, and hypothesis revision based on external evidence. UR² outperforms previous state-of-the-art (SOTA) methods by 5.8% (7B) and 19.0% (3B) on average, with peak gains of 9.5% and 29.6% . Notably, our 7B model matches GPT-4o-mini and GPT-4.1-mini 1 1 1 https://chat.openai.com/ , and generalizes well across domains and model architectures. Our main contributions are summarized as follows: • We propose UR 2 , the first unified framework integrating RAG and RL for dynamic retrieval-reasoning coordination, applicable beyond general QA to diverse tasks. • We design a unified data representation and training scheme bridging retrieval and reasoning, with difficulty-aware curricula and LLM-summarized corpora for accurate grounding and broad generalization, implemented via a modular two-stage framework. • Comprehensive experiments demonstrate that UR 2 surpasses advanced RAG and RL methods without expert demonstrations and generalizes robustly across domains. 2 Related Work 2.1 Retrieval - Augmented Generation Retrieval-Augmented Generation (RAG) enhances large language models by incorporating external information to reduce hallucinations [ 6 ] . Early RAG methods concatenate retrieved documents with input prompts [ 19 , 14 , 1 ] . Subsequent approaches have evolved in multiple directions: advanced RAG methods incorporate sophisticated retrieval and re-ranking strategies [ 6 , 25 ] ; post-hoc verification methods address hallucinations by retrieving documents based on generated responses [ 21 , 33 ] ; and Graph-based RAG methods integrate knowledge graphs for multihop reasoning [ 5 , 13 , 25 ] . Recent RL-RAG frameworks have explored retrieval integration during training via real-time or synthetic retrieval [ 45 , 34 ] . However, these approaches remain constrained by static retrieval strategies, limited domain generalization, and inability to dynamically coordinate retrieval with reasoning across diverse task types. 2.2 Reinforcement Learning for Retrieval - Enhanced Reasoning Reinforcement learning (RL) has emerged as a key technique for improving LLM reasoning, evolving from early policy gradient methods like REINFORCE [ 38 ] to stronger algorithms such as PPO [ 29 ] and GRPO [ 30 ] . Recent methods like DeepSeek - R1 [ 7 ] , Search - R1 [ 15 ] , and R1 - Searcher [ 31 ] show that RL enables LLMs to learn multi-step reasoning and retrieval strategies without human feedback. These works highlight a shift from fixed retrieval heuristics to learned, RL-driven retrieval policies, which form the basis for our unified framework. 3 Method We propose UR 2 , a general framework that tightly integrates retrieval-based grounding with explicit step-by-step reasoning via RL. Unlike previous approaches restricted to open-domain question answering or reliant upon static corpora, UR 2 supports a broad range of tasks, including mathematical problem solving and domain-specific decision-making. To achieve

Read the complete text