Q-RAG: Long Context Multi‑Step Retrieval via Value‑Based Embedder Training

Artyom Sorokin, Nazar Buzun, Alexander Anokhin, Oleg Inozemcev, Egor Vedernikov, Petr Anokhin, Mikhail Burtsev, Trushkov Alexey, Yin Wenshuai, Evgeny Burnaev, Applied AI, Moscow, Russia, Learnable Intelligence Lab, Moscow, Russia, CILAB.AI, Moscow, Russia, London Institute for Mathematical Sciences, · 2025

Read Q-RAG: Long Context Multi‑Step Retrieval via Value‑Based Embedder Training by Artyom Sorokin, Nazar Buzun, Alexander Anokhin, Oleg Inozemcev, Egor Vedernikov, Petr Anokhin, Mikhail Burtsev, Trushkov Alexey, Yin Wenshuai, Evgeny Burnaev, Applied AI, Moscow, Russia, Learnable Intelligence Lab, Moscow, Russia, CILAB.AI, Moscow, Russia, London Institute for Mathematical Sciences, (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-07328.txt.

Introduction

Artyom Sorokin and nine co-authors improve retrieval-augmented creation for questions that need several rounds of searching. Standard retrieval fetches context once, which fails on complex questions, and newer multi-step methods fine-tune small language models to do the searching, an expensive route that locks out larger models. Q-RAG instead fine-tunes only the embedder, the part that represents text for retrieval, using reinforcement learning with value-based training.

The embedder learns to retrieve step by step, gathering the pieces a long-context answer needs. Q-RAG proved competitive and resource-efficient for open-domain question answering and reached state-of-the-art results on the BabiLong and RULER long-context benchmarks, handling contexts up to ten million tokens. The code is publicly available, making the approach easy to adopt.

Catalog record

CREATORArtyom Sorokin, Nazar Buzun, Alexander Anokhin, Oleg Inozemcev, Egor Vedernikov, Petr Anokhin, Mikhail Burtsev, Trushkov Alexey, Yin Wenshuai, Evgeny Burnaev, Applied AI, Moscow, Russia, Learnable Intelligence Lab, Moscow, Russia, CILAB.AI, Moscow, Russia, London Institute for Mathematical Sciences,
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-07328

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Q-RAG: Long Context Multi‑Step Retrieval via Value‑Based Embedder Training Artyom Sorokin, Nazar Buzun, Alexander Anokhin, Oleg Inozemcev, Egor Vedernikov, Petr Anokhin, Mikhail Burtsev, Trushkov Alexey, Yin Wenshuai, Evgeny Burnaev, Applied AI, Moscow, Russia, Learnable Intelligence Lab, Moscow, Russia, CILAB.AI, Moscow, Russia, London Institute for Mathematical Sciences, London, UK, Higher School of Economics, Moscow, Russia, Independent Researcher [email protected], [email protected] Abstract Retrieval-Augmented Generation (RAG) methods enhance LLM performance by efficiently filtering relevant context for LLMs, reducing hallucinations and inference cost. However, most existing RAG methods focus on single-step retrieval, which is often insufficient for answering complex questions that require multi-step search. Recently, multi-step retrieval approaches have emerged, typically involving the fine-tuning of small LLMs to perform multi-step retrieval. This type of fine-tuning is highly resource-intensive and does not enable the use of larger LLMs. In this work, we propose Q-RAG, a novel approach that fine-tunes the Embedder model for multi-step retrieval using reinforcement learning (RL). Q-RAG offers a competitive, resource-efficient alternative to existing multi-step retrieval methods for open-domain question answering and achieves state-of-the-art results on the popular long-context benchmarks Babilong and RULER for contexts up to 10M tokens. 1 Introduction Large language models (LLMs) have achieved impressive results across a wide range of tasks ( 22 ; 9 ; 32 ) . However, they still face some several fundamental limitations such as static knowledge, computational inefficiency on long contexts, degraded performance caused by attention dilution, and hallucinations ( 11 ; 15 ; 18 ) . Retrieval-Augmented Generation (RAG) is one of the most widely used techniques to address these issues ( 35 ) . RAG works by extracting only the most relevant parts from a large external corpus or context, such as newly added knowledge or lengthy texts. This allows LLMs to operate on shorter and more focused inputs, improving efficiency and output quality. Most current RAG methods rely on single-step retrieval. This setup performs well in relatively simple tasks like Needle-in-a-Haystack ( 11 ) . Still, more complex problems require multi-step interaction with the context. Multi-step retrieval can be viewed as a form of search-based reasoning. There are several existing approaches to multi-step retrieval reasoning. One direction involves constructing a knowledge graph from the retrieved information ( 19 ; 16 ) . These methods are often slow at inference time, since the LLM must process the entire context to build the graph for each new input. Another line of work uses LLM agents, which interleave RAG queries with LLM-generated instructions ( 28 ; 1 ) . These systems are sensitive to noisy or inaccurate retrieved passages, which may disrupt the generation of future queries. This shows the need for joint optimization of the retrieval and generation components. Recently, methods have emerged that fine-tune LLMs to interact more effectively with retrieval tools ( 29 ; 14 ; 6 ) . These methods tend to perform better, but they require expensive fine-tuning of the LLM itself. This makes them impractical for large models and limits accessibility for most researchers and practitioners. In this work, we focus on developing a resource-efficient multi-step RAG approach using reinforcement learning. Instead of fine-tuning an LLM, we train an agent that performs retrieval directly in the latent space of text chunk embeddings. This allows us to learn a compact and efficient model using value-based RL methods. Our approach achieves state-of-the-art results on long-context commonsence reasoning, multi-hop QA, and NIAH tasks with contexts up to 10 million tokens. It also performs competitively on open-domain QA benchmarks such as Musique and HotpotQA ( 34 ; 34 ) , while being significantly faster and cheaper to train and run compared to existing multi-step RAG methods. Our contributions are the following: • We propose a new method for training a multi-step retrieval agent using temporal difference reinforcement learning. • We achieve state-of-the-art results on benchmarks that require commonsense reasoning and NIAH tasks over ultra long contexts (up to 10M tokens). • We introduce a new way to incorporate temporal information into the multi-step embedder, enabling temporal reasoning during retrieval. Our temporal reasoning mechanism generalizes well to long contexts at inference time. 2 Related Works There are several main directions for tackling complex retrieval scenarios on long context tasks. A highly popular approach involves building fine-tuning free LLM Agents that combine off-the-shelf retrievers with LLMs, such as Search-o1 ( 17 ) . Many of these works further enhance retrieval quality by constructing large knowledge graphs over the context, which, while requiring little additional training, are extremely slow at inference due to the need for LLMs to process the entire context, e.g. GraphReader ( 16 ) , HippoRAG ( 13 ) , AriGraph ( 1 ) . Another line of work fine-tunes LRMs to perform multi-step retrieval, allowing the model to generate intermediate search queries inside the reasoning for long contexts. The first work to apply this idea was IM-RAG ( 33 ) , which fine-tuned the LLM with a frozen embedder using PPO ( 24 ) . More recent papers, such as R1-Searcher ( 29 ) , Search-R1 ( 14 ) , RAG-RL ( 12 ) , and ReSearcher ( 6 ) , extended this direction by employing GRPO ( 26 ) for the task. Unlike these methods, which freeze the embedder and fine-tune the LLM, our approach fine-tunes only the embedder, allowing it to pair with LLMs of any size, including proprietary ones, while keeping fine-tuning efficient and inexpensive. A different approach is to fine-tune the retriever itself using feedback from the LLM, as in RePlug ( 27 ) . This direction is most similar to ours, but RePlug did not address multi-step reasoning or use reinforcement learning in this setting. BeamRetriever ( 36 ) achieves state-of-the-art results on short-context QA by training a reranker for BeamSearch-style planning. In contrast, Q-RAG trains the embedder with reinforcement learning, enabling faster inference and better scalability to long contexts through efficient vector similarity instead of transformer-based trajectory scoring. Extremely long-sequence processing is demonstrated by models that combine recurrence with Transformer architecture. The Mamba family of state space models ( 8 ) replaces attention with structured recurrent dynamics, offering linear-time scalability and strong performance on long sequences, though often at the cost of weaker in-context learning and less expressive token-to-token interaction compared to Transformer-based architectures. The Recurrent Memory Transformer (RMT) ( 5 ) introduces segment-level recurrence by passing memory tokens between fixed-size segments, enabling Q&A on sequences up to 10M tokens. Titans ( 3 ) frames recurrent memory training as a meta-learning problem, showing scaling beyond 2M tokens. Building on this idea, ATLAS ( 2 ) increases memory capacity, achieving better long-context performance than both RMT and Titans. The Associative Recurrent Memory Transformer (ARMT) ( 23 ) employs quasi-linear, associative attention in each layer and attains the best long-context scores among recurrent models. Our approach outperforms all of these models on contexts beyond 1M tokens while belonging to a different class of methods. LongRoPE2 ( 25 ) tackles the positional encoding bottleneck, extending the effective context window of pre-trained LLMs to 128K tokens while retaining short-context performance through RoPE rescaling and mixed-window training. 3 Methods Figure 1: Q - RAG agent interacts with multi-step retrieval environment. The starting state s_{0} contains the initial query q .

Read the complete text