Retriever Portfolios: A Principled Approach to Adaptive RAG

Miltiadis Stouras, Vincent Cohen-Addad, Silvio Lattanzi, Ola Svensson · 2026

Read Retriever Portfolios: A Principled Approach to Adaptive RAG by Miltiadis Stouras, Vincent Cohen-Addad, Silvio Lattanzi, Ola Svensson (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-31176.txt.

Introduction

Stouras and colleagues swap the single retriever in retrieval-augmented generation for a small set picked for the query mix. One retriever and one setting cannot serve short fact questions and long multi-hop reasoning equally well. The authors frame the choice as an expected best-of-k goal and give a build method with near-optimal guarantees. Learned sets beat single retrievers and naive multi-retriever baselines on question-answering tests. Fixed sets also cut delay and token cost compared with tuning settings at run time.

Catalog record

CREATORMiltiadis Stouras, Vincent Cohen-Addad, Silvio Lattanzi, Ola Svensson
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE74664
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.31176

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Retriever Portfolios: A Principled Approach to Adaptive RAG Miltiadis Stouras, Vincent Cohen-Addad, Silvio Lattanzi, Ola Svensson Abstract Retrieval-augmented generation (RAG) systems typically rely on a single retriever and a single set of hyperparameters, despite facing highly heterogeneous queries that range from simple factoid questions to complex multi-hop reasoning. We propose a method that automatically selects a small, diverse subset of retrievers (a portfolio) from a large pool of candidates, to cover different regions of the target query distribution. We formalize this setting via an expected best-of- k k objective over the query distribution and show that it admits an efficient portfolio construction algorithm with near-optimal guarantees. Across multiple QA benchmarks, our learned portfolios and router pipeline consistently outperform single-retriever and naive multi-retriever baselines on both retrieval metrics and answer quality. In addition, compared to inference-time hyperparameter tuning approaches, fixed portfolios enable parallel retrieval and LLM calls, achieving comparable (and sometimes better) accuracy with substantially lower latency and token cost. 1 Introduction Retrieval-augmented generation (RAG) has emerged as a standard paradigm for grounding large language models (LLMs) in external knowledge sources. By retrieving documents from a corpus and conditioning generation on the retrieved evidence, RAG can improve factual accuracy and reduce hallucinations across a range of tasks, including open-domain QA and knowledge-intensive dialogue ( 17 ; 10 ; 20 ) . In this setup, a retriever maps a user query to a ranked list of documents, and the LLM produces an answer conditioned on both the query and the retrieved context. Despite substantial progress on retrieval models and architectures, the current practice in RAG systems is overwhelmingly to select a single retriever (and a single set of hyperparameters) and deploy it uniformly for all queries. However, the query distributions faced by RAG systems are highly heterogeneous. Some questions are simple factoid queries that can be answered from a single, lexically similar passage, while others require multi-hop reasoning over multiple documents, aggregation of semantically diverse evidence, or domain-specific terminology. Therefore, different retrieval configurations tend to specialize in different regimes ( 15 ; 11 ) . Even for a fixed retriever, recent work shows that it can be beneficial to adapt its hyperparameters at inference time to better suit each query ( 19 ) . As a result, there is growing evidence that no single retriever is optimal for all queries, and that fixing a single retrieval strategy can leave substantial performance on the table across diverse information needs ( 13 ) . To address this limitation several recent works explicitly aim to adapt their retrieval strategy to the query. Adaptive-RAG ( 11 ) trains a classifier to predict the “complexity level” of the given query and then chooses among a small, fixed menu of strategies (no retrieval, single-step retrieval, or iterative multi-hop retrieval) based on this prediction. Vendi-RAG ( 19 ) instead focuses on a parametric retriever and iteratively tunes its hyperparameter at inference time: for each query, it generates candidate answers under different settings and employs an LLM judge to steer the retriever towards a better relevance–diversity trade-off. While these approaches validate the benefits of adapting retrieval, they either operate over a very small, hand-designed set of strategies or perform expensive per-query search in a hyperparameter space. In contrast, we propose a principled approach for selecting a small portfolio of retrievers with complementary behavior, from a large candidate pool, that is provably near-optimal for the underlying query distribution. We tackle this problem by explicitly modeling the fact that different retrievers may be best for different questions. We view each query q as a sample from an underlying distribution, and each retriever r (including its architecture and hyperparameters) as a candidate tool for answering that query. For every query-retriever pair (q,r) , we define a score s(q,r) that measures how suitable r is for q , for example in terms of retrieval quality or end-to-end answer correctness. Rather than searching for a single retriever that performs best on average, we instead choose a small set S of k retrievers and evaluate this set by asking: for a random query q , how good is the best retriever in S ? Formally, we measure the quality of S by the expected “best-of- k ” score \mathbb{E}_{q}[\max_{r\in S}s(q,r)] over the query distribution. This objective encourages portfolios whose members specialize on different regions of the query space, since redundant retrievers that behave similarly contribute little to the maximum. In the rest of the paper we analyze this objective and design algorithms that, given a finite sample of queries and a large candidate pool of retrievers, efficiently construct small portfolios with provable guarantees and strong empirical performance. Our contributions We introduce the new problem, RAG portfolio optimization, and analyze it theoretically and practically. More formally: • Formulation & Theory. Given a large pool of candidate retrievers, we formalize an expected best-of- k objective over the query distribution that measures how well a portfolio of k retrievers covers heterogeneous information needs. Furthermore, we prove that one can efficiently obtain a near-optimal portfolio of retrievers. • Methods. We propose a pipeline that learns a static portfolio of complementary retrievers offline (Algorithm 1 ) and trains a lightweight router to dynamically select the best retriever per query (Figure 2 ). This approach amortizes the cost of adaptation, avoiding expensive iterative search at inference time. • Empirical evaluation. We evaluate our approach on diverse open-domain and multi-hop QA benchmarks (HotpotQA, 2WikiMultihopQA, TriviaQA, and MusiQue). Our method consistently yields better retrieval recall and answer accuracy compared to single-retriever baselines and inference-time tuning methods like Vendi-RAG, while significantly reducing latency and token usage. Candidate Retrievers Pool \mathcal{R} (DS, Vendi, etc.) Query Distribution \mathcal{D} (HotpotQA, TriviaQA, etc.) Algorithm 1 Retriever Portfolio Selection \max_{|S|\leq k}\mathbb{E}_{q}[\max_{r\in S}s(q,r)] Learned Portfolio S=\{r_{1},\dots,r_{k}\} Figure 1 : Offline Portfolio Optimization: Selecting a diverse subset of size k from a large pool of size m to cover the query distribution. Query text \mathbf{q} Encoder (T5+MPNet/E5) Enc(\mathbf{q}) Router s_{1} s_{2} s_{3} s_{k} \vdots Similarities with retrievers’ learned embeddings top- \ell filtering ✓ \times ✓ \vdots \times \mathbf{q} r_{1} Answer LLM \mathbf{q} r_{\ell} Answer LLM \vdots Parallel Retrieval & Answering a_{1} \vdots a_{\ell} Selector LLM Final answer Figure 2 : Adaptive Inference Pipeline: The router encodes query \mathbf{q} and identifies best portfolio members. Selected components execute in parallel to minimize latency, followed by selector aggregation. Related Work There are three main areas that are closely related to our work: retrieval augmented generation, adaptive retrieval approaches (including mixture of retrievers and routing pipelines) and portfolio optimization. Retrieval Augmented Generation. Retrieval-augmented generation (RAG) has become a standard approach for grounding large language models (LLMs) in external knowledge. Instead of relying solely on parametric memory, a RAG system retrieves a small set of relevant documents from a large corpus and conditions generation on both the query and the retrieved context, which improves factual accuracy and knowledge coverage on open-domain and knowledge-intensive tasks ( 17 ; 10 ; 20 ) . Early work combined neural retrievers with sequence-to-sequence generators for open-domain QA ( 15 ; 17 ) , and subsequent work has extended this paradigm to more complex settings, including multi-hop reasoning and conversational agents ( 20 ) . Across these systems, the retriever is a

Read the complete text