RAISE: RAG Design as an Architecture Search Problem
Zhen Chen, Yibing Liu, Weihao Xie, [-1pt] Yu Liang, Peilin Chen, Shiqi Wang · 2026
Read RAISE: RAG Design as an Architecture Search Problem by Zhen Chen, Yibing Liu, Weihao Xie, [-1pt] Yu Liang, Peilin Chen, Shiqi Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-30029.txt.
Introduction
Zhen Chen and five colleagues argue that tuning a retrieval-augmented generation system should be treated as architecture search, not guesswork. Their RAISE framework and benchmark standardize the search space and budget for pipeline choices such as query rewriting, chunking, retrieval depth, reranking and context shrinking. They run 13 search methods over seven text and multimodal data sets with three random seeds each. Performance proves highly task-dependent: winners on one set often fail to generalize, warning against reading overall rankings as proof of a universally best method.
Catalog record
| CREATOR | Zhen Chen, Yibing Liu, Weihao Xie, [-1pt] Yu Liang, Peilin Chen, Shiqi Wang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53272 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.30029
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
RAISE: RAG Design as an Architecture Search Problem Zhen Chen, Yibing Liu, Weihao Xie, [-1pt] Yu Liang, Peilin Chen, Shiqi Wang Abstract Retrieval-augmented generation (RAG) systems expose numerous design choices spanning query rewriting, chunking, retrieval depth, reranking, and context compression. In practice, these choices are often configured through heuristics, hindering systematic evaluation and reproducibility across settings. We argue that this challenge is best formulated as RAG architecture search. To support controlled and reproducible study of this problem, we introduce the RAG Intelligence Search Engine (RAISE), a comprehensive framework and benchmark for RAG hyperparameter optimization, which evaluates optimization methods for RAG pipelines under standardized search spaces and budgets. RAISE implements 13 search algorithms and evaluates them across seven public text and multimodal datasets using three random seeds. Our experiments show that optimization performance is highly task-dependent: methods that perform strongly on one dataset may not generalize consistently across others, cautioning against interpreting aggregate rankings as evidence of universally superior strategies. RAISE provides a common experimental substrate for fair, reproducible, and systematic research on RAG hyperparameter optimization. 1 Introduction Figure 1: Overview of the RAG Intelligence Search Engine (RAISE). The framework couples a parameterized RAG pipeline, an evaluation layer that maps configurations to task-level rewards, and a controller interface for optimization algorithms. Retrieval-augmented generation (RAG) grounds LLM outputs in external knowledge 21 , but performance depends on tightly coupled choices such as chunker, query rewriting, retrieval depth, reranking, and context compression 1 . These choices are often set through heuristics or trial-and-error tuning, making RAG optimization costly, difficult to reproduce, and challenging to compare across systems. Recent work has explored RAG configuration through AutoRAG 19 and hyperparameter impact analysis 1 , adaptive retrieval through Self-RAG 2 and Adaptive-RAG 17 , and automated tuning through AutoRAG-HP 15 and RAG-HPO analyses 29 . However, these efforts use different pipelines, datasets, search spaces, optimization methods, and evaluation protocols, leaving the field without a shared framework for comparing optimization algorithms under controlled conditions. Such a framework should expose search spaces explicitly, standardize evaluation protocols, and support matched computational budgets. It should also remain extensible to emerging optimization algorithms and benchmark tasks, similar to standardized environments used in hyperparameter optimization research such as HPOBench 12 and YAHPO Gym 31 . In this paper, we formulate RAG design as an architecture search problem. Instead of treating RAG tuning as an implementation detail, we formulate it as the selection of pipeline configurations that optimize end-to-end RAG performance under a fixed evaluation budget. This perspective connects RAG system design with random search 9 and Bayesian optimization 41 , while retaining challenges specific to RAG: heterogeneous modules, coupled design choices, multimodal pipelines, and task-dependent objective landscapes. To enable systematic study of this problem, we introduce the RAG Intelligence Search Engine (RAISE), a comprehensive framework and benchmark for RAG hyperparameter optimization (Figure 1 ). RAISE makes the problem reproducible and comparable across algorithms and settings, while supporting both LLM and multimodal LLM pipelines. The framework is designed to remain extensible beyond the benchmark tasks included in this work. New optimization algorithms can be integrated through a common controller API, while new benchmark tasks can be added by specifying datasets, corpora, and evaluation protocols without changes to the core pipeline implementation. We benchmark 13 optimization algorithms on seven environments: TriviaQA 18 , HotpotQA 47 , MS MARCO 6 , ScienceQA 26 , SQuAD v2 33 , LongBench-Multifield 5 , and LongBench-Qasper 11 , with three seeds per setting. We further conduct three-seed module ablations to identify which pipeline choices account for optimization gains under different task requirements. The results show that module effects vary with task structure: long-document retrieval benefits from query rewriting and pruning, while multi-hop reasoning relies more on retrieval depth. Likewise, long-context settings are especially sensitive to retrieval-depth control. These findings show that RAISE is not only a software framework but also a common experimental basis for studying RAG architecture search – an open problem we hope will attract broader research attention. In summary, our contributions are as follows: • We establish RAG architecture search as a benchmark setting, enabling existing and future optimization algorithms to be studied on end-to-end RAG pipelines through RAISE’s unified interface. The project code is available at https://github.com/family99chen/RAISE . • We instantiate RAISE with a shared end-to-end RAG search space, public benchmark datasets, and a unified evaluation protocol. The benchmark spans query rewriting, chunking, retrieval, reranking, pruning, and generation, exposing heterogeneous task-specific optimization challenges under matched computational budgets. • We conduct a controlled three-seed evaluation of 13 optimization algorithms across seven text and multimodal datasets, showing that optimizer performance is strongly environment-dependent. Our results motivate reporting RAG architecture search as optimizer–environment interactions rather than a universal leaderboard. 2 Related Work RAG architectures and adaptive retrieval. Retrieval-augmented generation (RAG) was introduced as a means of combining parametric language models with external retrieval 21 . Subsequent work has shown that retrieval should often be adaptive rather than fixed: Self-RAG enables a model to decide when to retrieve and how to critique retrieved evidence during generation 2 , while Adaptive-RAG selects retrieval strategies based on question complexity 17 . Such studies confirm that end-to-end RAG quality hinges on retrieval and control decisions. Their focus, however, lies in designing or learning particular RAG architectures, rather than in general optimization over configurable RAG pipelines. RAG hyperparameter optimization and benchmark environments. Several recent studies have begun to investigate hyperparameter optimization for RAG. AutoRAG-HP formulates RAG hyperparameter tuning as an online multi-armed bandit problem and employs a hierarchical controller for online adaptation 15 . 29 compare several hyperparameter optimization methods for RAG across multiple datasets and a large search space. More broadly, our formulation builds on black-box and hyperparameter optimization, including random search 9 , Bayesian optimization 41 , budget-aware bandit methods such as Hyperband 22 , and hybrid approaches such as BOHB 14 . Benchmark suites such as HPOBench 12 and YAHPO Gym 31 further demonstrate the value of standardized environments for reproducible optimizer comparison. RAISE extends this benchmark perspective to end-to-end RAG pipelines, where coupled modules, expensive evaluations, and task-dependent objectives make controlled comparison especially important. Benchmarking and evaluating RAG pipelines. A complementary line of work investigates how RAG systems should be evaluated. ARES decomposes RAG quality into context relevance, answer faithfulness, and answer relevance 39 , while RAGAs provides reference-free metrics for similar dimensions 13 . Benchmarks such as RGB 10 and CRUD-RAG 27 further expose failure modes and component sensitivities in retrieval-augmented systems. Collectively, these studies establish that RAG quality is multi-dimensional and cannot be adequately captured by a single QA score. Our goal is complementary: rather than benchmarking RAG models or evaluation metrics in isolation, we benchmark optimization methods over shared end-to-end RAG configuration spaces under fixed budgets and standardized protocols. 3 Methodology We define RAG architecture search as optimizing a parameterized end-to-end RAG pipeline under a fixed evaluation protocol. RAISE realizes this view through three components: a pipeline abstraction that specifies the admissible configurations, an evaluation layer that maps configurations to task-level scores, and a controller interface through which optimization algorithms propose configurations and receive feedback from the environment. This separation lets us vary the search strategy while holding the pipeline family, benchmark data, and scoring