MA-RAG: Multi-Agent Retrieval-Augmented Generation via Collaborative Chain-of-Thought Reasoning

Thang Nguyen, Peter Chin, Yu-Wing Tai · 2025

Read MA-RAG: Multi-Agent Retrieval-Augmented Generation via Collaborative Chain-of-Thought Reasoning by Thang Nguyen, Peter Chin, Yu-Wing Tai (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20096.txt.

Introduction

Thang Nguyen, Peter Chin, and Yu-Wing Tai present a 2025 paper on question answering with search. Their system, MA-RAG, splits the work among four agents. One plans, one sets the steps, one pulls facts, and one writes the answer. Each agent does one stage and shares its thoughts with the next.

On hard question sets that need more than one step, the system beat single models and older search methods. Even a small base model with the system beat larger models alone. Tests that removed parts showed the plan and pull stages matter most. The system also did well on medical questions with no special training. Its steps stay open to read and check.

Catalog record

CREATORThang Nguyen, Peter Chin, Yu-Wing Tai
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-20096

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MA-RAG: Multi-Agent Retrieval-Augmented Generation via Collaborative Chain-of-Thought Reasoning Thang Nguyen, Peter Chin, Yu-Wing Tai Abstract We present MA-RAG, a Multi-Agent framework for Retrieval-Augmented Generation (RAG) that addresses the inherent ambiguities and reasoning challenges in complex information-seeking tasks. Unlike conventional RAG methods that rely on either end-to-end fine-tuning or isolated component enhancements, MA-RAG orchestrates a collaborative set of specialized AI agents: Planner, Step Definer, Extractor, and QA Agents, to tackle each stage of the RAG pipeline with task-aware reasoning. Ambiguities may arise from underspecified queries, sparse or indirect evidence in retrieved documents, or the need to integrate information scattered across multiple sources. MA-RAG mitigates these challenges by decomposing the problem into subtasks, such as query disambiguation, evidence extraction, and answer synthesis, and dispatching them to dedicated agents equipped with chain-of-thought prompting. These agents communicate intermediate reasoning and progressively refine the retrieval and synthesis process. Our design allows fine-grained control over information flow without any model fine-tuning. Crucially, agents are invoked on demand, enabling a dynamic and efficient workflow that avoids unnecessary computation. This modular and reasoning-driven architecture enables MA-RAG to deliver robust, interpretable results. Experiments on multi-hop and ambiguous QA benchmarks demonstrate that MA-RAG outperforms state-of-the-art training-free baselines and rivals fine-tuned systems, validating the effectiveness of collaborative agent-based reasoning in RAG. Figure 1 : Architectural Comparison of MA-RAG and Prior RAG Methods. a) A naive RAG system performs one-shot retrieval followed by direct answer generation. b) Enhanced systems incorporate post-retrieval processing such as document re-ranking or summarization. c) Iterative systems interleave retrieval and reasoning via query rewriting or multi-step refinement, yet often lack explicit modularity and planning. d) In contrast, MA-RAG adopts a collaborative multi-agent architecture where specialized agents handle distinct stages of the RAG pipeline, such as query disambiguation, targeted evidence extraction, and answer synthesis, using chain-of-thought reasoning. Agents are invoked dynamically and on demand, enabling fine-grained document analysis and step-by-step resolution of ambiguities, resulting in a more robust, interpretable, and efficient retrieval-to-generation process. 1 Introduction Recent advances in natural language processing have driven the development of Retrieval-Augmented Generation (RAG) models, which aim to enhance the factual accuracy and contextual relevance of generated text by integrating external knowledge sources ( Lewis et al. 2020 ; Guu et al. 2020 ; Izacard and Grave 2021 ; Lin et al. 2024 ) . These systems address core limitations of Large Language Models (LLMs), such as outdated knowledge ( Zhang et al. 2023b ; Kasai et al. 2023 ) and poor generalization to domain-specific queries ( Siriwardhana et al. 2023 ; Xiong et al. 2024 ) , by retrieving top- k documents from an external corpus ( Formal et al. 2022 ; Izacard et al. 2022 ; Wang et al. 2022a ) to ground the model’s output in relevant evidence. Prior research in RAG has largely concentrated on optimizing three key components—retrieval, augmentation, and generation ( Gao et al. 2024 ; Fan et al. 2024 ) (Figure 1 (a)). Retrieval strategies span sparse methods ( Jones 1972 ; Robertson and Zaragoza 2009 ) and dense retrieval ( Reimers and Gurevych 2019 ; Karpukhin et al. 2020 ) , each with respective weaknesses such as lexical gaps ( Berger et al. 2000 ) or retrieval failure on out-of-distribution and multi-hop queries ( Dai et al. 2023 ) . Augmentation methods often rely on post-retrieval processing such as re-ranking or document summarization ( Chen et al. 2020 ; Glass et al. 2022 ; Ma et al. 2024 ) (Figure 1 (b)) to improve input quality for the LLM, but add latency and may still fail to filter irrelevant or misleading evidence. More sophisticated approaches introduce iterative retrieval or query rewriting ( Jiang et al. 2023b ; Asai et al. 2024 ) (Figure 1 (c)), but commonly assume the input query is well-formed and overlook the broader reasoning process across the pipeline. Across retrieval, augmentation, and generation, most existing methods treat components in isolation, failing to resolve ambiguities and reasoning gaps that span multiple stages, such as vague queries, incomplete retrievals, or scattered evidence, ultimately limiting robustness and transparency in complex QA scenarios. To address these challenges, we propose MA-RAG , a modular, training-free Multi-Agent framework that performs step-by-step reasoning across the entire RAG pipeline (Figure 1 (d)). MA-RAG views the RAG process as a collaborative effort among specialized agents, each responsible for a specific subtask such as query disambiguation and task decomposition (Planner), document retrieval (Step Definer and Retrieval Tool), evidence extraction (Extractor), and answer synthesis (QA Agent). Rather than invoking all agents uniformly, MA-RAG adopts an on-demand strategy, calling only the necessary agents depending on the ambiguity and complexity at each step. Each agent is guided by chain-of-thought prompting, enabling explicit intermediate reasoning that improves interpretability and task alignment. For instance, agents can decompose an underspecified query into concrete sub-questions, retrieve documents tailored to each subtask, distill targeted evidence from multiple sources, and compose coherent answers by integrating dispersed information. This multi-agent design not only improves robustness to ambiguous and multi-hop questions, but also provides fine-grained control over the information flow in RAG, all without requiring model fine-tuning. Empirically, MA-RAG achieves new state-of-the-art performance on multiple open-domain QA benchmarks, including NQ, HotpotQA, TriviaQA, and 2WikimQA, consistently outperforming both standalone LLMs and strong RAG baselines across model scales. Ablation studies further reveal the importance of the modular architecture: the planner agent is essential for multi-hop reasoning, while the extractor agent significantly improves grounding by filtering irrelevant content. Moreover, our analysis shows that model size matters most for answer generation, planning and evidence extraction, while lighter-weight models can be effectively used for retrieval components, enabling more efficient deployments. These results highlight the effectiveness and flexibility of our multi-agent framework in tackling complex QA tasks without additional supervision or domain-specific training. Our key contributions are as follows: • We introduce MA-RAG , a modular multi-agent framework that performs reasoning-driven RAG via structured collaboration between agents, enabling fine-grained handling of ambiguity and complex queries. • MA-RAG operates entirely without model fine-tuning, offering a general and adaptable solution that outperforms or matches strong baselines across multiple QA datasets and LLM backends. 2 Related Works Large Language Models (LLMs) has spurred significant advancements over the past few years. Beginning with GPT-1 ( Radford et al. 2018 ) on the Transformer architecture ( Vaswani et al. 2017 ) , subsequent models like GPT-2 ( Radford et al. 2019 ) , GPT-3 ( Brown et al. 2020 ) , and the latest GPT-4 ( OpenAI 2024 ) have significantly enhance capabilities in text understanding and generation. Beyond the GPT series, models such as Mistral ( Jiang et al. 2023a ) , Gemini ( Gemini Team 2023 ) , and LLaMA ( ( Touvron et al. 2023a ) , Touvron et al. 2023b ) demonstrate robust performance across various tasks like question-answering and entity recognition ( Zhao et al. 2023 ) . Training LLMs involves unsupervised pre-training, supervised fine-tuning, and alignment with human feedback, yet challenges persist in domain-specific tasks ( Kandpal et al. 2023 ) . Techniques like PEFT ( Houlsby et al. 2019a ) optimize fine-tuning efficiency, with emerging methods such as prompt-based learning ( Lester et al. 2021 ; Li and Liang 2021 ) , adapters ( Houlsby et al.

Read the complete text