M 3 Prune: Hierarchical Communication Graph Pruning for Efficient Multi-Modal Multi-Agent Retrieval-Augmented Generation

Weizi Shao, Taolin Zhang, Zijie Zhou, Chen Chen, Chengyu Wang, Xiaofeng He · 2025

Read M 3 Prune: Hierarchical Communication Graph Pruning for Efficient Multi-Modal Multi-Agent Retrieval-Augmented Generation by Weizi Shao, Taolin Zhang, Zijie Zhou, Chen Chen, Chengyu Wang, Xiaofeng He (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-19969.txt.

Introduction

Weizi Shao, Taolin Zhang, Zijie Zhou, Chen Chen, Chengyu Wang, and Xiaofeng He present M3Prune, a way to cut chatter in multi-agent retrieval systems. Teams of multimodal agents that fetch outside knowledge beat single models, but their messages pile up token cost and computing cost. That overhead blocks large-scale use.

M3Prune prunes the communication graph in a hierarchy. It first sparsifies links inside the text and image modes, keeping the edges that matter most for the task. Those key edges then build a dynamic topology across modes, and redundant edges are removed step by step. On general and domain-specific benchmarks, the method beats single-agent and multi-agent rivals while using far fewer tokens.

Catalog record

CREATORWeizi Shao, Taolin Zhang, Zijie Zhou, Chen Chen, Chengyu Wang, Xiaofeng He
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-19969

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

M 3 Prune: Hierarchical Communication Graph Pruning for Efficient Multi-Modal Multi-Agent Retrieval-Augmented Generation Weizi Shao, Taolin Zhang, Zijie Zhou, Chen Chen, Chengyu Wang, Xiaofeng He Abstract Recent advancements in multi-modal retrieval-augmented generation (mRAG), which enhance multi-modal large language models (MLLMs) with external knowledge, have demonstrated that the collective intelligence of multiple agents can significantly outperform a single model through effective communication. Despite impressive performance, existing multi-agent systems inherently incur substantial token overhead and increased computational costs, posing challenges for large-scale deployment. To address these issues, we propose a novel M ulti- M odal M ulti-agent hierarchical communication graph PRUNING framework, termed M 3 Prune . Our framework eliminates redundant edges across different modalities, achieving an optimal balance between task performance and token overhead. Specifically, M 3 Prune first applies intra-modal graph sparsification to textual and visual modalities, identifying the edges most critical for solving the task. Subsequently, we construct a dynamic communication topology using these key edges for inter-modal graph sparsification. Finally, we progressively prune redundant edges to obtain a more efficient and hierarchical topology. Extensive experiments on both general and domain-specific mRAG benchmarks demonstrate that our method consistently outperforms both single-agent and robust multi-agent mRAG systems while significantly reducing token consumption. 1 1 1 Source code is provided in the supplementary material and will be released upon acceptance. 1 Introduction While retrieval-augmented generation (RAG) has achieved significant success in the textual domain 56 ; 20 ; 9 , extending it to the multi-modal realm presents substantial challenges 17 . Traditional methods typically rely on a cascaded pipeline in which a multi-modal retriever extracts relevant evidence, which is then forwarded to LLMs for answer synthesis 18 ; 36 ; 15 . This segmented approach inherently suffers from global semantic alignment gap 24 ; 19 . The emergence of multi-modal large language models (MLLMs) presents a transformative opportunity, as these models possess a remarkable ability to understand visual-language correlations 39 ; 3 ; 12 . This capability enables MLLMs to function not only as generators but also as integrative engines for multi-modal RAG (mRAG). By jointly processing retrieved multi-modal evidence and questions within a cohesive reasoning backbone, MLLMs can synthesize information and derive coherent conclusions from retrieved knowledge 58 ; 54 ; 31 . Nonetheless, when confronted with complex, multi-faceted multi-modal questions that demand diverse expertise or deliberative reasoning, a single agent often reaches its limitations 33 ; 10 ; 49 . Figure 1 : Comparison of our approach with existing methods. (1) Closed-book Reasoning does not consider the need for external knowledge. (2) Single-agent mRAG leverages an end-to-end MLLM solution combined with a retriever to answer all questions. (3) Multi-agent mRAG constructs a fixed communication topology to obtain collaborative answers, regardless of communication efficiency. (4) Our Multi-agent Pruning approach dynamically prunes unnecessary edge connections to enhance response consistency. Recent efforts have investigated multi-agent systems built upon MLLMs, in which multiple specialized agents collaboratively communicate through structured graph topologies to distribute reasoning workloads for complex multi-modal tasks 51 ; 34 ; 32 ; 17 . However, performance improvements often come at a substantial cost, notably a significant increase in computational and token overhead. Critically, we identify the root cause ( Communication Redundancy ), which not only induces inefficiency but also directly undermines the accuracy of the final response 53 ; 17 . This redundancy is especially pronounced in multi-modal settings, where the informational requirements for processing textual cues and visual patterns inherently differ. Yet, existing methods frequently employ intra-modal communication strategies across all modalities 22 ; 44 ; 27 . As shown in Fig. 1 , in the “first flight date” question, the date expert agent is overwhelmed by irrelevant messages about engine counts and models. This noise prevents filtering out incorrect dates and obscures the correct evidence, leading to an erroneous answer. We introduce M 3 Prune , a novel framework for hierarchical communication graph pruning in multi-agent mRAG systems. Two key modules of M 3 Prune are as follows: Intra-Modal Graph Sparsification : In mRAG tasks, agents assigned different roles may hold divergent viewpoints on the same question 57 ; 50 . Consequently, there can be both cooperation and conflict among agents within the same modality, which may hinder mutual enhancement. To address this, we design a spatio-temporal message-passing technique that facilitates effective exchange of viewpoints within the visual and textual modalities, respectively. The significance of edge connections between agents in each modality is determined by the quality of the final response and the structure of the communication topology. For redundant edges connecting agents with low contribution within a modality, communication with other key agents is substantially reduced by sparsifying edge weights. Inter-Modal Graph Sparsification : Given the diverse effects of information granularity across modalities, it is essential to comprehensively integrate key clues from multiple modalities before generating the final response 7 ; 23 ; 41 ; 2 . Therefore, we aggregate the roles of agents across modalities to construct an inter-modal spatio-temporal communication topology, facilitating robust inter-modal collaboration and resolution of viewpoint conflicts. Specifically, we initialize the inter-modal communication topology based on intra-modal sparse graphs and diverse agent role correlations, integrating cross-modal text-visual and visual-text role viewpoints to learn the significance of inter-modal edges. In addition to optimizing for task performance and structural regularity, analogous to intra-modal learning, we further incorporate a modality alignment loss to ensure consistency in task understanding across different modalities during inter-modal topology sparsification. Finally, after learning edge sparsity weights within hierarchical intra- and inter-modal graphs, we progressively prune invalid edges associated with redundant agent roles across modalities. In our experiments, we evaluate M 3 Prune against a range of strong baselines, including zero-shot, single-agent, and multi-agent multi-modal settings on mRAG benchmarks: Vidoseek 42 , MultimodalQA 38 , and ScienceQA 28 , which cover general and domain-specific tasks. Results show that our approach achieves state-of-the-art performance, with an improvement of 9.4% in accuracy while significantly reducing token consumption by 15.7% compared to strong multi-modal multi-agent baselines. 2 Related Works 2.1 Multi-Modal Retrieval-Augmented Generation Previous works in mRAG can be broadly categorized into two interconnected themes: (1) Modality-Adaptive Retrieval Strategies. A key limitation of early mRAG systems is their static retrieval approach, which often resulted in unnecessary computational overhead 11 ; 28 ; 30 . Recent works have introduced dynamic, query-aware strategies to determine whether to retrieve and which modality to retrieve from. For example, EchoSight 48 and RoRA-VLM 35 search wiki articles using visual-only information, then re-rank them according to their relevance to the combined text-visual question. (2) Retrieval-Aware Pre-Training/Fine-Tuning. CoRe-MMRAG 40 reconciles inconsistencies across different knowledge sources through a four-stage progressive multi-modal retrieval pipeline. LLaVA-mR 2 AG 55 introduces a fine-tuned adaptive retriever that derives answers using two straightforward reflection operations. Wiki-LLaVA 6 employs a hierarchical retrieval pipeline to integrate external knowledge from multi-modal documents. These methods leverage single MLLM as the backbone but overlook the substantial advantage of swarm intelligence in enhancing mRAG tasks 17 . 2.2 Multi-Agent Systems for mRAG The area can be categorized into several research directions: (1) General Agentic Collaboration. A significant paradigm shift has been the move from static retrieval and generation to agentic designs, transforming passive pipelines into active, decision-making systems 50 ; 26 ; 8

Read the complete text