Multi-LLM Text Summarization
Jiangnan Fang, Cheng-Tse Liu, Jieun Kim, Yash Bhedaru, Ethan Liu, Nikhil Singh, Nedim Lipka, Puneet Mathur, Nesreen K. Ahmed, Franck Dernoncourt, Ryan A. Rossi, Hanieh Deilamsalehy · 2024
Read Multi-LLM Text Summarization by Jiangnan Fang, Cheng-Tse Liu, Jieun Kim, Yash Bhedaru, Ethan Liu, Nikhil Singh, Nedim Lipka, Puneet Mathur, Nesreen K. Ahmed, Franck Dernoncourt, Ryan A. Rossi, Hanieh Deilamsalehy (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-15487.txt.
Introduction
Jiangnan Fang leads a team letting several language models summarize together and pick the best draft. Their paper appeared in two thousand twenty-four. Single models summarize unevenly, missing points or adding noise. A group can draft diversely, then judge. Each round has two steps: generation and evaluation. The team tests two ways of organizing the group.
In the centralized scheme, one model grades all drafts and picks a winner. In the decentralized scheme, every model takes part in grading. Many models generate varied summaries in both designs. The group results beat single-model baselines by a clear margin. Iteration across rounds polishes the text further. Summarization gains a committee approach with measured benefits.
Catalog record
| CREATOR | Jiangnan Fang, Cheng-Tse Liu, Jieun Kim, Yash Bhedaru, Ethan Liu, Nikhil Singh, Nedim Lipka, Puneet Mathur, Nesreen K. Ahmed, Franck Dernoncourt, Ryan A. Rossi, Hanieh Deilamsalehy |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-15487
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Multi-LLM Text Summarization Jiangnan Fang, Cheng-Tse Liu, Jieun Kim, Yash Bhedaru, Ethan Liu, Nikhil Singh, Nedim Lipka, Puneet Mathur, Nesreen K. Ahmed, Franck Dernoncourt, Ryan A. Rossi, Hanieh Deilamsalehy Abstract In this work, we propose a Multi-LLM summarization framework, and investigate two different multi-LLM strategies including centralized and decentralized. Our multi-LLM summarization framework has two fundamentally important steps at each round of conversation: generation and evaluation. These steps are different depending on whether our multi-LLM decentralized summarization is used or centralized. In both our multi-LLM decentralized and centralized strategies, we have k k different LLMs that generate diverse summaries of the text. However, during evaluation, our multi-LLM centralized summarization approach leverages a single LLM to evaluate the summaries and select the best one whereas k k LLMs are used for decentralized multi-LLM summarization. Overall, we find that our multi-LLM summarization approaches significantly outperform the baselines that leverage only a single LLM by up to 3x. These results indicate the effectiveness of multi-LLM approaches for summarization. 1 Introduction Large language models (LLMs) have been shown to have the potential to produce high-quality summaries Chowdhery et al. (2022) ; Zhang et al. (2023) ; Goyal et al. (2023) ; Pu et al. (2023b) . However, despite the remarkable progress in LLM-based summarization, limitations still exist for documents where useful information may be sparsely distributed throughout the text. Research by Liu et al. (2023) highlights that a naive application of LLMs may overlook critical details or fail to grasp the holistic meaning of a document, indicating the need for more refined methods. To address this, recent efforts have explored prompt-engineering techniques to guide LLMs towards producing better summaries Adams et al. (2023) . These techniques, while promising, still face limitations in consistently delivering high-quality summaries across different document types and structures. Instead of relying solely on a single model or simple prompt-engineering methods, we propose an approach novel to the summarization domain that focuses on aggregating the collective strengths of multiple LLMs. By combining the capabilities of multiple models with a diverse set of knowledge bases, we show it’s possible to achieve more robust summaries across domains. Summary of Main Contributions. The main contributions of this work are as follows: • We propose the first framework for multi-LLM text summarization and investigate two topologies: centralized and decentralized. • We find that multi-LLM text summarization often performs better than using a single LLM for summarization, and we show that the best performing method in the framework aligns with human judgments. • We conduct experiments on how prompting, number of LLMs, and various combinations of generating and evaluating LLMs can affect quality of summaries in the multi-LLM setup. 2 Related Work 2.1 Summarization Recent advancements in summarization have increasingly leveraged large language models (LLMs), moving beyond fine-tuned transformer models like Pegasus, BART, and T5. Studies consistently show that LLMs can generate summaries with higher coherence, relevance, and factual accuracy, often rivaling or surpassing human-written summaries Goyal et al. (2023) ; Zhang et al. (2023) ; Pu et al. (2023b) . For example, Goyal et al. (2023) demonstrated that GPT-3 (text-davinci-002) produced summaries preferred by human evaluators over fine-tuned models like Pegasus and BRIO on structured datasets such as CNN/DM Nallapati et al. (2016) and XSUM Narayan et al. (2018) . Similarly, Zhang et al. (2023) emphasized the importance of instruction tuning in achieving superior zero-shot performance for summarization tasks. Pu et al. (2023b) further highlighted improved factual consistency and reduced hallucinations when using LLMs. While these studies validate the potential of LLMs in summarizing well-structured texts, they may falter for inputs lacking clear structural cues and exhibiting greater complexity. Research focusing on longer text summarization, such as Keswani et al. (2024) , employed semantic clustering and multi-stage summarization with LLaMA2 to manage lengthy inputs. However, such approaches often rely on predefined hierarchical processing strategies that may oversimplify the nuanced relationships within the text. Moreover, as Liu et al. (2023) noted, LLMs tend to neglect content from the middle sections of longer documents, resulting in incomplete or unbalanced summaries. Our work aims to improve performance for both long and short text summarization, and it builds upon aforementioned foundations by proposing a multi-LLM framework designed to overcome these shortcomings through information exchange and collaborative synthesis. 2.2 Multi-LLM The concept of leveraging multiple LLMs collaboratively has gained traction in recent research, particularly for tasks requiring complex reasoning and factual accuracy. For instance, Liang et al. (2024) introduced the Multi-Agent-Debate (MAD) framework, where LLMs engage in iterative debates to refine their reasoning. This framework demonstrated that a multi-agent GPT-3.5-Turbo setup outperformed GPT-4 on reasoning datasets. Similarly, Chen et al. (2024) proposed RECONCILE, a framework where LLMs collaboratively refine answers and explanations, achieving significant improvements over single-agent systems. Li et al. (2024) extended this line of research by optimizing agent connections, showing that sparse networks can maintain performance while reducing computational overhead. Although these studies reveal the potential of multi-LLM approaches, their focus remains on structured reasoning tasks, such as question answering and fact-checking. They have not been adequately explored in the context of synthesizing distributed information, addressing content imbalances, and preserving the coherence of summaries across extended texts. We hope to bridge this gap by adapting multi-LLM frameworks to the domain of document summarization, addressing limitations of both single LLM and traditional hierarchical techniques, and positioning multi-LLM summarization as a promising solution. Multi-LLM Summarization Framework General Mechanism Stage Centralized ( Sec. 4 ) Single-Round ( Sec. 4.1 ) Generation (§ 4.1.1 ) Evaluation (§ 4.1.2 ) Conversational ( Sec. 4.2 ) Generation (§ 4.2.1 ) Evaluation (§ 4.2.2 ) Decentralized ( Sec. 5 ) Single-Round ( Sec. 5.1 ) Generation (§ 5.1.1 ) Evaluation (§ 5.1.2 ) Conversational ( Sec. 5.2 ) Generation (§ 5.2.1 ) Evaluation (§ 5.2.2 ) Table 1: Overview of Multi-LLM Summarization Framework (Sections 4 - 5 ). 3 Multi-LLM Summarization Framework In this work, we propose a novel multi-LLM summarization framework that leverages multiple large language models to enhance summarization quality of long document input. Through the distribution of generation and evaluation of candidate summaries across multiple models, our framework aims to provide better summaries than single LLM methods, leveraging expertise from different models. We present two interaction topologies, centralized and decentralized , to guide the collaboration, evaluation, and refinement of summaries between LLMs. Visually these two methods can be represented at a high level in Figure 1 . In the datasets we test, articles are typically tens of thousands of words long and exceed the context window of most standard LLMs. To handle this, we establish a two stage process that involves chunking the source document, independently summarizing each chunk of the source document, and then applying a second round of chunking and summarization on the concatenated intermediate results. Throughout both these stages, both frameworks allow multiple LLMs to collaborate and converge on a single final high quality summary of the entire original reference document. Table 1 provides an overview of our framework’s four main variations. (a) Centralized (b) Decentralized Figure 1: Centralized and Decentralized approaches using a 5-LLM example. Similar topologies can be applied to any (" k ") number of LLMs. In centralized interactions, all models communicate with a central model;