What are they talking about? Benchmarking Large Language Models for Knowledge-Grounded Discussion Summarization
Weixiao Zhou, Junnan Zhu, Gengyao Li, Xianfu Cheng, Xinnian Liang, Feifei Zhai, Zhoujun Li · 2025
Read What are they talking about? Benchmarking Large Language Models for Knowledge-Grounded Discussion Summarization by Weixiao Zhou, Junnan Zhu, Gengyao Li, Xianfu Cheng, Xinnian Liang, Feifei Zhai, Zhoujun Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-12474.txt.
Introduction
Weixiao Zhou and six co-authors set a new test for chat summary tools that need shared background. In group talks people skip context and use vague refs. A plain summary then confuses readers who lack that shared past.
Their task asks for two parts, a background brief and a clear view brief with refs spelled out. They built the first test set with news and talk pairs and expert notes. Tests of many large models show the task is hard. Models drop key facts, keep junk, and miss vague refs.
Catalog record
| CREATOR | Weixiao Zhou, Junnan Zhu, Gengyao Li, Xianfu Cheng, Xinnian Liang, Feifei Zhai, Zhoujun Li |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-12474
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
What are they talking about? Benchmarking Large Language Models for Knowledge-Grounded Discussion Summarization Weixiao Zhou, Junnan Zhu, Gengyao Li, Xianfu Cheng, Xinnian Liang, Feifei Zhai, Zhoujun Li Abstract In this work, we investigate the performance of LLMs on a new task that requires combining discussion with background knowledge for summarization. This aims to address the limitation of outside observer confusion in existing dialogue summarization systems due to their reliance solely on discussion information. To achieve this, we model the task output as background and opinion summaries and define two standardized summarization patterns. To support assessment, we introduce the first benchmark comprising high-quality samples consistently annotated by human experts and propose a novel hierarchical evaluation framework with fine-grained, interpretable metrics. We evaluate 12 12 LLMs under structured-prompt and self-reflection paradigms. Our findings reveal: (1) LLMs struggle with background summary retrieval, generation, and opinion summary integration. (2) Even top LLMs achieve less than 69 69 % \% average performance across both patterns. (3) Current LLMs lack adequate self-evaluation and self-correction capabilities for this task. 1 1 1 Our benchmark is available at zhouweixiao/KGDS 1 Introduction Dialogue summarization aims to distill main topics and interactions from a dialogue into a concise and faithful summary Jia et al. 2023 ; Rennard et al. 2023 ; Kirstein et al. 2024 . Current methods typically rely exclusively on dialogue content across dataset construction Gliwa et al. 2019 ; Chen et al. 2021 ; Zhu et al. 2021 ; Hu et al. 2023 , summarization methodologies Lin et al. 2022 ; Zhou et al. 2023 ; Tian et al. 2024 , and evaluation paradigms Wang et al. 2022 ; Gao and Wan 2022 ; Zhu et al. 2023 ; Gao et al. 2023 ; Tang et al. 2023 ; Tang et al. 2024b ; Ramprasad et al. 2024 . These efforts implicitly assume that " Information in the dialogue is sufficiently complete to support generation of a clearly understandable summary for outside observers ." However, we find this assumption has fundamental limitations and often breaks down, particularly in discussions involving shared background knowledge . Such discussions exhibit two main traits: (1) Information Omission and Implicit Reference : Participants naturally skip mutually known details and frequently use pronouns or phrases to refer to entities or facts within the contextual background. (2) Personal Opinion : Unlike straightforward information interactions in general dialogues, these discussions focus on exchanging viewpoints, with participants expressing personal opinions from various positions or perspectives (see Figure 1 ). These characteristics make understanding heavily reliant on background knowledge. Consequently, traditional dialogue summarization pattern maintains this dependency, leading to confusion for outsiders unfamiliar with the context , leaving them wondering: " What are they talking about? ". Figure 1: An overall example. Gray blocks in paragraphs represent the crucial background details omitted by discussion. Cyan blocks in discussion indicate referential pronouns or phrases with their corresponding referents. Pink blocks in traditional dialogue summary denote content that may cause confusion for outside observers. The summarization patterns of KGDS are formalized as EBS-AOS and ABS-AOS. Our evaluation framework is able to comprehensively and accurately assess sub-summaries and both patterns via fine-grained and interpretable metrics. To address this issue, we introduce K nowledge- G rounded D iscussion S ummarization ( KGDS ), a novel task specifically designed to combine shared background knowledge with discussion content to produce observer-friendly summaries . We determine that an effective summary must achieve two complementary objectives: (1) bridge the knowledge gap between outsiders and participants by providing necessary background-supporting information ; (2) present clear personal opinions of the participants with clarified implicit references . We formalize these requirements by modeling the task output as two components: background summary and opinion summary . The background summary, which retrieves or condenses relevant contextual information, can be either extractive or abstractive , while the opinion summary integrates clarified viewpoints and is inherently abstractive . Thus, we define two standardized summarization patterns: EBS-AOS and ABS-AOS (refer to Figure 1 ). To systematically investigate the challenges of KGDS and evaluate LLM capabilities in this task, we make two significant contributions. First , we construct the first KGDS benchmark containing 100 high-quality, multi-domain samples from realistic news discussions through strict consistency-controlled annotation protocols. Each sample comprises structured shared background knowledge, multi-turn human discussion, and expert-annotated evaluation components at multiple granularities. Second , we propose a novel hierarchical evaluation framework with fine-grained and interpretable metrics (as shown in Figure 1 ). Specifically: (1) For extractive background summary, we utilize retrieval metrics to evaluate recall, precision, and f1-score of supporting paragraphs. (2) For abstractive background summary, inspired by the minimal granularity of atomic facts Liu et al. 2023b and the excellent human-alignment of LLM-based fact verification Wei et al. 2024 , we assess quality via verification of key supporting fact inclusion and nonsupporting fact exclusion , proposing three metrics that multi-dimensionally quantify background coverage, focus, and summary overall performance. (3) For abstractive opinion summary, we evaluate quality by verifying coverage of clear atomic opinions , where each opinion is minimized and implicit references clarified . For uncovered opinions, we define five error types and perform fine-grained error detection. (4) For pattern overall performance, we use geometric mean for evaluation, which aligns with the KGDS workflow and reflects the joint fulfillment rate of both sub-summaries. We evaluate 12 leading LLMs under single-turn structured-prompt Li et al. 2023 and multi-turn self-reflection Shinn et al. 2023 paradigms. Our analysis reveals several critical insights: (1) LLMs struggle with precision-recall trade-off in background summary retrieval; (2) LLMs frequently miss key facts and retain irrelevant facts in background summary generation; (3) LLMs encounter difficulties with clarifying implicit references in opinion summary integration; (4) Even advanced LLMs achieve less than 69 \% average performance across both patterns; (5) LLMs lack sufficient self-evaluation and -correction capabilities for KGDS. Our contributions are as follows: (1) We introduce a novel KGDS task and establish two standardized summarization patterns. (2) We construct the first benchmark for this task and propose a comprehensive hierarchical evaluation framework with interpretable metrics. (3) Our extensive evaluation reveals specific challenges that current LLMs face with KGDS, providing valuable guidance for future improvements in coarse-grained retrieval, fine-grained generation, and knowledge integration. 2 Related Work Dialogue Summarization. Dialogue summarization is widely used in practical applications Lin et al. 2021 ; Hu et al. 2023 . Earlier approaches concentrated on feature modeling Chen and Yang 2021 ; Fang et al. 2022 and pre-training methods Zou et al. 2021 ; Zhong et al. 2022a . The emergence of LLMs has greatly enhanced dialogue summarization Wang et al. 2023a ; Tian et al. 2024 ; Lu et al. 2025 ; Zhu et al. 2025 , with recent research focusing on evaluating their effectiveness in this field Liu et al. 2024b ; Ramprasad et al. 2024 ; Tang et al. 2024b . However, these studies often overlook the importance of background knowledge . Knowledge-Grounded Dialogue Generation. The task involves generating responses that interact with external knowledge sources like documents Wang et al. 2023b ; Wang et al. 2024 , databases Zhou et al. 2018 ; Zhang et al. 2020 , or retrieval-augmented systems Lewis et al. 2020 ; Chen et al. 2024 . The main