Dial-SummEr : A Structured Evaluation Framework of Hierarchical Errors in Dialogue Summaries

Sahana Ramnath, Nima Chitsazan, Mingyang Zhou, Chia-Hsuan Lee, Shi-Xiong Zhang, Stephen Rawls, Sambit Sahu, Sangwoo Cho, Xiang Ren, Genta Indra Winata, Akshaj Kumar Veldanda · 2026

Read Dial-SummEr : A Structured Evaluation Framework of Hierarchical Errors in Dialogue Summaries by Sahana Ramnath, Nima Chitsazan, Mingyang Zhou, Chia-Hsuan Lee, Shi-Xiong Zhang, Stephen Rawls, Sambit Sahu, Sangwoo Cho, Xiang Ren, Genta Indra Winata, Akshaj Kumar Veldanda (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-08149.txt.

Description

Summarizing a conversation is harder than summarizing an article: information scatters across speakers and turns, and the summary rewrites first- and second-person speech in third person. Sahana Ramnath and ten co-authors offer DIAL-SUMMER, a method for evaluating dialogue summaries with a structured taxonomy of layered errors.

Errors are classified at two levels: dialogue level, concerning speakers and turns broadly, and within-turn level, concerning information inside a single turn. The method comes with a dataset of manually annotated dialogue summaries. Summarization teams and product teams gain an evaluation tool that distinguishes kinds of mistakes earlier metrics blurred as one, pointing builders toward the specific failures to fix.

Catalog record

CREATORSahana Ramnath, Nima Chitsazan, Mingyang Zhou, Chia-Hsuan Lee, Shi-Xiong Zhang, Stephen Rawls, Sambit Sahu, Sangwoo Cho, Xiang Ren, Genta Indra Winata, Akshaj Kumar Veldanda
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-08149

Browse the full library · About the Tower of Babel