Stress Testing Factual Consistency Metrics for Long-Document Summarization

Zain Muhammad Mujahid, Dustin Wright, Isabelle Augenstein · 2025

Read Stress Testing Factual Consistency Metrics for Long-Document Summarization by Zain Muhammad Mujahid, Dustin Wright, Isabelle Augenstein (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-07689.txt.

Introduction

Zain Muhammad Mujahid, Dustin Wright, and Isabelle Augenstein stress-test the tools that grade summary honesty. Factual consistency metrics were built for short texts, but long documents strain their input limits and long-range links. A metric that wobbles under harmless rewriting cannot be trusted.

Six widely used reference-free metrics face seven meaning-preserving attacks, from paraphrase and simplification to synonym swaps and compression. Sensitivity patterns show which scores stay steady and which flip without cause. Long-document summarizers are entering law, medicine, and research, so dependable honesty checks are safety equipment. The study tells builders which gauges deserve trust today.

Catalog record

CREATORZain Muhammad Mujahid, Dustin Wright, Isabelle Augenstein
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-07689

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Stress Testing Factual Consistency Metrics for Long-Document Summarization Zain Muhammad Mujahid, Dustin Wright, Isabelle Augenstein Abstract Evaluating the factual consistency of abstractive text summarization remains a significant challenge, particularly for long documents, where conventional metrics struggle with input length limitations and long-range dependencies. In this work, we systematically evaluate the reliability of six widely used reference-free factuality metrics, originally proposed for short-form summarization, in the long-document setting. We probe metric robustness through seven factuality-preserving perturbations applied to summaries, namely paraphrasing, simplification, synonym replacement, logically equivalent negations, vocabulary reduction, compression, and source text insertion, and further analyze their sensitivity to retrieval context and claim information density. Across three long-form benchmark datasets spanning science fiction, legal, and scientific domains, our results reveal that existing short-form metrics produce inconsistent scores for semantically equivalent summaries and exhibit declining reliability for information-dense claims whose content is semantically similar to many parts of the source document. While expanding the retrieval context improves stability in some domains, no metric consistently maintains factual alignment under long-context conditions. Finally, our results highlight concrete directions for improving factuality evaluation, including multi-span reasoning, context-aware calibration, and training on meaning-preserving variations to enhance robustness in long-form summarization. 1 1 1 We release all code, perturbed data, and scripts required to reproduce our results at https://github.com/zainmujahid/metricEval-longSum . 1 Introduction Figure 1: We aim to see how robust summary factuality metrics are for long and multi-document setups by applying meaning-preserving perturbations and comparing metric scores before and after these edits. Abstractive summarization has seen rapid advances with the advent of large language models (LLMs), but ensuring that generated summaries faithfully reflect the source content remains a persistent challenge Laban et al. 2024 ; Wright et al. 2025 . Summaries that read fluently can nonetheless introduce hallucinated details or omit critical facts Belém et al. 2025a , undermining their reliability for downstream tasks in domains such as medicine, law, and scientific review Asai et al. 2024 . Traditional evaluation measures like ROUGE Lin 2004 and BLEU Papineni et al. 2002 , which rely on n - gram overlap with reference summaries, are useful for measuring surface similarity but fail to capture factual consistency, since two summaries can overlap heavily in wording while still differ in correctness Maynez et al. 2020 . This gap has motivated the development of reference - free factuality evaluation metrics, which assess whether the statements in a summary are supported by the source document itself rather than by comparison with a human-written reference, using techniques such as question answering Wang et al. 2020 ; Scialom et al. 2021 ; Fabbri et al. 2022 , natural language inference Laban et al. 2022 ; Chen and Eger 2023 ; Zha et al. 2023 , or LLM - based scoring Liu et al. 2023 ; Fu et al. 2024 . While such metrics have demonstrated promise on short - document datasets, their scalability to long - form summarization is likely to be hindered by challenges unique to long context lengths Russak et al. 2024 ; Sarthi et al. 2024 ; Edge et al. 2024 . Important details may be dispersed across hundreds or thousands of tokens and thus overlooked by metrics that process only truncated inputs Laban et al. 2024 ; multi - document summaries must reconcile diverse writing styles and potentially conflicting information Asai et al. 2024 ; and the reference - free setting deprives evaluators of gold annotations, necessitating robust intrinsic evaluation protocols. Our goal is to systematically benchmark widely used factual consistency metrics under these long-document conditions, in order to reveal their robustness and limitations. Building on this foundation, we apply a stress-testing methodology established in previous work Ramprasad and Wallace 2024 to six widely-used reference-free metrics (§ 4 ) across seven factuality-preserving perturbations reflecting realistic long - form summarization phenomena. These perturbations (§ 3.1 ) broadly cover paraphrasing operations, simplification of complex constructions, synonym replacement, reduced lexical diversity, logically equivalent negations, further compression of the summary, and insertion of unrelated source sentences. They are designed to challenge metrics to distinguish genuine factual consistency from superficial cues and to maintain robustness in the face of lexical and structural variation. On top of this, we investigate the impact of long-document specific phenomena, including retrieval length and evidence diffusion Goldman et al. 2024 . Through extensive experiments using six evaluation metrics across three benchmark datasets in science fiction, legal, and scientific domains, we expose significant weaknesses in existing metrics, such as inconsistent scoring across semantically equivalent summaries. Our analysis reveals that current factuality metrics vary widely in robustness to meaning-preserving edits, with some highly sensitive to surface changes while others remain more stable. Most metrics benefit from broader retrieval context windows, though with notable domain-specific variation. We also find that metric reliability decreases for information-dense claims that overlap semantically with large portions of the source document, suggesting that current metrics struggle with compressed or globally entangled content. These insights point toward improving evaluation consistency by developing metrics that integrate multi-span reasoning and context-aware calibration. 2 Factual Consistency in Abstractive Summarization Factual consistency refers to whether a summary accurately reflects the content of the source document. While abstractive summarization systems have become increasingly fluent, they often produce factually incorrect or hallucinated statements Huang et al. 2025 . These hallucinations can range from minor misstatements to major distortions, particularly when the input is lengthy and semantically dense Belém et al. 2025b . A number of techniques have been proposed to mitigate this Gao et al. 2023 ; Qiu et al. 2023 ; Zhang et al. 2024 ; Mündler et al. 2024 ; Wang et al. 2024 . Despite these efforts, most prior work has focused on short-form summarization settings, where the task is relatively controlled. In contrast, long-form summarization requires condensing information spread across thousands of tokens, making reliable factuality evaluation substantially more difficult, especially without reference summaries or human annotations. 2.1 Factuality Evaluation Metrics To overcome the limitations of traditional reference-based metrics Maynez et al. 2020 , a range of reference-free metrics have emerged that assess the alignment between the summary and source directly. Entailment-based approaches such as FactCC Kryscinski et al. 2020 and SummaC Laban et al. 2022 use NLI models to judge whether summary sentences are supported by the source. QA-based methods like QAGS Wang et al. 2020 and QuestEval Scialom et al. 2021 evaluate factuality by generating and answering questions derived from the summary. Generation-based metrics, including BARTScore Yuan et al. 2021 and T5Score Trainin and Abend 2025 , estimate the likelihood of the summary given the source using pretrained sequence-to-sequence models. More recent tools like AlignScore Zha et al. 2023 , MiniCheck Tang et al. 2024 , and UniEval Zhong et al. 2022 aim to improve efficiency and generalization across tasks. While effective on short-document benchmarks, most of these metrics assume the full source and summary can be jointly encoded, limiting their utility for long-form inputs. Moreover, recent work shows that these metrics are often brittle, overreacting to edits like paraphrasing, reordering, or logically equivalent reformulations Ramprasad and Wallace 2024 . In this work, we study the behavior of six popular factuality metrics in long-document summarization using a retrieval-based scoring framework, and systematically evaluate their robustness to a set of

Read the complete text