Long-Form Information Alignment Evaluation Beyond Atomic Facts
Danna Zheng, Mirella Lapata, Jeff Z. Pan · 2025
Read Long-Form Information Alignment Evaluation Beyond Atomic Facts by Danna Zheng, Mirella Lapata, Jeff Z. Pan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-15792.txt.
Introduction
Danna Zheng, Mirella Lapata, and Jeff Pan expose a new way long texts can mislead fact checkers. Their twenty twenty-five paper builds MontageLie, a benchmark of deceptive stories made only of true statements. No single fact is false, yet the joined tale points the wrong way.
Both coarse LLM judges and fine fact-by-fact tools fell for the trick, with weak detection scores. The missing piece is links between facts, above all event order. Their fix, DoveScore, checks facts and their order together. It beat past fine-grained methods by a clear margin. Trust in auto-checks for long reports needs this wider lens.
Catalog record
| CREATOR | Danna Zheng, Mirella Lapata, Jeff Z. Pan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-15792
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Long-Form Information Alignment Evaluation Beyond Atomic Facts Danna Zheng, Mirella Lapata, Jeff Z. Pan Abstract Information alignment evaluators are vital for various NLG evaluation tasks and trustworthy LLM deployment, reducing hallucinations and enhancing user trust. Current fine-grained methods, like FactScore, verify facts individually but neglect inter-fact dependencies, enabling subtle vulnerabilities. In this work, we introduce MontageLie , a challenging benchmark that constructs deceptive narratives by “montaging” truthful statements without introducing explicit hallucinations. We demonstrate that both coarse-grained LLM-based evaluators and current fine-grained frameworks are susceptible to this attack, with AUC-ROC scores falling below 65%. To enable more robust fine-grained evaluation, we propose DoveScore , a novel framework that jointly verifies factual accuracy and event-order consistency. By modeling inter-fact relationships, DoveScore outperforms existing fine-grained methods by over 8%, providing a more robust solution for long-form text alignment evaluation. Our code and datasets are available at https://github.com/dannalily/DoveScore . 1 Introduction Previous studies Huang et al. 2025 have identified a critical issue in LLM deployment: factual inaccuracies, commonly known as hallucinations. To address this challenge, recent approaches Asai et al. 2024 ; Roy et al. 2024 ; Ji et al. 2023 ; Manakul et al. 2023 introduce reflection mechanisms that perform post-hoc verification by comparing generated texts against retrieved documents or given contexts, and subsequently regenerating erroneous segments when necessary. At the core of these approaches lies the information alignment evaluator, which determines whether a target text accurately aligns with a source text. Deng et al. 2021 highlight the importance of information alignment evaluation across various NLG evaluation tasks. Consequently, developing robust information alignment evaluators is essential. Unlike fact-checking tasks Si et al. 2024 ; Ma et al. 2024 , which typically involve short, sentence-level comparisons, information alignment evaluation often requires reasoning over extended contexts Zha et al. 2023 , where both the source and target may span multiple paragraphs. While recent advances in long-context LLMs Liu et al. 2025 support coarse-grained alignment evaluation across entire texts, many real-world applications Min et al. 2023 ; Zhang et al. 2024 ; Ye et al. 2025 demand fine-grained evaluators that can assess individual factual units and pinpoint detailed inaccuracies. Figure 1: The figure illustrates the limitation of existing fine-grained evaluators such as FactScore and AlignScore, which struggle to detect lies composed of the exact small units that make up the truth. Existing fine-grained frameworks Min et al. 2023 ; Song et al. 2024b ; Wei et al. 2024 typically decompose the target text into atomic facts, verify each against retrieved evidence, and then aggregate the results into an overall judgment. While effective at identifying surface-level errors, this approach has a critical blind spot: it overlooks relationships and dependencies between facts. Even when all individual statements are accurate, reordering them can reverse implied causal chains and mislead readers. As illustrated in Figure 1 , the Lie version suggests that Amy’s outing with John triggered Mike’s violence, subtly shifting blame onto her. The Truth reveals that the violence preceded the breakup and her subsequent actions. By altering the sequence of accurate statements, the text introduces a discourse-level manipulation that distorts causality without introducing any falsehoods. However, existing fine-grained evaluators are inherently incapable of detecting such manipulations. To investigate this vulnerability, we introduce MontageLie , a novel benchmark designed to test the limitations of current information alignment evaluators. Drawing inspiration from the cinematic concept of montage 1 1 1 https://en.wikipedia.org/wiki/Montage_(filmmaking) , which creates new meaning by rearranging real scenes in novel sequences, MontageLie constructs "montage-style lies": deceptive texts composed entirely of truthful statements, deliberately reordered to imply misleading narratives. These manipulations do not introduce fabricated facts but instead distort causal relationships by altering the sequence of events. To systematically assess model robustness, our benchmark includes four levels of difficulty, each reflecting increasing subtlety in the causal distortion. Such rearranging strategies, while factually accurate at the level of small textual units, exploit deceptive tactics commonly used in human communication and pose a sophisticated challenge that current evaluators are ill-equipped to detect. They also represent a realistic and underexplored attack vector in adversarial prompting Kim et al. 2024 and misinformation campaigns Hu et al. 2025 ; Macko et al. 2025 . Experimental results demonstrate that existing fine-grained frameworks, as well as state-of-the-art long-context LLMs in coarse-grained evaluation settings, struggle to identify these subtle manipulations, achieving AUC-ROC scores consistently below 65%. Besides, to address the limitations of current fine-grained evaluators, we propose DoveScore ( D escriptive and O rdered-Event Ve rification Score ), a fine-grained evaluation framework that explicitly incorporates both atomic factual accuracy and event ordering consistency. DoveScore decomposes target texts into descriptive and event-based facts, verifying their individual correctness and event sequencing against the source text, then computing a weighted precision score. Experimental evaluations show that DoveScore outperforms existing fine-grained methods by over 8%. We anticipate that both our benchmark, MontageLie , and our proposed evaluation method, DoveScore , will offer valuable insights and make a meaningful contribution to the ongoing development of robust and reliable information alignment evaluators. 2 Background and Related Works 2.1 Definition of Information Alignment The concept of information alignment (also termed factual consistency) was first formally defined by Deng et al. 2021 , who proposed a unified evaluation framework for NLG tasks and identified information alignment as a core principle. Initially, it was defined at the token level: each token in the target should be supported by the source. Later, Zha et al. 2023 introduced a more practical sequence-level definition: Text b aligns with source a if all information in b appears accurately in a without contradiction. An Information Alignment Evaluator can be formalized as: f:(\boldsymbol{a},\boldsymbol{b})\rightarrow y (1) where higher y indicates stronger alignment of b with respect to a . 2.2 Benchmarks Early benchmarks mainly addressed sentence-level alignment, such as FEVER Thorne et al. 2018 , FEVEROUS Aly et al. 2021 , and AVERITEC Schlichtkrull et al. 2023 . More recent efforts target longer texts and diverse domains. SummaC Laban et al. 2022 aggregates six summarization-focused datasets, while TRUE Honovich et al. 2022 combines 11 datasets across summarization, dialogue, fact verification, and paraphrasing. The latest, LLM-AggreFact Tang et al. 2024a , curates recent datasets such as AGGREFACT Tang et al. 2023 , TOFUEVAL Tang et al. 2024b , and ClaimVerify Liu et al. 2023a . Despite broader coverage, these benchmarks mostly focus on unsupported or contradictory claims. Our proposed dataset, MontageLie , introduces a harder case: each individual claim aligns with the source, yet their combination yields a misleading narrative. 2.3 Evaluators Coarse-grained Evaluators These methods assess alignment holistically, typically via overlap or semantic similarity. Traditional metrics like BLEU Papineni et al. 2002 , ROUGE Lin 2004 , and METEOR Banerjee and Lavie 2005 emphasize lexical overlap, while embedding-based metrics (e.g., BERTScore Zhang et al. 2020 , BartScore Yuan et al. 2021 ) and fine-tuned models (e.g., BLEURT Sellam et al. 2020 , FactCC Kryscinski et al. 2020 ) capture semantics more effectively. However, most struggle with long-form texts due to model limitations. More recently, LLM-as-Evaluator approaches Liu et al. 2023b ; Luo et al. 2023 prompt LLMs directly for quality scores, enabling more flexible, semantically rich assessments. Fine-grained Evaluators Fine-grained methods offer interpretable, diagnostic feedback. QA-based approaches (e.g., QuestEval Scialom et al.