Optimising Factual Consistency in Summarisation via Preference Learning from Multiple Imperfect Metrics
Yuxuan Ye, Raul Santos-Rodriguez, Edwin Simpson · 2026
Read Optimising Factual Consistency in Summarisation via Preference Learning from Multiple Imperfect Metrics by Yuxuan Ye, Raul Santos-Rodriguez, Edwin Simpson (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26840.txt.
Introduction
Ye, Santos-Rodriguez, and Simpson improve factual consistency in summaries by learning from several weak factuality metrics at once. Single metrics mislead, but their combination captures more error types, so the team maps metric scores to preferences and drops cases where metrics disagree sharply. Training on lexically similar summary pairs from varied decoding teaches the model to spot subtle factual shifts. Gains hold across model sizes, with small models nearly matching larger ones.
Catalog record
| CREATOR | Yuxuan Ye, Raul Santos-Rodriguez, Edwin Simpson |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42089 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.26840
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Optimising Factual Consistency in Summarisation via Preference Learning from Multiple Imperfect Metrics Yuxuan Ye, Raul Santos-Rodriguez, Edwin Simpson Abstract Reinforcement learning with evaluation metrics as rewards is widely used to enhance specific capabilities of language models. However, for tasks such as factually consistent summarisation, existing metrics remain underdeveloped, limiting their effectiveness as signals for shaping model behaviour. While individual factuality metrics are unreliable, their combination can more effectively capture diverse factual errors. We leverage this insight to introduce an automated training pipeline that improves factual consistency in summaries by aggregating scores from different weak metrics. Our approach avoids the need for complex reward shaping by mapping scores to preferences and filtering out cases with high disagreement between metrics. For each source document, we generate lexically similar summary pairs by varying decoding strategies, enabling the model to learn from factual differences caused by subtle lexical differences. This approach constructs a high-quality preference dataset using only source documents. Experiments demonstrate consistent factuality gains across models, ranging from early encoder-decoder architectures to modern large language models, with smaller models reaching comparable factuality to larger ones 1 1 1 Code is available at https://github.com/Haruhi07/MultiMetric . 1 Introduction Figure 1: Our method only requires source documents to build a preference dataset. Cutting-edge language models have demonstrated impressive capabilities in generating fluent and coherent responses to a wide range of prompts. However, maintaining faithfulness and factual consistency remains a persistent challenge, particularly in tasks like summarisation. Despite their surface plausibility, model-generated summaries often contain factual inconsistencies or hallucinated details ( 14 ) . Recent research has tried to mitigate this issue by incorporating reinforcement learning (RL) to guide models towards more factually consistent outputs. A critical obstacle lies in designing effective reward signals that can reliably capture and quantify factuality. Many existing automatic evaluation metrics ( 18 ; 15 ; 43 ; 16 ) have been adopted as reward signals for RL. However, even state-of-the-art metrics struggle with subtle inconsistencies and may penalise factually accurate outputs ( 31 ) . Furthermore, using a single metric directly as a scalar reward for RL can lead to unstable training, as explored by 27 . The training process is influenced by the metric’s reliability and the distribution of reward scores, but the distributions of existing evaluation metrics are not well-studied. Although combining metrics can broaden error detection coverage ( 39 ) , applying metric combination in RL often requires manual weighting of sub-rewards ( 8 ; 24 ; 34 ; 40 ) , thus the use of RL is impeded by the complexity of reward shaping. An alternative is Reinforcement Learning from Human Feedback ( 22 , RLHF ), which uses human - annotated preferences and has proven effective at aligning large language models (LLMs) with broad human values. However, recent work ( 12 ; 38 ) shows that existing RLHF datasets often overlook factuality, despite instructions to annotators to account for it. Annotator judgments integrate various considerations, such as trade - offs among properties, individual biases, and occasional misunderstandings, so the resulting overall preferences can fail to reliably capture factuality. Therefore, while constructing a factuality-focused preference dataset is crucial, doing so requires substantial resources and expertise, making scalability a major concern. To overcome these barriers, this paper proposes a fully automated training pipeline that improves factual consistency in summarisation without relying on human annotations or reference summaries. We adopt the sampling method from 4 , using the language model itself to generate two summaries by either selecting alternative candidate outputs from the same decoding strategy or using different decoding strategies, as illustrated in Figure 1 . In contrast to their work, which paired diverse samples together, our approach ensures that summaries in a pair are lexically similar. This lexical similarity minimises confounding stylistic or structural differences, allowing the model to focus specifically on factual distinctions, which facilitates the factuality improvement during training. With the generated summary pairs, we use an ensemble of factuality metrics to score them and derive preference labels from the scores. To mitigate the unreliability of any single metric, we include only those summary pairs for which all selected metrics agree along with preference learning. This agreement-based filter removes noisy and contradictory annotations, enhancing the robustness of the preference signal and making the training process more reliable and scalable. By leveraging lexically similar summary pairs and agreement-based preference labels derived from multiple factuality metrics, our method enables more targeted factuality training than previous RLHF or model-based approaches ( 30 ; 4 ) . Importantly, in contrast with previous work ( 27 ) that requires an LLM-sized reward model to stabilise the training, we never encounter catastrophic forgetting in our experiments, demonstrating the stability and effectiveness of our proposed method on various language models. Our method consistently demonstrates factuality improvements on different model architectures, including BART ( 17 ) , GPT-J ( 36 ) , LLaMA-3 ( 1 ) , and DeepSeek-R1 ( 5 ) , showing strong generalisation beyond a single model family or scale. Remarkably, our method empowers older and smaller models, such as BART, to achieve factuality performance comparable to that of significantly larger and more recent models, effectively revitalising their potential to produce accurate summaries at lower computational cost. Our contributions are threefold: • We present a novel, fully automated preference learning pipeline for optimising summarisation factuality, which not only improves LLMs’ factuality scores but also elevates the factuality performance of smaller models to the same level. • We show a promising way to adapt multiple existing factuality metrics into training targets. By leveraging preference learning and filtering cases with high disagreement, we improve the reliability of the training data, leading to more robust training in practice. • We analyse the contribution of lexical similarity between summary pairs and conclude that, with sufficiently accurate preference annotations, similar pairs are more effective for enhancing factuality for summarisers. 2 Related Work 2.1 Factuality Evaluation in Summarisation Factuality has become one of the most critical properties to evaluate in recent language models. Depending on the methodologies applied, existing factuality evaluation metrics can be broadly categorised into 3 types. Similarity-based metrics Classical similarity-based metrics such as ROUGE ( 18 ) and BLEU ( 23 ) reported the n-gram overlapping ratio between the system and the reference summaries. Subsequently, BERTScore ( 43 ) replaced the exact word matching with embedding-based cosine similarity to enhance robustness to lexical and syntactic variation. This idea was then extended to sentence embedding similarity by 39 , who found that using the source document as reference could shrink the performance gap between similarity-based metrics and other methodologies. Question Answering-based metrics This line of work frames factuality evaluation as a reading comprehension task. Key phrases are extracted from the summary, and questions are generated based on their context. A question-answering model answers these questions using the source document, then checks whether the answers are consistent with the summary ( 6 ; 28 ; 7 ) . While this approach has shown empirical effectiveness, it usually involves multiple processing stages and models, making it computationally expensive. Entailment-based metrics These methods assess whether the source document entails the summary using natural language inference (NLI) models. Early approaches that simply concatenated the entire document and the