Event-based evaluation of abstractive news summarization
Huiling You, Samia Touileb, Erik Velldal, Lilja Øvrelid · 2025
Read Event-based evaluation of abstractive news summarization by Huiling You, Samia Touileb, Erik Velldal, Lilja Øvrelid (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-01160.txt.
Introduction
Huiling You and co-authors propose judging machine-written news summaries by the events they carry, not just the words they share with a reference. Standard scores compare word overlap against human summaries, which can reward fluent text that drops the point. News, at its core, reports events, so summaries should too.
Their method counts overlapping events among the generated summary, the human reference, and the source article. They test it on a richly annotated Norwegian dataset with expert event labels and summaries. Event-level scoring shows which facts survive summarization and which vanish, insight that word scores blur. Summary tools could be steered by what happened, not merely by phrasing.
Catalog record
| CREATOR | Huiling You, Samia Touileb, Erik Velldal, Lilja Øvrelid |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-01160
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Event-based evaluation of abstractive news summarization Huiling You, Samia Touileb, Erik Velldal, Lilja Øvrelid Abstract An abstractive summary of a news article contains its most important information in a condensed version. The evaluation of automatically generated summaries by generative language models relies heavily on human-authored summaries as gold references, by calculating overlapping units or similarity scores. News articles report events, and ideally so should the summaries. In this work, we propose to evaluate the quality of abstractive summaries by calculating overlapping events between generated summaries, reference summaries, and the original news articles. We experiment on a richly annotated Norwegian dataset comprising both events annotations and summaries authored by expert human annotators. Our approach provides more insight into the event information contained in the summaries. 1 Introduction A summary of a news article provides a condensed version of its main content ( El-Kassas et al. 2021 ) . One of the primary practical applications of large language models (LLMs) is generating concise text summaries, and many news publishers in Norway have already integrated LLM-generated summaries into their articles. However, assessing the quality and accuracy of these summaries remains a challenge. Current evaluation metrics compare generated summaries to ideal summaries created by humans, in terms of overlapping words/units, such as ROUGE-L ( Lin 2004 ) , or semantic similarity, such as BERTScore ( Zhang* et al. 2020 ) . However, these metrics provide limited information on the semantic content of the summaries themselves. With the increasing usage of LLMs for text generation, there has been a growing number of studies on evaluating the factuality of these texts from the perspective of contained information, such as FACTSCORE ( Min et al. 2023 ) . For summarization, Zhang and Bansal 2021 propose to use semantic triplet units as a judgment of the semantic content units in generated texts, and Liu et al. 2023 also propose a similar protocol based on semantic units, named Atomic Content Units. Inspired by event extraction (EE), a NLP task that extracts event information from unstructured texts into structured forms ( Doddington et al. 2004 ) , we propose to analyze the quality of news article summaries by comparing the overlapping events between generated summaries, reference summaries, and the source articles. By using structured event information, we provide more insight into both the generated summaries and human-authored summaries. We experiment on a Norwegian dataset with rich annotations both for events (EDEN ( Touileb et al. 2024 ) ), and summaries (NorSumm ( Touileb et al. 2025 ) ), and demonstrate the usefulness of the proposed event-based evaluation metric which is grounded in the overlap of identified events. 2 Event-overlap Our proposed metric calculates the degree of overlapping events between summaries (generated and human-authored) and the source texts. First, an event extraction system is used to extract events from summaries and source articles. Second, standard event extraction evaluation metrics are adapted and applied to calculate the actual event overlaps. 2.1 Event extraction Figure 1: Example of a sentence with event annotation. The ARREST-JAIL event has the trigger “arrested”, and the VICTIM argument is “Over 450 people”. An event ( Doddington et al. 2004 ) contains four key elements: 1) event type is the specific type of event defined within an ontology; 2) event trigger is the word(s) in the text that describes the event; 3) event argument is the attribute and actual participant of an event in the text; 4) argument role is the role played by an argument in the specific event. Figure 1 shows an example of a Norwegian sentence annotated for an ARREST-JAIL event with “arrested” as the event trigger, and a VICTIM argument “Over 450 people”. We use an existing event extraction system NorEventGen ( You et al. 2025 ) to obtain event information in these structured formats. We perform event extraction on three different texts: 1) model-generated summaries; 2) human-authored summaries; and 3) original news articles. 2.2 Event-overlap analysis Our event-overlap metric is adapted from the classical evaluation metrics of event extraction ( Lin et al. 2020 ; Nguyen et al. 2021 ) , as follows: an event trigger is correctly identified (Trg-I) if its offsets match a reference trigger, and correctly classified (Trg-C) if its event type also matches a reference trigger; An argument is correctly identified (Arg-I) if its offsets match a reference argument, and correctly classified (Arg-C) if its argument role also matches the reference argument. Since an abstractive summary does not perform text extraction from the source article, we do not expect a perfect match between an event trigger / argument from the summary and one from the article. As an alternative, we use BERTScore ( Zhang* et al. 2020 ) as a reference to check if two pieces of texts are similar. 1 1 1 We use a heuristic threshold of 0.7. If the BERTScore is larger than 0.7, two text snippets will be considered similar, the same as perfect match in event extraction metric. Unlike in event extraction, we prioritize the labels, namely event type and argument role. We do not take trigger word(s) into account, because the event type information itself is sufficient, and unlike event arguments, which are named entities, trigger words are more often rephrased with a different choice of words in summaries. With the corresponding adaptation, our proposed event-overlap metric calculates the following three categories of scores: • An event type (eType-C) overlaps if it exists in both lists of extracted events. • An argument role (Role-C) overlaps if the event type and argument role overlap. • An argument (Arg-C) overlaps if the event type, argument role, and argument word(s) overlap. The Precision (P), Recall (R), and F1 scores of each category are calculated. The final event-overlap score is an aggregated score of the three categories of scores: \rm\textbf{Event{-}overlap}=\textbf{Average}([eType\textrm{-}C,Role\textrm{-}C,Arg\textrm{-}C]) . Depending on the event overlap of different texts, different scores are used: • Event-overlap between summaries : the final event-overlap score is the average Recall scores of eType-C, Role-C, and Arg-C. Recall scores prioritize the events that are in the gold summaries. • Event-overlap between summaries and original articles : the final event-overlap score is the average Precision scores of eType-C, Role-C, and Arg-C. Precision scores provide evaluation of identified events in the summaries that are also present in the original articles. 3 Experimental setup Datasets We use two recently released datasets: the Norwegian event detection dataset EDEN ( Touileb et al. 2024 ) and the human-authored summaries of Norwegian news articles dataset NorSumm ( Touileb et al. 2025 ) . The source articles of NorSumm are a subset of EDEN. These parallel annotations of events and summaries make it possible to evaluate our approach and contrast gold vs predicted event information on gold vs generated summaries. More concretely, we use the test set of NorSumm, which contains 33 news articles, each coupled with three unique human-authored summaries. LLMs For automatic summarization, we evaluate a range of Norwegian and Nordic open-source pretrained and instruction-finetuned decoder-only LLMs: Llama-3-8B-instruct, 2 2 2 https://huggingface.co/AI-Sweden-Models/Llama-3-8B-instruct Llama-3-8B, 3 3 3 https://huggingface.co/AI-Sweden-Models/Llama-3-8B Meta-Llama-3-8B-Instruct 4 4 4 https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct , Mistral-Nemo-Instruct-2407, 5 5 5 https://huggingface.co/mistralai/Mistral-Nemo-Instruct-2407 Normistral-11b-warm 6 6 6 https://huggingface.co/norallm/normistral-11b-warm , and Normistral-7b-warm-instruct. 7 7 7 https://huggingface.co/norallm/normistral-7b-warm-instruct All the LLMs are available via HuggingFace.