SCURank: Ranking Multiple Candidate Summaries with Summary Content Units for Enhanced Summarization

Bo-Jyun Wang, Ying-Jia Lin, Hung-Yu Kao · 2026

Read SCURank: Ranking Multiple Candidate Summaries with Summary Content Units for Enhanced Summarization by Bo-Jyun Wang, Ying-Jia Lin, Hung-Yu Kao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-19185.txt.

Introduction

Wang, Lin, and Kao introduce SCURank, a framework ranking summaries by the richness and semantic importance of their content, using Summary Content Units. Small models such as BART can match large models via distillation, but ranking strategies are unstable and classical metrics such as ROUGE rank high-quality summaries poorly. SCURank beats traditional metrics and model ranking methods across measures and datasets. Diverse model summaries also enhance abstractiveness and distilled model performance, showing the value of content ranking in multi-model distillation.

Catalog record

CREATORBo-Jyun Wang, Ying-Jia Lin, Hung-Yu Kao
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55846
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.19185

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SCURank: Ranking Multiple Candidate Summaries with Summary Content Units for Enhanced Summarization Bo-Jyun Wang, Ying-Jia Lin, Hung-Yu Kao Abstract Small language models (SLMs), such as BART, can achieve summarization performance comparable to large language models (LLMs) via distillation. However, existing LLM-based ranking strategies for summary candidates suffer from instability, while classical metrics (e.g., ROUGE) are insufficient to rank high-quality summaries. To address these issues, we introduce SCURank , a framework that enhances summarization by leveraging Summary Content Units (SCUs) . Instead of relying on unstable comparisons or surface-level overlap, SCURank evaluates summaries based on the richness and semantic importance of information content. We investigate the effectiveness of SCURank in distilling summaries from multiple diverse LLMs. Experimental results demonstrate that SCURank outperforms traditional metrics and LLM-based ranking methods across evaluation measures and datasets. Furthermore, our findings show that incorporating diverse LLM summaries enhances model abstractiveness and overall distilled model performance, validating the benefits of information-centric ranking in multi-LLM distillation. The code for SCURank is available at https://github.com/IKMLab/SCURank . 1 Introduction Following the advent of ChatGPT ( 30 ) , a paradigm shift has occurred in the domain of Natural Language Processing (NLP), marked by substantial advancements, including the field of summarization. In the wake of this development, GPT-4 ( 27 ) and numerous Large Language Models (LLMs) have emerged ( 9 ; 2 ; 22 ) , demonstrating superior performance. These models are easily accessible via APIs, allowing users to obtain responses with minimal effort. However, while these models are powerful, their resource demands, such as the cost of local deployments, are significant ( 12 ) due to their incredibly large model size. This has led to a growing trend of distilling these models into smaller ones, optimized for specific tasks such as summarization 11 ; 13 . The distilled models exhibit great resource efficiency without significant performance degradation, and in some cases, they demonstrate a capability to outperform LLMs in the summarization task 20 . Previous work by 20 leverages BRIO ( 18 ) as a contrastive learning framework for model distillation. In BRIO, positive and negative samples are constructed by ranking candidate summaries. Consequently, the quality of the ranking function is critical for effective contrastive learning. To address this challenge, they introduced GPTRank 20 , a novel method that ranks high-quality summaries generated by LLMs, ensuring more reliable supervision in BRIO training. However, there are two significant challenges to this approach. First, studies by 34 ; 37 indicate that LLMs remain unreliable and inconsistent in text comparison and candidate ranking. Second, relying on summaries from a single LLM introduces the risk of model-specific bias (e.g. content selection) and limits the diversity of generation patterns. Thus, we explore the use of multiple LLMs to generate candidate summaries for distillation, and investigate effective ranking methods for these summaries. To bypass the instability of direct LLM ranking, we propose shifting the evaluation focus back to the core goal of summarization: information retention. To capture the information from summaries, we draw upon the concept of Summary Content Units (SCUs) 26 . Every SCU presents simple, standalone, and unique information in the summary ( 33 ) . Generally, annotating SCUs requires manual efforts, thus expensive and not reproducible ( 40 ) . 25 suggested a new path to extract SCUs, which employed GPT-3.5 and GPT-4 ( 27 ) to generate Semantic GPT Units (SGUs). The SGUs are similar to SCUs, capturing the key information in the summaries, and have been proven to be high-quality in the evaluations of 25 . Consequently, we adopt the concept of SGUs as SCUs to evaluate the quality of summaries. In this paper, we introduce SCURank ( S ummary C ontent U nit Rank ing), a ranking framework that evaluates the information richness of candidate summaries by analyzing their SCUs. SCURank operates in three stages. First, it captures the key information in each summary by extracting SCUs. Next, all SCUs are aggregated by clustering to estimate their importance based on their frequency across summaries. Finally, each summary is assigned a score by summing the importance scores of its SCUs, which reflects its overall information richness. This score is further normalized by summary length to mitigate bias toward longer summaries. By focusing on the information content rather than direct comparison or ranking, SCURank provides a robust ranking approach. Specifically, as LLMs are only employed for SCU extraction, the framework avoids the unreliability associated with LLM-based comparison. Our contributions are as follows: (1) We propose SCURank, a novel method to rank high-quality summaries based on SCUs. (2) We investigate the effect of distilling models from multiple LLMs. (3) We demonstrate that SCURank, combined with contrastive learning, improves distilled model performance. Figure 1: Overview of our training framework. The Data Generation part generates candidate summaries from several LLMs for each article. SCURank, our proposed ranking method, ranks the candidate summaries via three steps: (1) extracting Summary Content Units (SCUs), (2) aggregating SCUs with sentence embeddings and HDBSCAN, and (3) scoring summaries based on SCU cluster distribution. Finally, the ranked summaries are then used to train a distilled model with BRIO. 2 Related Work SCURank, which ranks summaries by employing Summary Content Units (SCUs), is based on two concepts. The first method is to decompose a summary candidate into SCUs, which are brief and convey a single fact. The second is the ranking. The ranking results for the candidate summaries can be used for contrastive learning to improve the LMs’ summarization capabilities. 2.1 Summary Content Units The concept of SCUs was first introduced by 26 . In that study, the authors introduced a reliable, predictive, and diagnostic method for evaluating the summaries. However, due to the high cost and expertise required, 33 demonstrated a revised version that is more cost-effective and annotation-friendly. In this approach, they provided the instructions for crowd workers to extract a specific number of the SCU-like statements, eliminating the need to merge and weight SCUs. These efforts reduced the dependency on expertise, making the Pyramid method more efficient. Nevertheless, the significant cost of hiring workers to extract SCUs remains a significant challenge. In response, researchers have been investigating methods of automating this process in recent years. 40 proposed a system called List^{3}Pyramid , which employed a semantic role labeling model to extract Summary Triplets Units (STUs). Subsequently, 25 introduced two novel methods. One method leveraged Abstract Meaning Representation to extract Semantic Meaning Units (SMUs), and the other employed an LLM to extract Semantic GPT Units (SGUs). 25 have demonstrated the high quality of the SGUs in a wide range of evaluations. In our work, we adopt the concept of SGUs from 25 for extracting SCUs in SCURank. 2.2 Summarization with Ranking Contrastive learning is a method of training a model to identify and select superior candidates, while avoiding inferior ones. In summarization, SimCLS ( 19 ) was the first to employ a scoring model for contrastive learning. This scoring model evaluates the quality of the candidate summaries, thereby enhancing overall performance on summarization tasks. Subsequently, BRIO ( 18 ) integrated these tasks into a single model, which is capable of both generating and evaluating summaries. Both approaches used ROUGE ( 16 ) as a metric for evaluating summaries. However,

Read the complete text