Learning to Rank Salient Content for Query-focused Summarization

Sajad Sotudeh, Nazli Goharian · 2024

Read Learning to Rank Salient Content for Query-focused Summarization by Sajad Sotudeh, Nazli Goharian (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-00324.txt.

Description

Abstract This study examines the potential of integrating Learning-to-Rank (LTR) with Query-focused Summarization (QFS) to enhance the summary relevance via content prioritization. Using a shared secondary decoder with the summarization decoder, we carry out the LTR task at the segment level.

Catalog record

CREATORSajad Sotudeh, Nazli Goharian
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2411-00324

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Learning to Rank Salient Content for Query-focused Summarization Sajad Sotudeh, Nazli Goharian Abstract This study examines the potential of integrating Learning-to-Rank (LTR) with Query-focused Summarization (QFS) to enhance the summary relevance via content prioritization. Using a shared secondary decoder with the summarization decoder, we carry out the LTR task at the segment level. Compared to the state-of-the-art, our model outperforms on QMSum benchmark (all metrics) and matches on SQuALITY benchmark (2 metrics) as measured by Rouge and BertScore while offering a lower training overhead. Specifically, on the QMSum benchmark, our proposed system achieves improvements, particularly in Rouge-L (+0.42) and BertScore (+0.34), indicating enhanced understanding and relevance. While facing minor challenges in Rouge-1 and Rouge-2 scores on the SQuALITY benchmark, the model significantly excels in Rouge-L (+1.47), underscoring its capability to generate coherent summaries. Human evaluations emphasize the efficacy of our method in terms of relevance and faithfulness of the generated summaries, without sacrificing fluency. A deeper analysis reveals our model’s superiority over the state-of-the-art for broad queries, as opposed to specific ones, from a qualitative standpoint. We further present an error analysis of our model, pinpointing challenges faced and suggesting potential directions for future research in this field. 1 Introduction Query-focused summarization (QFS) is gaining prominence in research community. Unlike conventional summarization tasks that aim to capture the overall essence of a document or a set of documents, QFS focuses on generating concise summaries in response to posed queries. This specialization enables a more targeted information retrieval process, offering summaries that directly address the informational needs rather than providing a broad overview of the source material. The advancements in QFS have been notably driven by the introduction of invaluable datasets of long documents such as QMSum with an average of 9K tokens Zhong et al. (2022) and SQuALITY with an average of 5.2K tokens Wang et al. (2022a) , which have facilitated deeper exploration and innovation in this field. These datasets have laid the groundwork for the development of Transformers-based models which have shown strong potential in generating summaries that respond accurately to queries ( Su et al., 2021 ; Laskar et al., 2022 ; Vig et al., 2022 ; Pagnoni et al., 2022 ; Sotudeh and Goharian, 2023 ; Yu et al., 2023 ) . However, despite this proficiency, their ability to effectively prioritize information —assessing its importance relative to a query to enhance summary relevance—remains an area for improvement. This study seeks to address this limitation, with a particular focus on long-input QFS, where summarizing multiple segments 1 1 1 A chunk of document with a predefined length (e.g., 512 tokens). for a given query presents unique challenges in capturing and prioritizing relevant content. Particularly, in this study, we present a novel enhancement to QFS through the incorporation of learning-to-rank (LTR), a technique with established efficacy in Information Retrieval. Our approach aims to refine the system’s capability to discern and prioritize content segments not only by their relevance but also by their relative importance. This methodological advancement ensures that the produced summaries more accurately reflect the query’s intent and hierarchically organize information by its significance. Central to our approach is the augmentation of use of the decoder that shares parameters with the summarization decoder 2 2 2 Particularly, we use the single decoder for two tasks: summarization and learning-to-rank. , specifically designed for executing the LTR task at the segment level. While Learning-to-Rank (LTR) is a classic approach, our innovation lies in adapting LTR principles specifically for Query-Focused Summarization (QFS) at the segment level, which has been less explored in the literature. This strategy, inspired by the work of ( Zhuang et al., 2022 ) in adapting the T5 ( Raffel et al., 2020 ) encoder-decoder framework for text ranking in query-document scenarios, is tailored to address the nuances of segment ranking within the QFS context. Through the joint fine-tuning of summarization with cross-entropy loss, and LTR task—utilizing listwise cross-entropy softmax loss, our system not only elevates the relevance of generated summaries but also introduces a nuanced understanding of information importance. This strategy can aid the summarization system at attending to the source content given their relative importance. In short, our contributions are threefold: • We propose an LTR-assisted system for QFS that integrates the intuition of ranking and relative importance of segments during the summary generation process; • Our proposed system outperforms across all automatic metrics (QMSum) and attains comparable performance in two metrics (SQuALITY) with lower training overhead compared to the SOTA. Additionally, our system enhances the relevance and faithfulness of generated summaries without sacrificing fluency; • We undertake an error analysis to discern the challenges faced by our model including label imbalance, and segment summarizer’s hurdles, providing insights into potential avenues for further research. 2 Related work The field of Query-focused Summarization (QFS) Dang (2005) has evolved significantly over the years, transitioning from early unsupervised extractive models ( Mohamed and Rajasekaran, 2006 ; Wan et al., 2007 ; Zhao and Tang, 2010 ; Badrinath et al., 2011 ; Litvak and Vanetik, 2017 ) to recent approaches leveraging Transformer-based models ( Vaswani et al., 2017 ; Lewis et al., 2020 ; Zhang et al., 2020 ) . This evolution has been marked by the introduction of various techniques aimed at improving the relevance of summaries. Passage retrieval techniques ( Baumel et al., 2018 ; Laskar et al., 2022 ; Su et al., 2021 ; Zhong et al., 2022 ; Wang et al., 2022a ) , transfer learning from the QA task Xu and Lapata (2020) ; Zhang et al. (2021) ; Yuan et al. (2022) , query modeling Xu and Lapata (2021) ; Xu and Lapata (2022) ; Yu et al. (2023) , segment encoding Vig et al. (2022) , and attention mechanisms to capture query-utterance relations ( Liu et al., 2023 ) have all played a pivotal role in this advancement. Furthermore, the adoption of question-driven pretraining ( Pagnoni et al., 2022 ) and contrastive learning ( Sotudeh and Goharian, 2023 ) has introduced new dimensions to the task, simplifying the identification and summarization of salient content. More recently, Liu and Xu (2023) introduced the Ranker-Generator framework, which ranks utterances by learning from pairwise comparisons and global ordering. The top-ranked utterances are then used as input for the generator in summary generation. While these methods have advanced QFS, existing techniques tend to treat all content segments equally without explicitly considering their relative importance within long-input QFS tasks, which require processing large amounts of text and identifying key segments. Hence, the comprehensive modeling of segment importance within the long QFS task remains a less explored area. Our approach introduces a novel application of learning-to-rank (LTR) Burges et al. (2005) ; Cao et al. (2007) mechanism to address this challenge, drawing inspiration from the successful application of LTR in broader Information Retrieval contexts Wang et al. (2022b) ; Li et al. (2023) . 3 Background: Segment Summarizer (SegEnc) The backbone of current state-of-the-art systems for query-focused long summarization are built upon the Segment Encoding ( SegEnc ) approach Vig et al. (2022) . SegEnc operates by encoding fixed-length, overlapping segments of the source text, which are then integrated into a cohesive summary in an end-to-end manner, leveraging

Read the complete text