LCFO : Long Context and Long Form Output Dataset and Benchmarking

Marta R. Costa-jussà, Pierre Andrews, Mariano Coria Meglioli, Joy Chen · 2024

Read LCFO : Long Context and Long Form Output Dataset and Benchmarking by Marta R. Costa-jussà, Pierre Andrews, Mariano Coria Meglioli, Joy Chen (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-08268.txt.

Introduction

Marta Costa-jussà leads a team in a 2024 paper on LCFO, a dataset and benchmark for long context and long form output. Language models are asked to read ever longer texts and to write at length in reply. Tests for that skill, in steps of size, were thin on the ground.

Each LCFO item holds a long document, of some five thousand words on mean. Three summaries come with it, at set shares of the full length, plus a set of questions and answers on the content. Links tie some answers to the summaries that hold them, across seven domains. The graded sizes let gradual summarizing and summary growth be scored in a controlled way.

Catalog record

CREATORMarta R. Costa-jussà, Pierre Andrews, Mariano Coria Meglioli, Joy Chen
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE106 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-08268

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LCFO : Long Context and Long Form Output Dataset and Benchmarking Marta R. Costa-jussà, Pierre Andrews, Mariano Coria Meglioli, Joy Chen Abstract This paper presents the Long Context and Form Output (LCFO) benchmark, a novel evaluation framework for assessing gradual summarization and summary expansion capabilities across diverse domains. LCFO consists of long input documents (5k words average length), each of which comes with three summaries of different lengths (20%, 10%, and 5% of the input text), as well as approximately 15 questions and answers (QA) related to the input content. Notably, LCFO also provides alignments between specific QA pairs and corresponding summaries in 7 domains. The primary motivation behind providing summaries of different lengths is to establish a controllable framework for generating long texts from shorter inputs, i.e. summary expansion. To establish an evaluation metric framework for summarization and summary expansion, we provide human evaluation scores for human-generated outputs, as well as results from various state-of-the-art large language models (LLMs). GPT-4o-mini achieves best human scores among automatic systems in both summarization and summary expansion tasks ( ≈ \approx +10% and +20%, respectively). It even surpasses human output quality in the case of short summaries ( ≈ \approx +7%). Overall automatic metrics achieve low correlations with human evaluation scores ( ≈ \approx 0.4) but moderate correlation on specific evaluation aspects such as fluency and attribution ( ≈ \approx 0.6). 1 Introduction Robust long text generation capabilities are required to meet user demand for extensive content creation, including story writing and essay composition ( Xie and Riedl, 2024 ) , which is why recent models such as GPT-4 ( et al., 2024b ) are expanding the output lengths from 4k tokens in GPT-4o to 64k in the latest versions. Figure 1: LCFO annotation and evaluation framework for the gradual summarization (top) and summary expansion (bottom) tasks. The resulting benchmark can also be used for other tasks such as reading comprehension and automatic evaluation metric development. In gradual summarization, long documents are split into paragraphs and annotated by humans to generate 3 level of summaries, plus a questions-answers set. Additionally, different LLMs were used to construct automated summaries at corresponding levels. In summary expansion, different human summaries are prompted to LLMs to generate the output targeting the same length as the source document. The components are described in different sections (marked in parentheses) However, evaluating the performance of Large Language Models (LLMs) on summarization and summary expansion tasks (see definitions in Section 2.1 ) is particularly challenging, especially when it comes to summarizing very long input documents and generating either long summaries or long summary expansions. Although there is a lot of work and interest in studying summarization evaluation (e.g. Zhang et al., 2024a ), evaluation of long text outputs is an emerging area ( Que et al., 2024 ) . Indeed, long-context input processing tasks (such as summarization or comprehension question answering applied to long documents) and long-form output production both involve high cognitive loads for humans. This may be why evaluation work in these areas is not as mature as in others. To complement the summarization task, we include an “inverted” task of summary expansion : generating a longform text based on its shorter sketch. It requires more creativity than summarization (which can be approached extractively) but imposes more constraints than open-ended language modeling. A practical application for this task could be with a writing assistant expanding the text sketch created by the human writer into a longer output that the human subsequently revises. Another potential use is “back-translation” for summarization, with a model training to summarize the synthetic expanded documents into their original short forms. The LCFO dataset that we are presenting in this paper is an exclusively human annotated benchmark and challenge dataset involving natural language understanding and generation across multiple domains in the aforementioned tasks. Our dataset is carefully manually crafted with human revision from the selection of the documents to the final annotation, without relying on LLMs at any point. We provide detailed linguistic guidelines and abstractive QA. Furthermore, we provide another set of linguistic guidelines to evaluate the tasks of summarization and summary expansion. Together, we present the LCFO benchmark. Given a source of long structured documents, we generate multiple long outputs and associated QA pairs, and we evaluate human and model outputs with human annotations. The main contributions of this work are the following (see Figure 1 for a schematic representation of the dataset and tasks): • Dataset creation with structured inputs and alternative references; each input document is associated with 3 summaries of different lengths (20%, 10%, 5%). Gradual summarization is useful in that it provides both long and short summaries references. Moreover, the availability of summaries of various length is expected to improve the development of summary expansion approaches, which allows us to provide a more controllable summary expansion framework. • A set of QA pairs for each input document aligned with each of the different length summaries. Our QA pairs are in free-form short-answer format (i.e. not multiple choice) and are of the abstractive type (i.e. they are not copies of parts of the source document). This QA can potentially be used to evaluate the model outputs, based on the appropriateness of the responses, similarly to previous proposals ( Wang et al., 2022 ) . • Selection of automatic metrics. Most of these metrics can be used to evaluate at the paragraph level, and more widely can be used to evaluate summary expansion or long-form generation in multiple tasks and languages. • Evaluation of several LLMs on our dataset both automatically and manually; and evaluation of automatic metrics on summarization and summary expansion. 2 The LCFO Creation Framework: Principles and Methods This section describes the detailed principles and methods that underlie LCFO (Figure 1 (right)). 2.1 Definitions Long context/form. We define long text as text that exceeds 5k words. For reference, the attention span for reading and taking notes has been considered 10 to 15 minutes since a 1978 seminal paper ( Hartley and Davies, 1978 ) . More recent research cited in a survey paper ( Bradbury, 2016 ) shows that attention paid to lectures declines significantly after 20 minutes. The tasks we describe here are closer to note taking than lecture attendance; therefore, we should keep the 10–15 minute reference. If we base ourselves on the reported reading average time for first-language, secondary-education level readers (125 words per minute on average), we can consider that we will reach high cognitive load at around 1.5k words, and peak cognitive load at around 2.5k words. The quality of cognitive processing then starts to decrease significantly after. Structured/hierarchical. The input and output documents are partitioned into sections and, if necessary, nested sub-sections. Their structure is determined by task- and domain-specific guidelines. Gradual summarization (GS). The input is a long document (defined here as 5k words or more). The summarization task can be described as the act of generating a much shorter corresponding document that includes the essential information contained in the long document and the same logical structure linking the various pieces of essential information. The summarization task presented here consists in taking long documents as inputs and generating three corresponding summaries of

Read the complete text