Large Language Models and Book Summarization: Reading or Remembering, Which Is Better?
Tairan Fu, Javier Conde, Pedro Reviriego, Javier Coronado-Blázquez, Nina Melero, Elena Merino-Gómez · 2026
Read Large Language Models and Book Summarization: Reading or Remembering, Which Is Better? by Tairan Fu, Javier Conde, Pedro Reviriego, Javier Coronado-Blázquez, Nina Melero, Elena Merino-Gómez (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-09981.txt.
Introduction
Tairan Fu and coauthors compare book summaries a language model writes from memory against ones built from the full text. Long context windows now let a model read a whole book in one prompt, while famous books are already known from training. Full text summaries are richer in general, the team finds. Yet for some books the memory based versions score higher, raising doubts about long text summarising skill.
Catalog record
| CREATOR | Tairan Fu, Javier Conde, Pedro Reviriego, Javier Coronado-Blázquez, Nina Melero, Elena Merino-Gómez |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25870 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.09981
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Large Language Models and Book Summarization: Reading or Remembering, Which Is Better? Tairan Fu, Javier Conde, Pedro Reviriego, Javier Coronado-Blázquez, Nina Melero, Elena Merino-Gómez Abstract Summarization is a core task in Natural Language Processing (NLP). Recent advances in Large Language Models (LLMs) and the introduction of large context windows reaching millions of tokens make it possible to process entire books in a single prompt. At the same time, for well-known books, LLMs can generate summaries based only on internal knowledge acquired during training. This raises several important questions: How do summaries generated from internal memory compare to those derived from the full text? Does prior knowledge influence summaries even when the model is given the book as input? In this work, we conduct an experimental evaluation of book summarization with state-of-the-art LLMs. We compare summaries of well-known books produced using (i) only the internal knowledge of the model and (ii) the full text of the book. The results show that having the full text provides more detailed summaries in general, but some books have better scores for the internal knowledge summaries. This puts into question the capabilities of models to perform summarization of long texts, as information learned during training can outperform summarization of the full text in some cases. Keywords LLMs \cdot Evaluation \cdot Summarization \cdot Long-context Models 1 Introduction Summarization is one of the most common tasks in Natural Language Processing (NLP) [ 14 ] . Over the years, many techniques and models have been used for summarization, typically trained on pairs of documents, summaries. For example, early methods relied on statistical features such as term frequency, sentence position, and cue phrases to select relevant sentences [ 23 ] . Later, neural network–based approaches, particularly sequence-to-sequence models, performed summarization by generating new sentences that capture the meaning of the source document [ 16 ] . More recently, transformer-based pre-trained encoders like BERT have been used to improve summarization performance by using rich contextual representations [ 12 ] . The development of Large Language Models (LLMs) opened new directions for summarization, since the models can be given the text and asked to summarize it in a single prompt [ 3 ] . The use of LLMs can be more sophisticated by using prompt engineering, model fine-tuning, or knowledge distillation to achieve better performance or use smaller models [ 23 ] . For well-known texts, another option is to rely on the model’s internal knowledge to provide a summary, for example, of a book without providing the text [ 9 ] . In fact, LLMs are commonly seen to have two types of memories: internal (or parametric) and external [ 4 ] . The internal memory is encoded in the model parameters, representing the knowledge acquired during training. This parametric memory has been shown to store facts, linguistic patterns, and in some cases even verbatim passages from training data [ 15 ] . In contrast, the external memory is provided through the context window at inference time. In the case of summarization, it allows one to provide the model with the text to summarize. The size of the context window limits the amount of external memory, with the first generation of models having windows of just a few thousand tokens, while newer models such as GPT-4.1 and Gemini 2.5 reach the million-token range. Since most books have at most a few hundred thousand words, one million tokens is sufficient to pass the entire book in the input prompt for summarization. These two forms of memory can interact with parametric knowledge overriding contextual input, or the two may reinforce each other [ 8 ] . However, the interaction between memorization in weights and processing of contextual input remains an open research question. In this context, the study of book summarization based on internal and external memory is an interesting topic that can improve our understanding of the inner workings of LLMs and also of their limitations when performing summarization of large texts. This paper takes an initial step in this direction by performing a comparison of book summaries generated by LLMs based on internal and external knowledge with high-quality human summaries. In particular, the following contributions are made: 1. Conduct to the best of our knowledge the first evaluation comparing LLM generated book summaries with internal and external memory using state-of-the-art models with large context windows. 2. Create an open repository with the code, books, and summaries of the experiments to facilitate further research on the topic and reproducibility of our results. 3. Show that for most books, the use of the full text for summarization improves performance compared to summaries based only on parametric memory. 4. Show that for some books, internal knowledge summaries get better scores than summaries done with access to the full text of the book. 5. Discuss the potential reasons for the better performance of internal summaries. The rest of the paper is organized as follows. Section 2 summarizes related works on text summarization focusing on the use of LLMs and on the different types of memory in LLMs and their interactions. The methodology used for the evaluation is described in Section 3 and the results are presented and discussed in Section 4 . The paper ends with the conclusion in Section 5 . 2 Related work There are many recent works on summarization and LLM memory. The following subsections provide an overview of current research activities on these topics as well as some historical context. 2.1 Text summarization As discussed in the Introduction, text summarization has been a central task in NLP for decades [ 14 ] . The techniques used to produce summaries have evolved from early statistical-based methods that rely on word frequencies [ 13 ] to the use of complex neural models such as LLMs. There are two main types of automatically generated summaries: extractive and abstractive [ 23 ] . In the first case, the summary is built by extracting fragments from the original text, while in the latter, the process generates new text to summarize the input. Initially, summarization approaches tried to find the most relevant parts of the text by computing metrics on words and sentences and extracting them to build the summary. The use of classical machine learning algorithms was also explored for extractive summarization [ 6 ] . The development of powerful deep learning models led to the use of more advanced neural architectures such as recurrent neural networks (RNN), convolutional neural networks (CNN), and long-short-term memory networks (LSTM) for summarization [ 16 ] , [ 18 ] . The introduction of the Transformer architecture revolutionized NLP in general and text summarization in particular. Bidirectional Encoder Representations from Transformers (BERT), with its bidirectional encoder, has been widely used in extractive summarization, where embeddings are leveraged to select the most salient content from a document [ 12 ] . Instead, the Text-to-Text Transfer Transformer (T5) has been used for abstractive summarization, as autoregressive decoders enable the generation of fluent and coherent summaries beyond simple sentence extraction [ 17 ] . The use of Transformers set a new state-of-the-art and laid the foundations for the emergence of large language models designed to handle summarization as part of a broader