BookAsSumQA: An Evaluation Framework for Aspect-Based Book Summarization via Question Answering
Ryuhei Miyazato, Ting-Ruen Wei, Xuyang Wu, Hsin-Tai Wu, Kei Harada · 2025
Read BookAsSumQA: An Evaluation Framework for Aspect-Based Book Summarization via Question Answering by Ryuhei Miyazato, Ting-Ruen Wei, Xuyang Wu, Hsin-Tai Wu, Kei Harada (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-06183.txt.
Introduction
Ryuhei Miyazato and colleagues confront an evaluation problem: how to judge summaries of whole books that focus on one chosen aspect. Long books lack ready-made reference summaries, so scoring is hard. Their framework, BookAsSumQA, grades a summary by how well it answers questions about the book.
The framework builds aspect-specific question and answer pairs automatically from a narrative knowledge graph of the book. Experiments with the framework showed a clear pattern across text lengths. Language model approaches answered more accurately on shorter texts. Retrieval-based methods grew stronger as documents got longer, making them the more practical choice for full-length book summarization.
Catalog record
| CREATOR | Ryuhei Miyazato, Ting-Ruen Wei, Xuyang Wu, Hsin-Tai Wu, Kei Harada |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-06183
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
BookAsSumQA: An Evaluation Framework for Aspect-Based Book Summarization via Question Answering Ryuhei Miyazato, Ting-Ruen Wei, Xuyang Wu, Hsin-Tai Wu, Kei Harada Abstract Aspect-based summarization aims to generate summaries that highlight specific aspects of a text, enabling more personalized and targeted summaries. However, its application to books remains unexplored due to the difficulty of constructing reference summaries for long text. To address this challenge, we propose BookAsSumQA, a QA-based evaluation framework for aspect-based book summarization. BookAsSumQA automatically constructs a narrative knowledge graph and synthesizes aspect-specific QA pairs to evaluate summaries based on their ability to answer these questions. Our experiments on BookAsSumQA revealed that while LLM-based approaches showed higher accuracy on shorter texts, RAG-based methods become more effective as document length increases, making them more efficient and practical for aspect-based book summarization 1 1 1 https://github.com/ryuhei-miyazato/bookassumqa . 1 Introduction Automatic summarization condenses long texts into concise and informative representations, allowing readers to grasp key information efficiently. Book summarization applies this to novels, which are often lengthy and complex. The progress of automatic book summarization has been accelerated by the release of the BookSum dataset ( 11 ) , which contains novels paired with human-written summaries. With the growing volume of books, there is increasing interest in aspect-based summarization (ABS), which produces summaries tailored to specific aspects, such as themes or genres. Although ABS helps readers quickly access desired information and has been more actively explored in domains such as reviews ( 21 ) and lectures ( 10 ) , its application to books remains relatively understudied. This is mainly because summarization research relies on manually created reference summaries, and building evaluation datasets for long documents is a labor-intensive and costly process. The longer the original document and the greater the number of aspects, the higher the human and financial costs become. Figure 1: In BookAsSumQA, we generate aspect-specific QA pairs from a knowledge graph and evaluate summaries by testing whether they can answer these questions, thereby assessing aspect coverage without costly human-written references. To address this challenge, we propose BookAsSumQA, a QA-based evaluation framework for aspect-based book summarization that enables evaluation without manually created reference summaries. We synthesize aspect-specific QA pairs from the narrative through a knowledge graph, and evaluate aspect-based summaries by testing whether an LLM can answer these questions using the generated summary as reference. This allows us to measure how well the summary captures information about the aspects of the narrative. In this study, we define aspects as literary genres in novels (example: Figure 1 ). First, we construct a knowledge graph that represents relationships among entities in the narrative. Using an LLM, we extract relationships between entities (e.g., characters) with a textual description, keywords, and an importance score, and incrementally upsert them into the graph to capture the global relationships within the narrative. Next, we construct aspect-specific QA pairs from the knowledge graph. To do so, we first identify edges that are relevant to a target aspect by calculating the cosine similarity between the text embeddings of the aspect term and the edge keywords, and then generate aspect-specific QA pairs based on the descriptions of those edges. Finally, we evaluate ABS methods using the generated QA pairs by assessing whether each generated summary can correctly answer the questions. We then compare the generated answers against the ground-truth using ROUGE-1, METEOR, and BERTScore. By comparing the accuracy, we investigate which method is most suitable for aspect-based book summarization. 2 Related Work In the field of book summarization, as the BookSum dataset ( 11 ) provides pairs of public domain novels and generic summaries, obtaining the summaries is well studied ( 19 ; 20 ; 13 ) . In this study, we focus on ABS, which generates summaries centered on specific aspects of a text. Unlike Query-Focused Summarization (QFS), which generates summaries in response to specific user queries (e.g., SQuALITY ( 18 ) ), ABS instead focuses on predefined aspects such as genres or themes. ABS has been actively studied in domains such as news ( 22 ) , reviews ( 21 ) , lecture materials ( 10 ) , and multi-domain documents ( 8 ) , where reference summaries are often manually created or readily available. However, for long documents like books, creating such references is labor-intensive and costly, limiting the application of ABS in this domain. To overcome this difficulty, we propose a framework that evaluates aspect-based summaries of novels without manual reference summaries. While several studies have proposed reference-free evaluation metrics for summarization that assess summary quality without relying on gold reference summaries 3 ; 14 ; 6 , we introduce a QA-based framework that evaluates summaries without manual references by generating QA pairs from the source text, measuring how much information from the source text is captured in the summary 9 ; 17 ; 15 . In this work, we further extend this approach by generating aspect-specific QA pairs to evaluate how well each aspect-based summary captures information related to its corresponding aspect in the original text. 3 BookAsSumQA Figure 2: BookAsSumQA: Evaluation framework for aspect-based book summarization. 3.1 ABS Evaluation with BookAsSumQA In BookAsSumQA (Figure 2 ), we shift the evaluation of aspect-based summaries into a Question-Answering task. QA pairs are automatically synthesized through a knowledge graph of the narrative, where nodes are enriched with keywords and description to generate comprehensive aspect-specific questions. The quality of a summary is then assessed by measuring how well the generated aspect-based summary enables an LLM to answer these aspect-specific QA, indicating how much information about the target aspect the summary truly captures. 3.2 QA Generation Process Figure 3: QA Generation Process. (1) splitting the text into chunks and extracting entities and relations, (2) inserting the extracted entities and relations into a knowledge graph as nodes and edges, and (3) synthesizing aspect-specific QA pairs from the completed graph. An overview of the QA generation process is illustrated in Figure 3 . The process consists of three stages: (1) splitting the text into chunks and extracting entities and relations, (2) inserting the extracted entities and relations into a knowledge graph as nodes and edges, and (3) synthesizing aspect-specific QA pairs from the completed graph. (1). Chunking and Extraction Each book is split into chunks of 1,200 characters with an overlap of 100 characters, following the parameters of GraphRAG ( 4 ) . From each chunk, entities (e.g., characters, events, concepts) are extracted using an LLM with a specifically designed prompt (2-shot, Appendix C , Figure 6 ). For each extracted relation, the prompt instructs the LLM to output a textual description, representative keywords, and an importance score ranging from 1 to 10, reflecting the importance of the relationship within the local context. (2). Knowledge Graph Construction The extracted entities and relations are incrementally inserted into a knowledge graph, where each edge is labeled with keywords, a textual description, and an importance score. If an entity already exists, its information is updated and summarized as needed, with keywords regenerated accordingly. In addition, importance score is accumulated by adding the newly assigned value to reflect repeated or strengthened relationships across chunks. (3). QA Generation Once the knowledge graph is constructed, we generate aspect-specific QA pairs. We first filter edges to keep