LongSumEval: Question-Answering Based Evaluation and Feedback-Driven Refinement for Long Document Summarization

1 st Huyen Nguyen, 2 nd Haoxuan Zhang, 3 rd Yang Zhang, 4 th Haihua Chen, 5 rd Junhua Ding 1 · 2026

Read LongSumEval: Question-Answering Based Evaluation and Feedback-Driven Refinement for Long Document Summarization by 1 st Huyen Nguyen, 2 nd Haoxuan Zhang, 3 rd Yang Zhang, 4 th Haihua Chen, 5 rd Junhua Ding 1 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-25130.txt.

Introduction

Nguyen and four colleagues introduce LongSumEval, a framework that judges long-document summaries through structured question answering. Old metrics match human judgment weakly and give aggregate scores with no account of flaws or path to improvement. Here summary quality means answerability and factual alignment of question-answer pairs, giving clear scores plus feedback on coverage gaps and factual errors. Across seven benchmarks the method agreed far more with human judges. Its feedback improved summaries by self-refinement with no retraining.

Catalog record

CREATOR1 st Huyen Nguyen, 2 nd Haoxuan Zhang, 3 rd Yang Zhang, 4 th Haihua Chen, 5 rd Junhua Ding 1
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60590
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.25130

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LongSumEval: Question-Answering Based Evaluation and Feedback-Driven Refinement for Long Document Summarization 1 st Huyen Nguyen, 2 nd Haoxuan Zhang, 3 rd Yang Zhang, 4 th Haihua Chen, 5 rd Junhua Ding 1 Abstract Evaluating long document summaries remains the primary bottleneck in summarization research. Existing metrics correlate weakly with human judgments and produce aggregate scores without explaining deficiencies or guiding improvement, preventing effective refinement in applications requiring verifiable accuracy. We introduce LongSumEval , a unified framework bridging evaluation and generation through structured question-answering feedback. The framework operationalizes summary quality as answerability and factual alignment of question-answer pairs, generating interpretable scores and actionable feedback that identifies coverage gaps and factual inconsistencies. This resolves the misalignment where evaluation operates independently of generation objectives. Meta-evaluation of our QA-based evaluation module across seven benchmarks demonstrates substantially stronger agreement with human judgments compared to established metrics. Structured feedback enables significant quality improvements through self-refinement without retraining. By demonstrating that evaluation feedback can serve as executable instructions for generation, this work establishes a generalizable paradigm for aligning assessment with improvement, with direct implications for controllable text generation requiring verifiable accuracy and transparent quality control. All code and datasets will be released in GitHub for reproducibility. I Introduction The exponential growth of textual information overwhelms manual review capacity, necessitating automatic summarization systems that distill lengthy documents into concise representations. Recent advances in large language models (LLMs) have transformed the field [ 25 , 47 ] , enabling deployment in critical domains including biomedical literature review [ 21 , 4 ] , legal document analysis [ 28 ] , and clinical decision support [ 16 ] , where efficient information access directly impacts decision quality. Despite significant progress in generation capabilities, evaluating summarization quality remains a fundamental bottleneck [ 29 , 47 ] . Traditional metrics relying on lexical overlap (ROUGE [ 24 ] , BLEU [ 30 ] ) or semantic similarity (BERTScore [ 46 ] ) correlate poorly with human judgments on factual consistency, coverage, and fluency [ 21 , 11 ] . They penalize valid paraphrases, fail without reference summaries [ 5 ] , and exhibit fundamental misalignment with human quality perceptions. LLM-generated summaries often receive lower ROUGE scores despite superior linguistic quality [ 13 ] . Recent evaluations on long documents reveal that current automatic metrics fail to detect unfaithful claims effectively [ 20 ] , particularly in high-stakes domains like legal and medical summarization [ 6 , 4 ] . Moreover, conventional metrics produce opaque scores without diagnostic feedback [ 5 , 32 ] . Question-answering (QA) based evaluation offers a promising alternative: high-quality summaries should answer salient questions about source documents while maintaining verifiable claims [ 5 , 32 , 37 , 7 ] . Representative methods include QuestEval [ 32 ] for bidirectional consistency assessment, QAEval [ 5 ] for importance-weighted coverage evaluation, and QAGS [ 37 ] for factual inconsistency detection, with extensions targeting dialogue evaluation [ 18 ] , fine-grained assessment [ 45 ] , and answer quality scoring [ 10 ] . However, existing frameworks face critical limitations: they generate predominantly factoid questions while neglecting conceptual information [ 10 ] , return aggregated scores without structured feedback specifying missing or incorrect information [ 5 , 32 ] , and struggle with computational costs and context constraints [ 21 ] . These challenges intensify for long documents exceeding 10,000 words, which contain distributed information, dense technical content, and complex argumentation structures [ 21 , 27 , 2 ] . Evaluation frameworks designed for short news articles fail to scale effectively [ 25 , 47 ] , struggling to identify whether summaries capture scattered key information or introduce subtle factual distortions [ 22 , 17 ] . Beyond evaluation, the disconnect between assessment and improvement represents a critical gap. While self-refinement methods demonstrate that LLMs can iteratively improve outputs with explicit feedback [ 26 , 3 , 19 ] , existing approaches face two limitations. Summarization-specific methods like SummIt [ 44 ] and multi-fact correction [ 8 ] lack explicit feedback specifications, while recent advances in structured feedback [ 38 , 41 , 1 ] operate independently of evaluation metrics, creating misalignment between assessment and correction [ 12 ] . For long documents, generic critiques prove insufficient. Targeted refinement requires precise identification of coverage gaps and factual inconsistencies [ 21 ] . Therefore, we propose LongSumEval , a unified framework integrating LLM-based QA evaluation with feedback-driven refinement. Our approach introduces two key innovations. First, we design a QA evaluation module assessing coverage and factual consistency [ 21 , 11 ] through diverse question types [ 10 ] , producing structured feedback that identifies unanswered questions (coverage gaps) and inconsistent fact triplets (erroneous claims with ground-truth corrections). Unlike conventional QA metrics returning scalar scores [ 5 , 32 , 33 ] , our method enables transparent diagnosis through human-inspectable question-answer pairs. Second, inspired by meta-evaluation and critique-guided improvement [ 38 , 41 ] , we develop a self-refinement module converting structured feedback into natural language instructions for targeted revision. Unlike previous frameworks lacking explicit feedback specifications [ 26 , 44 ] , this approach provides fine-grained, actionable guidance for correcting specific deficiencies. To validate LongSumEval’s effectiveness, we formulate three research questions: • RQ1: How effectively does LLM-based QA evaluation correlate with human quality judgments across diverse domains and document lengths compared to conventional QA-based metrics? • RQ2: What impact do design choices (question generation strategies, answer similarity measures, and LLM model selection) have on evaluation reliability and domain robustness? • RQ3: Does iterative refinement guided by structured QA-based feedback lead to measurable improvements in summary coverage and factual consistency without model fine-tuning? We conduct systematic meta-evaluation across seven human-annotated datasets spanning five domains (news, scientific writing, government reports, social media, and patents) with document lengths ranging from 400 to 27,000 words. For RQ1, we benchmark our QA-based evaluation module against established QA-based metrics including QuestEval [ 32 ] , QAEval [ 5 ] , and SummaQA [ 33 ] , demonstrating superior correlation with human judgments particularly on long-document benchmarks. For RQ2, we analyze how question diversity [ 10 ] , similarity thresholds, and LLM backend capabilities influence metric reliability across domains. Addressing RQ3, we demonstrate that structured feedback-driven self-refinement [ 26 ] achieves substantial quality improvements, with coverage scores increasing by up to 83.72% and consistency scores improving by up to 47.42% on initially low-quality summaries. Our contributions are fourfold: • We design an interpretable LLM-based QA evaluation framework that produces fine-grained structured feedback alongside quantitative scores, enabling transparent reference-free assessment of long document summaries across coverage and factual consistency dimensions. • We introduce a feedback-driven self-refinement mechanism that bridges evaluation and generation by converting structured QA-based feedback into actionable natural language instructions, enabling iterative summary improvement through targeted revisions that address specific coverage gaps and factual inconsistencies. • We conduct a comprehensive meta-evaluation of our QA-based evaluation module across seven human-annotated datasets spanning five domains and multiple document lengths, demonstrating superior correlation with human judgments compared to existing QA-based metrics, achieving Kendall’s \tau_{b} up to 0.683 for consistency evaluation and 0.738 for coverage assessment on long-document benchmarks. • We introduce PatentSumEval , a human-annotated benchmark for patent document summarization

Read the complete text