Beyond “Not Novel Enough”: Enriching Scholarly Critique with LLM-Assisted Feedback
Osama Mohammed Afzal, Preslav Nakov, Tom Hope, Iryna Gurevych · 2025
Read Beyond “Not Novel Enough”: Enriching Scholarly Critique with LLM-Assisted Feedback by Osama Mohammed Afzal, Preslav Nakov, Tom Hope, Iryna Gurevych (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-10795.txt.
Introduction
Afzal and co-authors build an AI assistant that judges how novel a research paper is, in a 2025 study of peer review. Reviewers often dismiss work as not novel enough, with little evidence shown. In busy fields like natural language processing, that judgment is hard to make well. The team's system mimics expert reviewers in three stages: pull claims from the submission, gather related work, and compare the two in a structured way.
The method was shaped by a large study of real novelty reviews written by humans. On 182 submissions to a major conference, it matched human reasoning in 86.5 percent of cases and human novelty verdicts in 75.3 percent. It beat existing language model baselines by a wide margin. The authors offer it as support for reviewers, not a replacement.
Catalog record
| CREATOR | Osama Mohammed Afzal, Preslav Nakov, Tom Hope, Iryna Gurevych |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 64 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-10795
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Beyond “Not Novel Enough”: Enriching Scholarly Critique with LLM-Assisted Feedback Osama Mohammed Afzal, Preslav Nakov, Tom Hope, Iryna Gurevych Abstract Novelty assessment is a central yet understudied aspect of peer review, particularly in high-volume fields like NLP where reviewer capacity is increasingly strained. We present a structured approach for automated novelty evaluation that models expert reviewer behavior through three stages: content extraction from submissions, retrieval and synthesis of related work, and structured comparison for evidence-based assessment. Our method is informed by a large-scale analysis of human-written novelty reviews and captures key patterns such as independent claim verification and contextual reasoning. Evaluated on 182 ICLR 2025 submissions with human annotated reviewer novelty assessments, the approach achieves 86.5% alignment with human reasoning and 75.3% agreement on novelty conclusions—substantially outperforming existing LLM-based baselines. The method produces detailed, literature-aware analyses and improves consistency over ad hoc reviewer judgments. These results highlight the potential for structured LLM-assisted approaches to support more rigorous and transparent peer review without displacing human expertise. Data and code are made available. 1 1 1 https://ukplab.github.io/arxiv2025-assessing-paper-novelty 1 Introduction Figure 1: Comparison of a surface level human (Top) vs. LLM-written novelty assessment (Bottom). The peer review system is collapsing under its own success. Two independent committees at NeurIPS 2021 disagreed on 23% of identical papers Beygelzimer et al. 2023 —a breakdown in consistency that signals deeper problems than mere capacity constraints. With manuscript submissions doubling every 15 years Larsen and Ins 2010 and reviewers now handling 14 evaluations annually Díaz et al. 2024 , the system’s 15 million annual reviewing hours Aczel et al. 2021 are producing increasingly unreliable outcomes. Among peer review tasks, novelty assessment stands out as one of the most problematic Ernst et al. 2020 Horbach and Halffman 2018 . Novelty assessment requires reviewers to determine whether a submission makes sufficiently original contributions by identifying what specific advances it makes beyond existing work, evaluating whether these advances are significant enough to warrant publication, and verifying that the authors have accurately characterized their contributions relative to prior research. This knowledge-intensive process demands that reviewers maintain comprehensive awareness of related work across their field and can precisely distinguish between meaningful innovations and incremental modifications—a task that becomes exponentially more difficult as publication rates accelerate and research domains specialize. Overwhelmed reviewers often resort to superficial analyses, producing vague feedback like "not novel enough" without clear justification. The challenge compounds when reviewers encounter papers outside their specific expertise, leading to either overly conservative rejections or inadequate assessments that fail to catch incremental work Kuznetsov et al. 2024 . Recent advances in large language models present an unprecedented opportunity to address these novelty assessment challenges at scale. These breakthrough technologies have revolutionized text processing and demonstrated remarkable performance across knowledge-intensive tasks Raiaan et al. 2024 , with recent technical advancements expanding capabilities to specialized reasoning and efficient inference Li et al. 2024a ; Zhang et al. 2025 . While recent LLM advances create this opportunity, no existing work specifically addresses novelty assessment as a dedicated task within the peer review process. Prior research incorporates novelty evaluation within idea generation pipelines Radensky et al. 2025 ; Lu et al. 2024 ; Li et al. 2024b , generates peer reviews with novelty assessments occurring as a result of them existing in peer reviews from training data Idahl and Ahmadi 2025 ; D’Arcy et al. 2024 , or adds novelty assessment steps to review synthesis pipelines for improvement Zhu et al. 2025 . However, these approaches either operate on synthetic ideas rather than real research contributions or fail to evaluate novelty assessment capabilities in isolation. This represents a critical gap requiring specialized methodologies for peer review novelty assessment. To address this gap, we propose an end-to-end novelty assessment pipeline for peer review submissions. Our approach consists of three stages: document processing and content extraction, related work retrieval and ranking, and structured novelty assessment. The final stage implements four sequential steps: novelty related content selection from the submission pdf, building comprehensive understanding of related work from retrieved papers, comparing claimed novelty against the comprehensive analysis from the prior step, and generating a summary with cited evidence from the comparison. This pipeline operates on real research papers and directly evaluates novelty assessment capabilities, addressing the limitations of existing approaches. Importantly, we conduct the first evaluation of LLMs for novelty assessment using actual human data, including annotated novelty assessment statements, and provide comprehensive evaluation across multiple dimensions. Research Questions and Contributions This work aims to address the following research questions: 1. How does our human-informed novelty assessment pipeline compare to existing approaches? 2. How well do our assessments align with human reviewer preferences across key evaluation dimensions? 3. Can automated evaluation reliably substitute for human judgment in assessing novelty assessment quality? Our contributions are threefold: • Human Analysis Dataset and Insights : A systematically curated dataset of 182 papers with annotated human novelty assessments from ICLR 2025, along with empirical insights into expert reviewer reasoning patterns, evaluation criteria, and argument structures that inform AI system design for novelty assessment. • Human-Informed Pipeline : A literature-grounded pipeline that incorporates insights from human novelty assessment practices, featuring structured prompting strategies and targeted content extraction informed by observed expert reviewer behavior. • Comprehensive Evaluation and Analysis : Systematic comparison of our human-informed approach against existing baselines and human reviewers, with fine-grained evaluation across multiple dimensions and validation of automated assessment methods. Figure 2: Automated novelty assessment pipeline. The system processes manuscripts through three stages: (1) Document Processing extracts content using GROBID, (2) Related Work Discovery identifies and ranks relevant papers via embedding similarity and LLM reranking, and (3) Novelty Assessment performs structured analysis to generate evidence-based novelty evaluations. 2 Related Work AI-Assisted Peer Review Systems Our work is positioned at the peer review stage of scientific research, where our system operates when a manuscript is submitted for evaluation. While previous works D’Arcy et al. 2024 Idahl and Ahmadi 2025 Zhu et al. 2025 Chitale et al. 2025 Chang et al. 2025 Nemecek et al. 2025 have developed end-to-end peer review generation pipelines that may implicitly include novelty assessment steps, we are the first to focus specifically on building a dedicated pipeline for novelty assessment and the first to systematically evaluate LLMs on this task. A related line of work operates at the ideation stage of research Radensky et al. 2025 Shahid et al. 2025 Li et al. 2024b Lu et al. 2024 , developing pipelines for research idea generation that aim to improve novelty through feedback loops from a novelty assessor. In contrast, we operate at a more mature stage where ideas have been fully executed and comparative analyses are well-formulated. The evaluation in ideation-stage works focuses on synthetic ideas that are typically abstract and loosely defined, whereas we evaluate concrete, polished research contributions that have undergone the refinement process of execution and manuscript preparation. Scientific Literature Analysis & Retrieval Our work employs an extensive related work discovery pipeline that collects papers cited within the submission and additionally retrieves related papers by querying with prompts generated by GPT-4.1. Papers are then ranked using an embedding-based method and reranked using RankGPT. We adapt this general approach from existing work Radensky et al. 2025 Shahid