NovBench: Evaluating Large Language Models on Academic Paper Novelty Assessment
Wenqing Wu, Yi Zhao, Yuzhuo Wang, Siyou Li, Juexi Shao, Yunfei Long, Chengzhi Zhang · 2026
Read NovBench: Evaluating Large Language Models on Academic Paper Novelty Assessment by Wenqing Wu, Yi Zhao, Yuzhuo Wang, Siyou Li, Juexi Shao, Yunfei Long, Chengzhi Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-11543.txt.
Introduction
Wu and six colleagues introduce NovBench, the first large-scale benchmark for judging how well language models assess the novelty of academic papers. It holds 1,684 paper-review pairs from a leading NLP conference, pairing novelty claims from introductions with expert-written novelty evaluations. A four-part framework scores generated evaluations on relevance, correctness, coverage, and clarity. Tests on general and specialized models show limited grasp of novelty, with fine-tuned models often failing to follow instructions.
Catalog record
| CREATOR | Wenqing Wu, Yi Zhao, Yuzhuo Wang, Siyou Li, Juexi Shao, Yunfei Long, Chengzhi Zhang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56409 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.11543
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
NovBench: Evaluating Large Language Models on Academic Paper Novelty Assessment Wenqing Wu, Yi Zhao, Yuzhuo Wang, Siyou Li, Juexi Shao, Yunfei Long, Chengzhi Zhang Abstract Novelty is a core requirement in academic publishing and a central focus of peer review, yet the growing volume of submissions has placed increasing pressure on human reviewers. While large language models (LLMs), including those fine-tuned on peer review data, have shown promise in generating review comments, the absence of a dedicated benchmark has limited systematic evaluation of their ability to assess research novelty. To address this gap, we introduce NovBench, the first large-scale benchmark designed to evaluate LLMs’ capability to generate novelty evaluations in support of human peer review. NovBench comprises 1,684 paper–review pairs from a leading NLP conference, including novelty descriptions extracted from paper introductions and corresponding expert-written novelty evaluations. We focus on both sources because the introduction provides a standardized and explicit articulation of novelty claims, while expert-written novelty evaluations constitute one of the current gold standards of human judgment. Furthermore, we propose a four-dimensional evaluation framework (including Relevance, Correctness, Coverage, and Clarity) to assess the quality of LLM-generated novelty evaluations. Extensive experiments on both general and specialized LLMs under different prompting strategies reveal that current models exhibit limited understanding of scientific novelty, and that fine-tuned models often suffer from instruction-following deficiencies. These findings underscore the need for targeted fine-tuning strategies that jointly improve novelty comprehension and instruction adherence. 1 Introduction Novelty is a fundamental aspect of publication decisions in academic research, requiring that a paper’s content or methodology makes a meaningful contribution to advancing existing knowledge, rather than simply replicating or validating established findings ( 56 ) . As the primary quality control mechanism in scientific research, assessing novelty is one of the core functions of the peer review system ( 3 ; 8 ) . However, the peer review process is currently facing pressure due to the explosive growth in academic submissions ( 46 ) and the widening gap in the availability of qualified reviewer resources ( 29 ) . This pressure directly manifests as a significant challenge for the robust evaluation of a paper’s novelty ( 72 ) . Although methods for novelty evaluation have been proposed based on bibliometric ( 55 ; 41 ; 51 ) and deep learning ( 18 ; 20 ; 37 ) approaches, these techniques predominantly focus on quantitative metrics (or numerical indicators). Compared to the textual evaluations provided by reviewers, these quantitative metrics inherently lack the necessary interpretability, making it difficult to effectively assist reviewers in their judgment during the peer review process or to provide authors with targeted advice for manuscript improvement. Large Language Models (LLMs) have demonstrated exceptional capabilities across a wide range of scientific tasks ( 58 ; 4 ) and show potential in assisting academic peer review ( 73 ; 76 ) . Recent researches have attempted to enhance the capabilities of LLMs in automated peer review through various avenues, such as applying more effective prompting strategies ( 33 ) , fine-tuned model ( 66 ; 75 ) , and multi-agent frameworks ( 22 ; 13 ) . Despite the promising progress demonstrated by recent advances, several critical issues remain insufficiently explored. Although these methods achieve competitive performance on paper-level scoring tasks, the evaluation of generated review text itself has received relatively little attention. Existing evaluation approaches are largely limited to metrics such as ROUGE ( 34 ) , BLEU ( 45 ) , and BERTScore ( 71 ) , or the adoption of the LLM-as-judge ( 5 ) . However, these methods either rely on surface-level lexical similarity or depend on non-transparent LLM-based judgments, and therefore fail to reliably assess the semantic adequacy and aspect-specific correctness required for evaluating free-form novelty evaluations ( 26 ) . Furthermore, existing research primarily focuses on generating holistic peer review text, treating the review as a monolithic output. As a result, the ability of both general purpose and fine tuned models to perform aspect specific evaluation, particularly novelty evaluation, remains poorly characterized. Without isolating novelty as an independent evaluation target, it is difficult to determine whether a model genuinely evaluates research novelty or merely generates fluent, plausible sounding review like language. Therefore, the performance of existing general and fine-tuned models on the novelty evaluation task constitutes an important area for further investigation. To address these issues, we present a novel evaluation framework, to be more specific: (1) We introducing a structured benchmark for novelty evaluation. This resource, which incorporates the’ textual evaluations of the novelty dimension of the reviewers alongside the novelty descriptions of the paper introductions, constitutes a critical resource for future research. (2) We propose a four-dimensional, interpretable, and semantics-aware framework for evaluating LLM-generated novelty review text. This metric suite surpasses the limitations of traditional lexical overlap metrics (e.g., ROUGE/BLEU) and more effectively captures the quality of "novelty" evaluation in free-form text. (3) We conduct systematic evaluation and benchmarking of current general LLMs (e.g., GPT-5 ( 44 ) , Gemini-2.5-flash ( 16 ) ) and specialized LLMs on the novelty evaluation task. Based on this benchmark, we further analyze the gap between LLM-generated evaluations and human judgments, providing an in-depth understanding of these models’ advantages and limitations in identifying and articulating novelty. (4) We conduct a comprehensive empirical analysis that identifies key factors for generating high-quality, highly interpretable novelty review text, and reveals important behavioral patterns of LLMs in novelty evaluation, thus directing the future development of more reliable AI-assisted peer review. 1 1 1 All resources publicly available at https://github.com/njust-winchy/llm4novelty . 2 Related Work Automated Scholarly Paper Review (ASPR) ( 36 ) refers to the process in which computers or intelligent machines independently evaluate the content of a scholarly paper and generate a review report automatically. Early research concerning ASPR predominantly focused on paper rating recommendation ( 23 ; 31 ; 57 ) . Furthermore, some studies ( 68 ; 69 ) have attempted to fine-tune pre-trained models to generate paper reviews. With the LLMs demonstrate powerful text generation capabilities ( 65 ; 7 ) , the application of LLMs to help peer review has rapidly become a significant research focus. Numerous studies have evaluated or benchmarked the performance of LLMs in generating reviews of academic papers ( 73 ; 33 ; 11 ) . The results of these investigations indicate that while LLMs are capable of providing meaningful feedback, they often lack critical analysis, and the comments generated frequently lack the insights and specificity found in human-written reviews. Consequently, the research community has devoted significant efforts for enhancing LLM performance in review generation ( 67 ; 15 ; 19 ; 59 ; 74 ; 6 ) . However, recent advancements have broadly evaluated and improved the overall performance of LLMs in review generation. Despite this progress, their efficacy in assessing specific, fine-grained aspects of a paper (especially novelty) remains an area requiring further investigation. Novelty is recognized as a key aspect for measuring the quality and contribution of academic papers. Concurrently, as LLMs continue to demonstrate increasing capability, researchers have advanced their investigation into the role of LLMs in scientific novelty, including LLM-based assessment of paper novelty ( 18 ; 37 ; 35 ; 38 ; 61 ;