Is Peer Review Really in Decline? Analyzing Review Quality across Venues and Time
Ilia Kuznetsov, Rohan Nayak, Alla Rozovskaya, Iryna Gurevych · 2026
Read Is Peer Review Really in Decline? Analyzing Review Quality across Venues and Time by Ilia Kuznetsov, Rohan Nayak, Alla Rozovskaya, Iryna Gurevych (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-15172.txt.
Introduction
Ilia Kuznetsov, Rohan Nayak, Alla Rozovskaya, and Iryna Gurevych ask whether peer review is truly getting worse. Their 2026 arXiv paper deals with the system at the heart of modern science. Submissions keep rising and communities keep growing, and many fear review quality is in decline.
But is that fear true? Review quality is hard to measure in the first place. Review customs also change over time, which makes fair comparison tricky. The authors analyze reviews across venues and across years. They test the decline story against what the records actually show.
Catalog record
| CREATOR | Ilia Kuznetsov, Rohan Nayak, Alla Rozovskaya, Iryna Gurevych |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 79 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-15172
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Is Peer Review Really in Decline? Analyzing Review Quality across Venues and Time Ilia Kuznetsov, Rohan Nayak, Alla Rozovskaya, Iryna Gurevych Abstract Peer review is at the heart of modern science. As submission numbers rise and research communities grow, the decline in review quality is a popular narrative and a common concern. Yet, is it true? Review quality is difficult to measure, and the ongoing evolution of reviewing practices makes it hard to compare reviews across venues and time. To address this, we introduce a new framework for evidence-based comparative study of review quality and apply it to major AI and machine learning conferences: ICLR, NeurIPS and *ACL. We document the diversity of review formats and introduce a new approach to review standardization. We propose a multi-dimensional schema for quantifying review quality as utility to editors and authors, coupled with both LLM-based and lightweight measurements. We study the relationships between measurements of review quality, and its evolution over time. Contradicting the popular narrative, our cross-temporal analysis reveals no consistent decline in median review quality across venues and years. We propose alternative explanations, and outline recommendations to facilitate future empirical studies of review quality. 1 1 1 Data and code are available at https://github.com/UKPLab/arxiv2026-review-quality-estimation † † * Equal Contribution. 1 Introduction Peer review is central to the modern academic practice, and helps establish the epistemic status of research findings (“Is this science?”) and prioritize research outputs. The results of peer review determine funding decisions, shape public opinion, and inform policy Russo et al. 2025 . Peer review is deployed globally and by most scientific fields, across a wide range of publication and presentation formats. Computer science and machine learning conferences are no exception and use peer review as the primary quality control mechanism. Despite its merits, peer review is time-consuming Publons 2019 and prone to bias, error and randomness Lee et al. 2013 ; Rogers and Augenstein 2020 ; Beygelzimer et al. 2023 . Acceptance decisions at major computer science conferences are accompanied by frustration on social media, 2 2 2 Paraphrasing a Reddit user, “You spend a week working like a dog, running pointless experiments to make reviewers happy so that they increase your score.” Reddit 2025 and the concerns about the declining review quality are widespread Kim et al. 2025 ; Adam 2025 . There are indeed reasons to suspect that peer review is in decline: qualified reviewers are scarce, increasing specialization requires deeper expertise, reviewers and editors suffer high workloads, and irresponsible use of large language models (LLMs) creates new risks Russo et al. 2025 . Yet, there are also reasons to believe that review quality didn’t decline after all. Conferences refine their reviewing workflows, the pool of experts grows with each completed reviewing campaign, and ethical use of AI assistance can support reviewers in their work. All in all, the evidence for either side remains largely anecdotal Adam 2025 , and the true state of peer review is unknown. Figure 1: Framework overview. A review report r is written for a paper p by a reviewer a , and originates from a reviewing campaign c that takes place as part of a venue v at time t , e.g. ICLR-2018. Reviews differ in terms of their quality Q(r) , approximated through a range of measurements m(r)\in M along a set of quality criteria. Aggregating review-level measurements allows comparative analysis of review quality across reviewing campaigns. To study trends in review quality over time, evaluate the effectiveness of new tools and policies, and detect low-quality reviews, an objective and scalable measure of review quality is needed. Yet, arriving at such measure is difficult: review quality is hard to formalize, human judgment is prone to bias, and extrinsic measurements such as citation counts are easily distorted (Section 3 ). Measuring review quality on real-world data presents further challenges: the data is scarce, biased and noisy, and the diversity and evolution of reviewing workflows makes consistent application of review quality measures difficult. While the need to quantify review quality has been much acknowledged, a general framework for a multi-dimensional analysis of review quality at scale is missing. To close this gap, this paper makes the following contributions : • We propose the first holistic framework for the computational study of review quality at scale (Figure 1 ) and instantiate it in the domain of computer science conferences; • We identify challenges in unifying reviewing data across conferences and time, and develop a novel LLM-based approach for meaning-preserving review structure unification ; • We develop a novel review quality schema that spans multiple dimensions – substantiveness, actionability, and grounding. Each dimension is represented by several automated measurements , both lightweight (simple and efficient) and LLM-based. • We apply this schema to study the interactions between different measurements and to analyse the changes in review quality at computer science conferences over time. We find that peer reviewing data is very diverse, but LLMs can mitigate some of these differences by transforming reviews into a common form. We observe that the measurements of review quality are correlated but complementary, and that lightweight measurements can approximate LLM-based measurements. Contrary to the widespread perception, we find that the median review quality at computer science conferences shows no consistent decline. Yet, the question proves nuanced, and we provide five hypotheses coupled with recommendations to researchers, conference organizers and community leaders to move this work forward. 2 Scope Peer review is a multi-stage process spanning from the call for papers to acceptance decisions, and a failure at any stage can undermine the goals of peer review Kuznetsov et al. 2024 . Review reports 3 3 3 We refer to reviews and review reports interchangeably. are central to this process and are the most consistently documented reviewing artifact Kang et al. 2018 ; Lin et al. 2023 ; Dycke et al. 2023a . Thus, our study focuses on the quality of review reports . Peer review involves stakeholders with distinct and often contradictory goals: reviewers seek efficiency; authors seek acceptance and constructive feedback; editors aim to select sound, impactful and thematically fitting papers. We focus on modeling review quality in terms of utility for both editors and authors . In terms of domain, we focus on conferences in computer science that make their peer reviews available: The International Conference on Learning Representations (ICLR), The Conference on Neural Information Processing Systems (NeurIPS), and ACL Rolling Review (ARR). These venues employ double-blind pre-publication review Wiley 2025 and experience rapid submission growth, serving as a representative sample of peer reviewing in the community. 3 Related work Approaches to measuring review quality fall into three categories. Survey-based approaches rely on human judgment, from flagging problematic reviews Rogers et al. 2023 to structured measurement instruments van Rooyen et al. 1999 . While insightful, human judgment is prone to bias Goldberg et al. 2025 , does not scale, and is hard to apply retroactively, making it a poor fit for comparative analysis. Extrinsic approaches evaluate the downstream effectiveness of peer review, e.g. through the use of future paper citations Plank and van Dalen 2019 ; Maddi and Miotti 2024 . While useful for measuring the impact of peer review as a system, extrinsic measurements