LazyReview A Dataset for Uncovering Lazy Thinking in NLP Peer Reviews
Sukannya Purkayastha, Zhuang Li, Anne Lauscher, Lizhen Qu, Iryna Gurevych · 2025
Read LazyReview A Dataset for Uncovering Lazy Thinking in NLP Peer Reviews by Sukannya Purkayastha, Zhuang Li, Anne Lauscher, Lizhen Qu, Iryna Gurevych (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-11042.txt.
Introduction
Sukannya Purkayastha and four co-authors build LazyReview, a dataset for catching shallow, heuristic-driven peer reviews in natural language processing. Reviewers under heavy load fall back on quick judgments, and the field had no real-world data for training detectors. Their dataset annotates individual review sentences with fine-grained lazy thinking categories.
Large language models, tested cold, struggled to spot these patterns. Fine-tuning on the new data lifted performance by ten to twenty points, underlining the value of quality annotations. A controlled experiment added practical weight: reviews revised with lazy-thinking feedback became more comprehensive and actionable. The authors will release the dataset and guidelines, aiming also to train junior reviewers.
Catalog record
| CREATOR | Sukannya Purkayastha, Zhuang Li, Anne Lauscher, Lizhen Qu, Iryna Gurevych |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 94 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-11042
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LazyReview A Dataset for Uncovering Lazy Thinking in NLP Peer Reviews Sukannya Purkayastha, Zhuang Li, Anne Lauscher, Lizhen Qu, Iryna Gurevych Abstract Peer review is a cornerstone of quality control in scientific publishing. With the increasing workload, the unintended use of ‘quick’ heuristics, referred to as lazy thinking , has emerged as a recurring issue compromising review quality. Automated methods to detect such heuristics can help improve the peer-reviewing process. However, there is limited NLP research on this issue, and no real-world dataset exists to support the development of detection tools. This work introduces LazyReview , a dataset of peer-review sentences annotated with fine-grained lazy thinking categories. Our analysis reveals that Large Language Models (LLMs) struggle to detect these instances in a zero-shot setting. However, instruction-based fine-tuning on our dataset significantly boosts performance by 10-20 performance points, highlighting the importance of high-quality training data. Furthermore, a controlled experiment demonstrates that reviews revised with lazy thinking feedback are more comprehensive and actionable than those written without such feedback. We will release our dataset and the enhanced guidelines that can be used to train junior reviewers in the community. 1 1 1 Code available here: https://github.com/UKPLab/acl2025-lazy-review 1 Introduction Peer Reviewing is widely regarded as one of the most effective ways to assess the quality of scientific papers Ware and Mabe (2009) . It is a distributed procedure where the experts ( reviewers ) independently evaluate whether a submitted manuscript adheres to the standards of the field. With the Mathew effect Merton (1968) in science (“rich get richer”), where the researchers receive benefits throughout their career for having papers at prestigious venues, it is of utmost importance to ensure sound practices in the reviewing process. With the growing load of paper submissions, reviewers often face an overwhelming workload Landhuis (2016) ; Künzli et al. (2022) to assess multiple manuscripts quickly . When presented with a cognitively hard task (e.g., reviewing) coupled with information overload and limited time, humans often end up using simple decision rules, also known as heuristics Tversky et al. (1974) . Though simple and efficient, these heuristics can often lead to errors and unfair evaluations Raue and Scholl (2018) . Figure 1: Illustration of lazy thinking in ARR-22 reviews sourced from NLPeer Dycke et al. (2023) . The first review segment belongs to the class ‘The results are not novel.’ The last segment pertains to, ‘The approach is tested only on [not English], so unclear if it will generalize to other languages.’ as per ARR-22 guidelines. The usage of such heuristics to dismiss research papers in the Natural Language Processing (NLP) Community has been termed as lazy thinking Rogers and Augenstein (2021) . One such example is shown in Fig 1 . Here, the reviewer dismisses the paper in the first review segment for not being an “eye-opener”. However, they do not provide any references regarding similar prior work or feedback to improve the paper. This corresponds to the lazy thinking class ‘The results are not novel’. In 2021, when ACL Rolling Review (ARR) was adopted as one of the main reviewing platforms for major NLP conferences, these heuristics were added to the guidelines Rogers and Augenstein (2021) , aiming to discourage reviewers from relying on such approaches. 2 2 2 https://aclrollingreview.org/ However, in their ACL 2023 report, Rogers et al. (2023) identified the usage of these heuristics as one of the leading factors ( 24.3 %) of author-reported issues in peer reviews. Therefore, in this work, we focus on this pertinent issue, lazy thinking in NLP peer reviewing and heed Kuznetsov et al. (2024) ’s call for automated methods to signal such occurrences in order to improve the overall reviewing quality. To have a finer look at the problem of lazy thinking in NLP peer reviews, we introduce LazyReview , a dataset with 500 expert-annotated review segments and 1276 silver annotated review segments from the best-performing model tagged with lazy thinking classes with a review segment consisting of 1 or more review sentences. We develop this dataset over three rounds of annotation and incrementally sharpen the guidelines guided by inter-annotator agreements. We further provide positive examples, i.e., annotated examples for each class, to enhance the understanding of annotators for this task and reach annotation consensus sooner. Annotating review segments by a new batch of annotators who were not involved in developing the guidelines resulted in Cohen’s \kappa Landis and Koch (1977) values of 0.32 , 0.36 , and 0.48 , respectively. The steady increase in agreement across rounds justifies the effectiveness of the developed guidelines. With this dataset, we further test the zero-shot capabilities of LLMs in identifying lazy thinking , which can be leveraged downstream to identify rule-following behaviour Sun et al. (2024b) in peer-review scenarios. Despite likely exposure to peer reviews via platforms like OpenReview during pre-training, LLMs struggle to accurately identify the type of lazy thinking when presented with the current guidelines. To enhance their comprehension of lazy thinking , we employ instruction-tuning on the LLMs using our dataset, leading to significant performance gains—around 10-20 accuracy points compared to their zero-shot and few-shot performances. Finally, we perform a controlled experiment where human reviewers rewrite peer reviews with(out) using annotations of lazy thinking from our dataset. Human preference-based evaluations reveal that reviews written with the lazy thinking guidance are more comprehensive and actionable than their vanilla counterparts. Contributions. We make the following contributions: (1) Introduce LazyReview , a dataset annotated with lazy thinking classes for a new task in model development and evaluation. (2) Enhance annotation guidelines to improve both automated and human annotations. (3) Demonstrate that positive examples boost annotation quality and in-context learning. (4) Show that instruction tuning on our dataset enhances model performance. (5) Find that annotated lazy thinking classes improve review quality. 2 LazyReview : A Dataset for detecting Lazy Thinking in Peer Reviews In line with Rogers and Augenstein (2021) , we first define lazy thinking as follows. Definition . ‘Lazy thinking, in the context of NLP research paper reviews, refers to the practice of dismissing or criticizing research papers based on superficial heuristics or preconceived notions rather than thorough analysis. It is characterized by reviewers raising concerns that lack substantial supporting evidence and are often influenced by prevailing trends within the NLP community.’ Rogers and Augenstein (2021) enlist a total of 14 types of lazy thinking heuristics in the ARR 2021 guidelines adapted from the study in Rogers and Augenstein (2020) . We show some of the classes as described in the guidelines in Table 1 . 3 3 3 The full table is provided in Table 7 in Appendix § A.1 Re-iterating the example in Fig 1 , we note that such claims about novelty need to be backed up with supporting evidence and hence constitute a classic example of lazy thinking as per the guidelines. In this section, we describe the creation of our dataset, LazyReview guided by the ARR 2022 Rogers and Augenstein (2021) and the EMNLP 2020 guidelines Liu et al. (2020) . We describe the dataset curation process followed by an analysis of the dataset. This is the first dataset of annotated instances with fine-grained lazy thinking classes for NLP peer reviews. Heuristics Description Example review