Policies Permitting LLM Use for Polishing Peer Reviews Are Currently Not Enforceable

Rounak Saha, Gurusha Juneja, Dayita Chaudhuri, Naveeja Sajeevan, Nihar B Shah, Danish Pruthi · 2026

Read Policies Permitting LLM Use for Polishing Peer Reviews Are Currently Not Enforceable by Rounak Saha, Gurusha Juneja, Dayita Chaudhuri, Naveeja Sajeevan, Nihar B Shah, Danish Pruthi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-20450.txt.

Introduction

Saha and co-authors test whether rules allowing LLM polishing of peer reviews can be enforced. Many venues now ban LLM use by reviewers except for polishing and grammar help. They built a data set spanning levels of human-AI collaboration on reviews. Five leading detectors, including two commercial systems, judged the texts. All detectors misread a real share of polished reviews as fully AI-written. That error risks false misconduct charges against honest reviewers. Peer-review signals helped slightly, but no method reached the needed accuracy.

Catalog record

CREATORRounak Saha, Gurusha Juneja, Dayita Chaudhuri, Naveeja Sajeevan, Nihar B Shah, Danish Pruthi
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE85695
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.20450

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Policies Permitting LLM Use for Polishing Peer Reviews Are Currently Not Enforceable Rounak Saha, Gurusha Juneja, Dayita Chaudhuri, Naveeja Sajeevan, Nihar B Shah, Danish Pruthi Abstract A number of scientific conferences and journals have recently enacted policies that prohibit LLM usage by peer reviewers, except for polishing, paraphrasing, and grammar correction of otherwise human-written reviews. But, are these policies enforceable? To answer this question, we assemble a dataset of peer reviews simulating multiple levels of human-AI collaboration, and evaluate five state-of-the-art detectors, including two commercial systems. Our analysis shows that all detectors misclassify a non-trivial fraction of LLM-polished reviews as AI-generated, thereby risking false accusations of academic misconduct. We further investigate whether peer-review-specific signals, including access to the paper manuscript and the constrained domain of scientific writing, can be leveraged to improve detection. While incorporating such signals yields measurable gains in some settings, we identify limitations in each approach and find that none meets the accuracy standards required for identifying AI use in peer reviews. Importantly, our results suggest that recent public estimates of AI use in peer reviews through the use of AI-text detectors should be interpreted with caution, as current detectors misclassify mixed reviews (collaborative human-AI outputs) as fully AI generated, potentially overstating the extent of policy violations. 1 1 footnotetext: Equal contribution. Code : https://github.com/FLAIR-IISc/ai-in-peer-review Data : https://huggingface.co/datasets/rounaksaha12/ai-in-peer-review 1 Introduction Scientific journal editors and conference program chairs routinely rely on experts to evaluate submitted manuscripts through peer review. However, there have been growing concerns lately about reviewers delegating this task to AI systems, prompting journals and conferences to adopt a range of policies governing AI use in the review process ( 23 ; 58 ) . These concerns have also been a part of public discourse. A notable recent example involves the ICLR 2026 conference, whose reviews are publicly released. An AI-text detection company claimed that 21 % of submitted reviews were AI-generated ( 39 ) . This claim attracted widespread attention, but was discussed and reported ( 35 ) with little scrutiny. Two types of policies are prominently adopted by conferences and journals. One policy is to completely prohibit use of LLMs by reviewers (e.g., by 23 ), and we will refer to this as No-LLM-use policy . A second common policy is to allow reviewers to use LLMs only for paraphrasing and grammar correction of their reviews, and where the core arguments and content of the review must be human-written (e.g., by 22 , 1 , 51 ). We collectively refer to these policies as Polishing-only policies . For instance, 13 , citing the ACL Policy on Publication Ethics, states: “it is acceptable to use LLMs for paraphrasing, grammatical checks and proof-reading, but not for the content of the (meta-)reviews.” — Criteria for Determining Irresponsible Reviewers, 13 With such policies precisely delineating acceptable LLM use, and detectors claiming to be capable of screening tens of thousands of reviews overnight, it might be tempting to conclude that the machinery for enforcing these policies is already in place. Enforcement of policies, however, hinges critically on the key question: To what extent can the extent of AI use in peer reviews be detected? To put things into perspective, at the scale of NeurIPS 2025 (over 75,000 reviews), even a 0.1\% false positive rate can translate to 75 wrongful accusations of academic misconduct. A 3\% false positive rate can result in an alarmingly high 2,250 wrongful accusations. Therefore, any credible enforcement regime would demand a far lower false positive rate still. Yet, it remains largely unexamined how reliable are these detectors in detecting AI use in peer reviews, and how much confidence we should place on their outputs. In this work, we directly investigate these questions by evaluating state-of-the-art AI text detectors on a curated dataset of reviews spanning multiple levels of human-AI collaboration. Data. We curate a dataset of 50,156 peer reviews spanning multiple levels of human-AI collaboration. We start with human -written reviews from the pre-ChatGPT era to ensure reviews are free of AI assistance, specifically from CoNLL 2016, ACL 2017, ICLR 2017, and NeurIPS 2013–2017, sourced from the PeerRead dataset ( 25 ) . We then use popular LLMs to generate synthetic reviews for the same papers to simulate the following levels of AI-assistance: (1) AI-generated with Basic Prompts ( AI-BP ): Reviews generated by prompting an LLM with the paper and conference reviewing guidelines; (2) AI-generated with Elaborate Prompts ( AI-EP ): Reviews generated by prompting with additional tips apart from paper and official reviewing guidelines; (3) AI-generated with Human Input ( AI-HI ): Reviews generated by an LLM by expanding on key assessment points extracted from a human-written review, simulating minimal human input; (4) Human-written AI-polished ( H-AI ): Human-written reviews polished by an LLM for grammar, flow and clarity, the only form of LLM use permitted under Polishing-only policies; and (5) Humanized AI-BP and H-AI reviews, adversarially paraphrased to evade AI detection. Detectors evaluated. We evaluate five AI text detectors, comprising three open-source and zero-shot AI detectors, namely LogLikelihood ( 50 ) , Fast-DetectGPT ( 5 ) , and Binoculars ( 18 ) , which output scalar detection scores, as well as two commercial classifiers, Pangram ( 12 ; 52 ) and GPTZero ( 54 ) , which produce multi-class categorical predictions (“AI”, “Mixed”, “Human”). For the zero-shot detectors, we generate binary predictions by thresholding their output scores. The thresholds are chosen to achieve a hard 0\% false positive rate on human-written reviews in a calibration set. Potential peer-review-specific advantages. As opposed to general AI-text detection, the application of detecting AI use in peer reviews presents unique opportunities. For instance, the paper manuscript (under review) is likely a part of the prompt used to generate the review. Most open-source detectors like LogLikelihood, DetectGPT, Fast-DetectGPT and Binoculars rely on perplexity or surprisal-based metrics, which can be naturally extended to incorporate paper context by computing the likelihood of a review conditioned on the manuscript. Another way to incorporate paper context is to first generate a set of LLM-written reference reviews for the manuscript, and then use similarity between a candidate review and these references as a detection signal ( 60 ) . Secondly, peer reviews follow certain writing norms and styles, and the content is scientific in nature, which makes supervised training on domain-specific examples a promising candidate. This raises the question: Does the presence of peer-review-specific signals help overcome the limitations of fine-grained AI detection in peer reviews? Main findings. We summarize our key findings below: • Pangram and GPTZero successfully identify 98.3\% and 95.8\% of fully AI-generated reviews (AI-BP and AI-EP), respectively. For reviews that are fully human written, Pangram flags none of them as “AI” or “Mixed”, while GPTZero flags 1\% of such reviews as ‘‘AI.’’ 1 1 1 The human reviews are drawn from the PeerRead dataset, which might have appeared in the training data of Pangram and GPTZero. Therefore, these results are potentially optimistic estimates. More on this in the note on contamination (Sec. 4.1 ) • All detectors struggle with the Human-written AI-polished (H-AI) reviews. These reviews are compliant under a Polishing-only policy. However, Pangram and GPTZero have a non-trivially high false positive rate, classifying 3.1\% and 3.4\% of

Read the complete text