Detecting LLM-Generated Peer Reviews

Vishisht Rao, Aounon Kumar, Himabindu Lakkaraju, Nihar B. Shah · 2025

Read Detecting LLM-Generated Peer Reviews by Vishisht Rao, Aounon Kumar, Himabindu Lakkaraju, Nihar B. Shah (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-15772.txt.

Introduction

Peer review guards science, but some reviewers now feed papers to chat models and pass off the output as their own reading. Vishisht Rao and co-authors tackle that abuse in a 2025 paper. Hidden prompts tucked inside a paper file can nudge a chat model to plant a secret mark in any review it writes.

The trick only helps if the mark can be tested with firm math and few false alarms. The team builds tests that hold the error rate down across many reviews at once, with more power than the standard fix. They try font tricks and scrambled prompts to hide the seed text. Across several chat models the marks took hold often, and common reviewer defenses did not wash them out. The work gives venues a real tool, not just a ban on paper.

Catalog record

CREATORVishisht Rao, Aounon Kumar, Himabindu Lakkaraju, Nihar B. Shah
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE75 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-15772

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Detecting LLM-Generated Peer Reviews Vishisht Rao, Aounon Kumar, Himabindu Lakkaraju, Nihar B. Shah Abstract The integrity of peer review is fundamental to scientific progress, but the rise of large language models (LLMs) has introduced concerns that some reviewers may rely on these tools to generate reviews rather than writing them independently. Although some venues have banned LLM-assisted reviewing, enforcement remains difficult as existing detection tools cannot reliably distinguish between fully generated reviews and those merely polished with AI assistance. In this work, we address the challenge of detecting LLM-generated reviews. We consider the approach of performing indirect prompt injection via the paper’s PDF, prompting the LLM to embed a covert watermark in the generated review, and subsequently testing for presence of the watermark in the review. We identify and address several pitfalls in naïve implementations of this approach. Our primary contribution is a rigorous watermarking and detection framework that offers strong statistical guarantees. Specifically, we introduce watermarking schemes and hypothesis tests that control the family-wise error rate across multiple reviews, achieving higher statistical power than standard corrections such as Bonferroni, while making no assumptions about the nature of human-written reviews. We explore multiple indirect prompt injection strategies–including font-based embedding and obfuscated prompts–and evaluate their effectiveness under various reviewer defense scenarios. Our experiments find high success rates in watermark embedding across various LLMs. We also empirically find that our approach is resilient to common reviewer defenses, and that the bounds on error rates in our statistical tests hold in practice. In contrast, we find that Bonferroni-style corrections are too conservative to be useful in this setting. 1 Introduction In scientific peer review, expert reviewers are expected to provide thoughtful, critical, and constructive evaluations of submitted manuscripts and proposals [ Sha22 ] . Traditionally, the peer-review process has depended on reviewers’ intellectual engagement, domain expertise, and ability to articulate meaningful insights that enhance scholarly work. While natural language processing tools like large language models (LLMs) have improved certain aspects of peer review–such as identifying errors in manuscripts [ LS23 ] , verifying checklists [ GUK+24a ] , and assigning reviewers to papers [ CFB+20 ] –they have also introduced new challenges. A growing concern is the misuse of LLMs by disengaged reviewers who generate reviews using these tools with little or no personal input. Recent studies have sought to quantify this concern by analyzing review data from top AI conferences, including ICLR, NeurIPS, and EMNLP [ LRD+24 , LIZ+24 ] . For example, Latona et al. [ LRD+24 ] report that at least 15.8% of reviews submitted to ICLR 2024 were at least AI-assisted. These reviews were often associated with higher recommendation scores than those written by humans, raising concerns about potential bias in the peer review process. Such practices undermine the integrity of peer review and raise ethical issues related to originality and accountability. In response, many journals, conferences, and funding agencies have implemented policies that prohibit the use of LLMs to generate peer reviews [ NIH23 , Sci23a , Sci23b , ICM25 ] . For example, a notice released by NIH in 2023 [ NIH23 ] states that “NIH prohibits NIH scientific peer reviewers from using natural language processors, large language models, or other generative Artificial Intelligence (AI) technologies for analyzing and formulating peer review critiques for grant applications and R&D contract proposals.” However, enforcing these policies remains difficult in practice. Reviewers can easily upload a manuscript to an LLM and ask the LLM to write a review, thereby circumventing policy restrictions. Although detection tools such as GPTZero [ TC23 ] have been developed to identify LLM-generated text, they often struggle to distinguish between entirely LLM-generated reviews and those that have merely been edited or paraphrased using AI tools [ GCZ+23 , TZL23 , YLM+25 ] . In this work, we address the challenge of detecting peer reviews that are generated by large language models (LLMs). Reviewers often upload manuscripts provided by conference or journal organizers into LLMs to produce reviews, presenting an opportunity for intervention. We take advantage of this by embedding hidden instructions or prompts [ GAM+23 , YXZ+23 ] directly into the manuscript file. These prompts are indiscernible to human reviewers but are processed by LLMs when the manuscript is uploaded. They instruct the LLM to insert a distinctive watermark–such as a fake citation or a rare technical phrase–into the generated review. The presence of these watermarks enables reliable post hoc identification of reviews that were likely written by LLMs. More formally, we consider the following three-component framework for detecting LLM-generated reviews. First, we develop a watermarking strategy that stochastically selects specific phrases—such as fake citations or rare technical terms—to serve as detectable signals in LLM-generated reviews. Second, we employ indirect prompt injection techniques—including white-text cues, font manipulations, and cryptic prompts—to embed instructions into manuscript PDFs. These prompts are indiscernible to human reviewers but are processed by LLMs when the document is uploaded, prompting the model to embed the chosen watermark in its output (i.e., the review). Third, we develop a statistical detection method that identifies embedded watermarks across multiple reviews while controlling the family-wise error rate (FWER), without relying on assumptions about human-written reviews. Our statistical test provides formal guarantees on the probability of false positives across the entire collection of reviews, and achieves a higher statistical power as compared to standard corrections like Bonferroni or Holm-Bonferroni which are sometimes infeasible in our setting. While the high-level idea of using indirect prompt injection has been explored anecdotally, prior approaches lack rigorous statistical testability, as we discuss subsequently. Separately, prior work [ KSG+24 ] has proposed detection techniques based on stylistic repetition (term frequency) and output consistency under re-prompting (review regeneration), but these approaches rely on assumptions about LLM behavior that may not hold when the generated text is paraphrased or partially rewritten. Furthermore, each of the aforementioned methods lack formal statistical guarantees–such as bounds on the family-wise error rate (FWER)–making them less reliable in settings involving the simultaneous evaluation of many reviews. Finally, because these approaches depend on features derived from historical human writing, they are also susceptible to systematic false positives, misclassifying reviewers whose natural writing style resembles that of LLMs. In this work, we address these limitations and investigate previously unexplored methods for implementing this technique. Specifically, we show that the watermarks generated by our framework are statistically testable, with formal bounds on the FWER that do not depend on properties of human-written reviews. We also examine obfuscation strategies beyond basic white-text cues, including font embedding and cryptic prompt injection via adversarial jailbreaking. Although such jailbreaking techniques have been used in the broader LLM safety literature [ ZWC+23 , WHS24 ] to elicit harmful outputs (e.g., instructions for building a bomb), our work is the first to repurpose them for detecting LLM-generated peer reviews. To evaluate the efficacy of our proposed approach, we conducted extensive experiments using several real-world peer review datasets, including papers submitted to the International Conference on Learning Representations (ICLR) 2024, abstracts from the International Congress on Peer Review and Scientific Publication 2022, the PeerRead dataset [ KAD+18 ] , and National Science Foundation (NSF) grant proposals. We tested our framework on a range of widely used LLMs, including OpenAI’s ChatGPT

Read the complete text