Aspect-Guided Multi-Level Perturbation Analysis of Large Language Models in Automated Peer Review WARNING: This paper contains model outputs that may be considered offensive.
Jiatao Li, Yanheng Li, Xinyu Hu, Mingqi Gao, Xiaojun Wan · 2025
Read Aspect-Guided Multi-Level Perturbation Analysis of Large Language Models in Automated Peer Review WARNING: This paper contains model outputs that may be considered offensive. by Jiatao Li, Yanheng Li, Xinyu Hu, Mingqi Gao, Xiaojun Wan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-12510.txt.
Introduction
Jiatao Li, Yanheng Li, Xinyu Hu, Mingqi Gao, and Xiaojun Wan probed language models used as peer reviewers in two thousand twenty-five. Automated review is tempting, but small changes to a paper, a review, or a rebuttal might sway the outcome. The authors set out to measure that risk.
Their framework makes targeted changes tied to aspects like contribution, soundness, and tone. They found strong reject advice swayed final meta-reviews, harsh reviews could be misread as thorough, and weak or hostile rebuttals could oddly raise acceptance. The biases held under several chain-of-thought prompts. The work gives a practical way to test review systems for such flaws.
Catalog record
| CREATOR | Jiatao Li, Yanheng Li, Xinyu Hu, Mingqi Gao, Xiaojun Wan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 96 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-12510
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Aspect-Guided Multi-Level Perturbation Analysis of Large Language Models in Automated Peer Review WARNING: This paper contains model outputs that may be considered offensive. Jiatao Li, Yanheng Li, Xinyu Hu, Mingqi Gao, Xiaojun Wan Abstract We propose an aspect-guided, multi-level perturbation framework to evaluate the robustness of Large Language Models (LLMs) in automated peer review. Our framework explores perturbations in three key components of the peer review process—papers, reviews, and rebuttals—across several quality aspects, including contribution, soundness, presentation, tone, and completeness. By applying targeted perturbations and examining their effects on both LLM-as-Reviewer and LLM-as-Meta-Reviewer , we investigate how aspect-based manipulations, such as omitting methodological details from papers or altering reviewer conclusions, can introduce significant biases in the review process. We identify several potential vulnerabilities: review conclusions that recommend a “strong reject” may significantly influence meta-reviews, negative or misleading reviews may be wrongly interpreted as thorough, and incomplete or hostile rebuttals can unexpectedly lead to higher acceptance rates. Statistical tests show that these biases persist under various Chain-of-Thought prompting strategies, highlighting the lack of robust critical evaluation in current LLMs. Our framework offers a practical methodology for diagnosing these vulnerabilities, thereby contributing to the development of more reliable and robust automated reviewing systems. 1 Introduction Figure 1: Overview of Our Single-Target Perturbation Approach. We perturb only one component at a time – either the paper, the reviews, or the rebuttal – and then observe the effect of that single change on the LLM acting as either a reviewer or a meta-reviewer. Peer review, the cornerstone of scientific validation, is facing increasing strain due to the rapid growth in manuscript submissions ( Bornmann and Mutz, 2015 ) . The traditional review process is facing challenges in securing sufficient qualified reviewers, and it remains both time-consuming and inherently subjective. ( Aczel et al., 2021 ) . This has led to explorations of automated and semi-automated reviewing pipelines, leveraging recent advances in Large Language Models (LLMs) ( OpenAI et al., 2024 ) . LLMs hold the potential to accelerate peer review by drafting critiques, summarizing papers, and consolidating feedback ( Weissgerber et al., 2021 ) . Early studies suggest some overlap between LLM-assisted reviews and human reviews, and AI conferences already report non-trivial use of AI in the review process ( Liang et al., 2024a ) . This has spurred proposals for fully automated peer review systems ( Tyser et al., 2024 ) . However, concerns about the reliability, fairness, and transparency of LLM-based reviews remain paramount. While existing research has highlighted LLM vulnerabilities like hallucination, bias, and susceptibility to adversarial edits ( Zeng et al., 2024 ; Gallegos et al., 2024 ; Xie et al., 2023 ; Ye et al., 2024 ) , these studies often focus on isolated aspects of the review process. They do not fully capture the complex interplay among the paper , the reviews , and the author rebuttal —the three core components of peer review. Crucially, prior work lacks a unified, multi-level framework to systematically analyze how targeted manipulations of each component propagate through an LLM-based review pipeline, impacting both individual review scores and the overall meta-review decision. To address this critical gap, we propose an aspect-guided, multi-level perturbation analysis for evaluating LLMs in automated peer review. Our framework (Figure 1 ) modifies each component—paper, review, and rebuttal—along key aspects (e.g., contribution, tone). We focus on single-target perturbations: we alter only one of the paper, the reviews, or the rebuttal at a time. This creates a perturbed input alongside the original, unperturbed input. These inputs are fed into either LLM-as-Reviewer (for paper perturbations) or LLM-as-Meta-Reviewer (for any of the three perturbation types). Comparing outputs from these scenarios reveals how single-target manipulations, such as removing key methods or flipping a reviewer’s recommendation, cascade through the system, impacting scores and decisions. Analyzing 508 accepted ICLR 2024 papers, we find that LLM-based reviewing pipelines often fail to discount biased or erroneous signals. Even without altering a paper’s underlying quality, manipulations such as adding false criticisms, flipping a positive recommendation, or withholding key rebuttal details can disproportionately shift system verdicts. These distortions even persist under various Chain-of-Thought prompting strategies. Our findings highlight the need for stronger oversight, adversarial defenses, and transparent review processes before fully automating scholarly peer review. Our contributions are threefold: 1 1 1 Our code and data will be released to the community to facilitate future research. : • We curate a dataset comprising 79,302 adversarial, aspect-guided text edits, which incorporate perturbations across nine distinct aspects affecting reviews, papers, and rebuttals, all guided by our perturbation aspect taxonomy. • Our aspect-guided, multi-level approach effectively integrates both LLM reviewer and meta-reviewer perturbations, employing directional and invariance tests to assess the impact of these perturbations. • Our findings illuminate vulnerabilities within the LLM-based review system, revealing that the “strong reject” conclusion can overshadow all other feedback. Furthermore, hostile or erroneous critiques may be misinterpreted as thoroughness, underscoring the urgent need for enhanced safeguards. 2 Perturbation Design We present our framework for assessing the robustness of LLMs in automated peer review. By introducing targeted perturbations at various components—specifically to the paper, the reviews, and the rebuttal—and across multiple quality dimensions, we are able to evaluate the effects of these manipulations on the behavior of LLMs functioning as both reviewers and meta-reviewers. We first describe the LLM roles within our framework, then formalize our single-target perturbation approach, detail the specific perturbation strategies, and finally describe the statistical tests used for analysis. 2.1 LLM Roles in Peer Review We define two distinct roles for the LLMs in our framework, mirroring the human roles in the peer review process: LLM-as-Reviewer This LLM role, denoted as M_{\mathrm{rev}} , processes only the paper text, p , and outputs dimension-specific scores (e.g., Contribution, Soundness, Presentation) plus a final recommendation: M_{\mathrm{rev}}(p)\to\text{LLM Review} . 2 2 2 In practice, a human reviewer may also consult the rebuttal or engage in follow-up discussions, but some do not consistently do so. For simplicity and experimental tractability, our LLM-as-Reviewer setting only ingests the paper. LLM-as-Meta-Reviewer This LLM role, denoted as M_{\mathrm{meta}} , takes as input the paper, p , multiple reviews, \{r_{i}\} , and the rebuttal, b : M_{\mathrm{meta}}(p,r_{1},\dots,r_{k},b)\to\text{LLM Meta-Review} . It produces an overall score and final decision, reflecting the integrated viewpoints of multiple reviewers and the authors’ responses. 2.2 Single-Target Perturbation To analyze how each component (paper, review, rebuttal) affects LLM-based evaluations, we introduce a single-target perturbation approach. This means that we modify only one of these components at a time. Let T=(p,r_{1},\dots,r_{k},b) denote the original, unperturbed inputs (the paper, a set of reviews, and the rebuttal). We define three perturbation functions: 1) F_{\mathrm{paper}}(T) : Modifies the paper, resulting in (p^{\prime},r_{1},\dots,r_{k},b) . 2) F_{\mathrm{review}}(T) : Modifies the reviews, resulting in (p,r_{1}^{\prime},\dots,r_{k}^{\prime},b) . 3) F_{\mathrm{rebuttal}}(T) : Modifies the rebuttal, resulting in (p,r_{1},\dots,r_{k},b^{\prime}) . Applying any of these functions, F_{\alpha}(T) , produces a perturbed input, T^{\prime} . We then compare the LLM outputs for the perturbed input, o_{\mathrm{pert}} , with the unperturbed baseline outputs, o_{\mathrm{base}} . For paper perturbations, we evaluate both M_{\mathrm{rev}}(p^{\prime}) and M_{\mathrm{meta}}(p^{\prime},r_{1},\dots,r_{k},b) . For review or rebuttal perturbations, we keep the paper, p , constant and evaluate only the meta-reviewer: M_{\mathrm{meta}}(p,r_{1}^{\prime},\dots,r_{k}^{\prime},b) or M_{\mathrm{meta}}(p,r_{1},\dots,r_{k},b^{\prime}) . 2.3 Perturbation Taxonomy Figure 2: Taxonomy of