Vulnerability of Text-Matching in ML/AI Conference Reviewer Assignments to Collusions

Jhih-Yi (Janet) Hsieh, Aditi Raghunathan, Nihar B. Shah · 2024

Read Vulnerability of Text-Matching in ML/AI Conference Reviewer Assignments to Collusions by Jhih-Yi (Janet) Hsieh, Aditi Raghunathan, Nihar B. Shah (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-06606.txt.

Introduction

Jhih-Yi Hsieh, Aditi Raghunathan, and Nihar Shah present a 2024 paper showing that the text matching step in conference reviewer assignment can be gamed by colluding researchers. Top AI conferences match papers to reviewers partly by how alike the paper reads to a reviewer's past work. Rings of researchers exploit assignments to review each other kindly.

Past defenses mostly guarded the bidding step, trusting text similarity to stay honest. The authors show that trust is misplaced. Even with no bidding at all, colluders can craft their visible record to pull each other's papers toward them. The paper measures the attack and weighs countermeasures.

Catalog record

CREATORJhih-Yi (Janet) Hsieh, Aditi Raghunathan, Nihar B. Shah
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE102 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-06606

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Vulnerability of Text-Matching in ML/AI Conference Reviewer Assignments to Collusions Jhih-Yi (Janet) Hsieh, Aditi Raghunathan, Nihar B. Shah Abstract In the peer review process of top-tier machine learning (ML) and artificial intelligence (AI) conferences, reviewers are assigned to papers through automated methods. These assignment algorithms consider two main factors: (1) reviewers’ expressed interests indicated by their bids for papers, and (2) reviewers’ domain expertise inferred from the similarity between the text of their previously published papers and the submitted manuscripts. A significant challenge these conferences face is the existence of collusion rings, where groups of researchers manipulate the assignment process to review each other’s papers, providing positive evaluations regardless of their actual quality. Most efforts to combat collusion rings have focused on preventing bid manipulation, under the assumption that the text similarity component is secure. In this paper, we demonstrate that even in the absence of bidding, colluding reviewers and authors can exploit the machine learning based text-matching component of reviewer assignment used at top ML/AI venues to get assigned their target paper. We also highlight specific vulnerabilities within this system and offer suggestions to enhance its robustness. 1 Introduction The rapid growth of machine learning and artificial intelligence research has led to major scientific conferences in these fields receiving thousands to tens of thousands of paper submissions and similar numbers of reviewers [ Sha22 ] . To manage this overwhelming volume, the assignment of reviewers to papers has become largely automated. A key component of this automated assignment pertains to text-matched similarity scores between reviewers’ past work and submitted manuscripts, where natural language processing algorithms compute the similarity between the text of each submitted manuscript and the texts of the reviewer’s previously published manuscripts. Models like SPECTER [ CFB+20 ] , which generates embeddings based on textual content, have become widely used across various prestigious venues, including the Conference on Neural Information Processing Systems (NeurIPS), and are a default model on the popular OpenReview conference management platform. 1 1 1 While our work focuses on the peer-review process in ML/AI venues, the continuous evolution and scaling up of other communities suggest that they may soon have to adopt similar processes and face similar challenges. Although automation facilitates efficiency, it also introduces potential vulnerabilities. One major challenge threatening the integrity of the peer review process is the existence of collusion rings – groups of individuals who conspire to manipulate the review system for personal gain [ Lit21 , Vij20a ] . These rings can unfairly influence the acceptance of certain papers by orchestrating favorable reviews from colluding reviewers. As noted in [ Vij20a ] , colluders may attempt to “game the system by exploiting vulnerabilities in the assignment algorithms to have their collaborators review their submissions.” Studies to identify or mitigate collusion have thus far focused on analyzing manipulations in bidding , the part of the reviewer assignment process where reviewers can indicate their interest in reviewing each paper [ WGW+21 , JSFA24 ] . The algorithm in [ WGW+21 ] considers text similarities as ground truth when examining the bids. In practice, program chairs have also focused on bidding when investigating collusion rings [ PC24 ] ; to address concerns over bidding manipulation, venues such as the Conference on Computer Vision and Pattern Recognition (CVPR) and ACL Rolling Review (ARR) have removed the bidding process altogether. This focus on bidding implicitly or explicitly assumes that the text-matching algorithms resistant to manipulation. However, this assumption warrants scrutiny. In this paper, we investigate whether the text-matching algorithms used in automated reviewer assignments are indeed robust against manipulation. Since most papers are assigned to 3–6 reviewers at conferences [ Sha22 ] , we focus our evaluation on whether a pair of colluding author and reviewer can successfully manipulate the text-matching algorithms to give the colluding reviewer one of the top-1, top-3, or top-5 highest text similarity to the colluding paper among all reviewers at a conference. We find that the text matching algorithms are susceptible to attacks that can significantly increase the calculated similarity between a paper and its colluding reviewer. Here are some salient details: • We find that the SPECTER [ CFB+20 ] text similarity matching algorithm, used by top ML/AI conferences for reviewer assignments, can be manipulated by colluding authors and reviewers. In evaluations on NeurIPS 2023 data, our attack successfully increases a colluding reviewer’s similarity from ranked 101 to top-5 for a colluding paper about 92\% of the time. • Considering NeurIPS 2022 as a “past” conference whose data is publicly available to an attacker, and NeurIPS 2023 as the “current” conference whose data is unavailable, we find that the attack performance on NeurIPS 2022 is reflective of the performance in NeurIPS 2023. We find strong correlations of 0.62, 0.83, 0.92 and 0.93 between colluding reviewers’ similarity rankings in NeurIPS 2022 and NeurIPS 2023 in four different settings. • Many conferences allow reviewers to select a subset of papers in their profile. We find that a careful selection of the past papers by the reviewer – and particularly a choice of fewer papers – can significantly increase the attack’s success rate. When a colluding reviewer selects only one paper that is the most similar to the colluding paper, it increases the colluding reviewer’s similarity from ranked 101 to top-5 for the colluding paper 41% of the time even without any modifications to the colluding paper’s abstract. • When reviewers have multiple papers in their profile, similarity-computation algorithms which assign papers via the maximum of its similarities with the reviewer’s past papers are more vulnerable than those which take the mean of the similarities with the reviewer’s past papers. When the mean of similarities is taken, the attack successfully increases a colluding reviewer’s similarity from ranked 101 to top-5 for a colluding paper 32% of the time compared to 49% when the maximum is taken. • We conduct a human-subject experiment to test for the identifiability of adversarial abstracts, in which researchers are asked to evaluate manipulated or control abstracts. For scalability, this experiment uses only automated attacks without the human-in-the-loop interventions, thereby obtaining an upper bound on the detectability (as attackers would otherwise check and iterate if they think the attack is detectable). We find that the rate at which participants complain about coherence or consistency of the abstracts is higher in the attacked abstracts, but there is also a non-trivial rate of complaints about the control abstracts, which in practice can give attackers plausible deniability. In the final section of the paper, we discuss the implications of our findings and propose recommendations to enhance the security and robustness of automated reviewer assignments systems. We have also informed the concerned conference management platform about these results. Based on the findings of this work, the platform has implemented safeguards for conference organizers to guard against such attacks, and the safeguards have been used by a major AI/ML conference. Subsequent peer-review venues have also expressed interest in deploying safeguards, and we are closely helping these venues in doing so. We release our research artifacts including code, datasets, and manipulation examples on Zenodo https://doi.org/10.5281/zenodo.15588237 . This paper aims to make the community aware of this vulnerability, which has implications for the

Read the complete text