Are the confidence scores of reviewers consistent with the review content? Evidence from top conference proceedings in AI

Wenqing Wu, Haixu Xi, Chengzhi Zhang* · 2025

Read Are the confidence scores of reviewers consistent with the review content? Evidence from top conference proceedings in AI by Wenqing Wu, Haixu Xi, Chengzhi Zhang* (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-15031.txt.

Introduction

Wenqing Wu, Haixu Xi, and Chengzhi Zhang check whether the confidence scores reviewers give at top AI conferences match what their written reviews actually say. Their 2025 study works with reviews from leading machine learning conferences. Reviewers score how sure they are, and those scores help decide if a paper is trusted, but fine-grained checks of text against score were missing.

The team measured hedging words, sentence patterns, topics covered, and tone at word, sentence, and aspect levels. Scores matched the text well at every level. Higher confidence also tracked with paper rejection, which the authors read as a sign that expert judgment and the review process are working fairly.

Catalog record

CREATORWenqing Wu, Haixu Xi, Chengzhi Zhang*
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-15031

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Are the confidence scores of reviewers consistent with the review content? Evidence from top conference proceedings in AI Wenqing Wu, Haixu Xi, Chengzhi Zhang* Abstract Peer review is a critical process used in academia to assess the quality and validity of research articles. Top-tier conferences in the field of artificial intelligence (e.g. ICLR and ACL et al.) require reviewers to provide confidence scores to ensure the reliability of their review reports. However, existing studies on confidence scores have neglected to measure the consistency between the comment text and the confidence score in a more refined way, which may overlook more detailed details (such as aspects) in the text, leading to incomplete understanding of the results and insufficient objective analysis of the results. In this work, we propose assessing the consistency between the textual content of the review reports and the assigned scores at a fine-grained level, including word, sentence and aspect levels. The data used in this paper is derived from the peer review comments of conferences in the fields of deep learning and natural language processing. We employed deep learning models to detect hedge sentences and their corresponding aspects. Furthermore, we conducted statistical analyses of the length of review reports, frequency of hedge word usage, number of hedge sentences, frequency of aspect mentions, and their associated sentiment to assess the consistency between the textual content and confidence scores. Finally, we performed correlation analysis, significance tests and regression analysis on the data to examine the impact of confidence scores on the outcomes of the papers. The results indicate that textual content of the review reports and their confidence scores have high level of consistency at the word, sentence, and aspect levels. The regression results reveal a negative correlation between confidence scores and paper outcomes, indicating that higher confidence scores given by reviewers were associated with paper rejection. This indicates that current overall assessment of the paper’s content and quality by the experts is reliable, making the transparency and fairness of the peer review process convincing. We release our data and associated codes at https://github.com/njust-winchy/confidence_score. keywords Peer review, Confidence score, Paper decision, Hedge sentences detection, Consistent analysis 1 Introduction Peer review is an essential mechanism for evaluating research work and ensuring research quality Siler et al. 2015 . However, with the significant increase in submissions, the reliability of peer review has faced substantial challenges Batagelj et al. 2017 ; Bornmann and Mutz 2015 ; Huisman and Smits 2017 . Figure 1: An Example of high confidence score but expressed doubts. (https://openreview.net/forum?id=Mos9F9kDwkz) Figure 2: An example of a low confidence score but provided detailed explanations. (https://openreview.net/forum?id=PULSD5qI2N1) In top-tier conferences in the field of artificial intelligence, it is common to require reviewers to provide scores or categories, such as confidence scores and rating scores, to indicate the reliability of their review reports. Ensuring the consistency between confidence scores and textual content is crucial for maintaining the reliability and fairness of the peer-review process. For examples 1 1 1 https://openreview.net/forum?id=Mos9F9kDwkz (AnonReviewer3), https://openreview.net/forum?id=PULSD5qI2N1 (AnonReviewer1) , as shown in Figure 1 and Figure 2 , a reviewer gave a relatively high confidence score (4 or 5) to the reviewed paper, despite expressing doubts and reservations in their review report. On the other hand, reviewer gave a relatively low confidence score (1 or 2) but provided detailed explanations of the paper’s issues and shortcomings, along with constructive feedback and suggestions. Therefore, it is an important problem arises regarding the alignment between the content of review reports provided by reviewers and their corresponding confidence scores. There have been studies Ghosal et al. 2022 ; Bharti et al. 2022 that measure reviewer confidence by identifying hedge cues in the review texts or directly predicting confidence scores based on the textual content. However, they have not considered the impact of more fine-grained information (such as aspects) on reviewer confidence. This may lead to incomplete understanding of the results and insufficient objective analysis of the results. Therefore, our objective is to study the consistency between confidence scores and review text by conducting more fine-grained analysis, in order to obtain more comprehensive and objective results. Additionally, we also investigate the correlation between aspects and confidence scores, as well as the impact of confidence scores on the fate of the paper. Through the analysis results above, we can validate the transparency, fairness, and credibility of the peer-review process. Simultaneously, this enables the assessment of the efficiency and fairness of peer review, contributing to the maintenance of the quality of academic research and the advancement of scholarly progress. To this end, we propose to study the consistency between confidence scores and review report text from multiple dimensions, including the sentence-level, word-level, and aspect-level. The data used in this study is derived from the peer review comments of conferences in the fields of deep learning and natural language processing. With the circumstances of open science Zhang et al. 2021 and the openness of the OpenReview 2 2 2 https://openreview.net/ platform Tran et al. 2020 , a large number of unlabeled peer review reports for accepted and rejected papers can be easily accessed. For the word level, we utilized the usage of hedges in the peer review reports to study the consistency between the confidence scores and the reviewers’ confidence. The term ”hedge” is introduced by Lakoff Lakoff 1970 , who define it as introduce ambiguity or reduce certainty in a given context. Previous studies Demir 2018 ; Hyland 1998 ; Pei and Jurgens 2021 ; Szarvas et al. 2008 ; Ghosal et al. 2022 have indicated a correlation between the use of hedges and the expression of uncertainty and confidence. Ghosal et al. Ghosal et al. 2022 introduced a for uncertainty detection from peer-review texts. Other studies propose hedge is an important meta discourse device that should be used for various motives particularly in academic writing because it acts as a face-saving strategy and represents the certainty of the scientist’s knowledge on the study field, but nevertheless it is not an obvious consideration for many non-native writers of English. We refer to the sentences associated with confidence as ”hedge sentences”. Contrary to the word level, a sentence that includes hedge words does not always qualify as a hedge sentence. For example, consider the sentence ”Although the results were consistent and significant, we can confidently conclude that there is a strong relationship between the variables.” While it contains the hedge word ”can”, it is not a hedge sentence. So, we trained a deep learning model to predict hedge sentences and leveraged hedge cues to enhance the prediction process. Furthermore, we utilize a pre-trained aspect labeling model by Yuan et al. Yuan et al. 2022 , trained on peer review-related data, to perform aspect annotation. Our paper offers the following three contributions. Firstly, we collected a large amount of data containing confidence scores and attempted to process the data through automatic recognition by identifying contradictory statements and their aspects in the comment texts. The accuracy of identifying hedge sentences can reach 0.88. And, we trained a deep learning model to identify hedge sentences and improved the issue of inaccurate predictions by leveraging aspect information. Secondly, this paper investigates and analyzes the consistency between confidence scores and review

Read the complete text