Rejoinder: The ICML 2023 Ranking Experiment: Examining Author Self-Assessment in ML/AI Peer Review
Buxin Su, Jiayao Zhang, Natalie Collina, Yuling Yan, Didong Li, Kyunghyun Cho, Jianqing Fan, Aaron Roth, Weijie Su · 2026
Read Rejoinder: The ICML 2023 Ranking Experiment: Examining Author Self-Assessment in ML/AI Peer Review by Buxin Su, Jiayao Zhang, Natalie Collina, Yuling Yan, Didong Li, Kyunghyun Cho, Jianqing Fan, Aaron Roth, Weijie Su (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-25172.txt.
Introduction
Buxin Su and colleagues answer critics of their study of the ICML 2023 ranking experiment, in which authors ranked their own submissions to probe peer review. Organized as a rejoinder, the reply frames review as a statistical estimation problem. It defends the Isotonic Mechanism against fairness and gaming worries. It also weighs extra signals like rankings supplied by reviewers and structured article data, closing with a human-centered vision for review in the age of generative artificial intelligence.
Catalog record
| CREATOR | Buxin Su, Jiayao Zhang, Natalie Collina, Yuling Yan, Didong Li, Kyunghyun Cho, Jianqing Fan, Aaron Roth, Weijie Su |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30228 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.25172
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Rejoinder: The ICML 2023 Ranking Experiment: Examining Author Self-Assessment in ML/AI Peer Review Buxin Su, Jiayao Zhang, Natalie Collina, Yuling Yan, Didong Li, Kyunghyun Cho, Jianqing Fan, Aaron Roth, Weijie Su Abstract We thank Editor Hongtu Zhu for selecting our article “The ICML 2023 Ranking Experiment: Examining Author Self-Assessment in ML/AI Peer Review” ( Su et al., 2025b ) for discussion, and we are grateful for the insightful feedback and thoughtful comments offered by Professor Yoshua Bengio and Dr. Dinghuai Zhang, Professor Sanmi Koyejo and Dr. Andreas Haupt, Professor Xiao-Li Meng, Professors Weichen Wang and Chengchun Shi, Ying Wang and Professor Mengye Ren, and Professors Linjun Zhang and Lexin Li. We are delighted to see that many discussants endorse our formulation of machine learning (ML) and artificial intelligence (AI) conference peer review as a statistical problem, provide constructive comments on the Isotonic Mechanism, propose additional signals that can be used in conjunction with the method, and articulate a broader agenda for the reform of peer review in the AI era. Our rejoinder is organized into four themes in the sections that follow. 1 Peer Review as an Estimation Problem A somewhat radical question that has begun to circulate in the ML/AI community is whether the field could simply abandon peer review and let the “market” (that is, readers, gauged through GitHub stars and social media engagement) decide. One motivation for this sentiment is that AI progresses at an unprecedented speed, and much of the research conducted in this field, especially that produced by frontier AI labs such as OpenAI, Anthropic, and DeepSeek, is disseminated as research reports that cannot wait for the peer review process. Another is that the number of submissions to leading ML/AI conferences such as ICML and NeurIPS has grown exponentially, in part because AI is now widely used at various stages of research and substantially accelerates research output, while the supply of qualified reviewers cannot keep pace with this explosive growth ( Kim et al., 2025 ) . For example, ICML 2026 received 26,600 valid submissions, exceeding the count at NeurIPS 2025 and thereby becoming, at present, the largest ML/AI conference in history. 1 1 1 NeurIPS 2026 will receive even more submissions; the point we wish to make is that, for the first time in history, the growth in submissions over a span of half a year is sufficient to offset the gap in submission volume between the two conferences. Research papers, however, vary enormously in quality, correctness, novelty, and long-term value, and an average reader cannot reliably evaluate every new submission without some form of filtering. Peer review is, in essence, an attempt to estimate the quality of submitted work, a responsibility that academia has upheld throughout its long history. That need has not changed; what has changed is that the estimation problem has become considerably harder to carry out, as more papers must be reviewed at greater speed. Consequently, we see no compelling reason for the scientific community to abandon formal peer review. For statisticians, viewing peer review as an estimation problem is entirely natural. In ML/AI conferences, each submission typically receives three to five review scores, often on a scale such as 1--8 or 1--10, and the average of these scores is displayed prominently on OpenReview. 2 2 2 Journal review can also be rendered numerical by encoding decisions such as reject, reject with resubmit, major revision, minor revision, and accept as numerical values. The mean review score anchors attention and inevitably shapes downstream decisions. Let \theta_{i} denote the latent quality of submission i , where i ranges over tens of thousands of submissions. The displayed average on OpenReview is therefore a noisy estimator of \theta_{i} . Conferences must accordingly make high-stakes decisions on tens of thousands of submissions within a short turnaround, drawing on noisy scores, written reviews, reviewer confidence ratings, area chair (AC) summaries, and author rebuttals. In this sense, peer review is an estimation problem before it is a decision problem. This estimation viewpoint is a perspective shared by many of the discussants. Professors Wang and Shi formulate review scores as noisy measurements affected by reviewer bias and variance, and propose reviewer rankings as an additional source of information for estimating latent paper quality. Professors Zhang and Li similarly emphasize latent-quality models, pairwise comparisons, Bayesian or likelihood-based calibration, and reviewer matching under uncertainty. Professor Bengio and Dr. Zhang describe the problem as one of finding signal in noise, while Wang and Professor Ren characterize the current reviewer-centric process as a high-variance gatekeeper. Professor Meng situates the issue within the broader context of data science, experimental design, and causal inference. Ultimately, we must rely on statistical studies if we wish to understand deeply how authors, reviewers, editors, and AI tools interact within the peer review system. 2 The Need for Mechanism Design in Estimation The framing of peer review as a statistical problem is not a mere formality. It is the starting point for asking what statisticians can contribute to ML/AI peer review, and it indeed served as the motivation for our Isotonic Mechanism. As Wang and Professor Ren put it, the mechanism can be thought of as an approach to eliciting signals from authors in order to denoise review scores. At a high level, this differs from prior work that has focused exclusively on the reviewer side, in that it shifts attention to a complementary actor in the process, namely the authors. While the intuition that authors possess substantial private knowledge about their submissions is appealing, authors are also stakeholders in the process and may therefore not provide the credible information we seek. This reality calls for mechanism design, the study of how strategic stakeholders can be induced, under appropriately chosen rules, to produce desirable outcomes. Specifically, the question reduces to identifying constraints under which authors will truthfully report their private information, and to determining how this information can be incorporated into a statistical procedure for improved estimation of submission quality. This is, in a nutshell, the conceptual origin of the Isotonic Mechanism developed five years ago ( Su, 2021 ; Su, 2026 ) . In fact, there is a long history of implicit mechanism design within the peer review process ( Aziz et al., 2016 ) . Conflict-of-interest rules and randomization of reviewer assignments ( Xu et al., 2023 ) are among the many constraints designed to prevent reviewer manipulation, ensuring the collection of credible, albeit still noisy, assessments of submission quality. A notable recent example is the insertion of prompt watermarks in ICML 2026 submissions, aimed at restricting reviewers from utilizing AI tools in specific unauthorized scenarios. However, it is important to note that historically, these rules have been almost exclusively imposed on the reviewers. The Isotonic Mechanism should be understood as a continuation of this tradition, but charting a new direction by designing incentive-aligned rules for the authors . While many discussants appreciate the novelty of this mechanism, they also raised several practical considerations regarding its real-world deployment, which we address below. Heterogeneous improvements. A common concern across the discussions is that the improvement delivered by the Isotonic Mechanism may be heterogeneous across authors. Wang and Professor Ren note that, while the mechanism improves review scores