From Isolated Scoring to Collaborative Ranking: A Comparison-Native Framework for LLM-Based Paper Evaluation
Pujun Zheng, Jiacheng Yao, Jinquan Zheng, Chenyang Gu, Guoxiu He, Jiawei Liu, Yong Huang, Tianrui Guo, Wei Lu · 2026
Read From Isolated Scoring to Collaborative Ranking: A Comparison-Native Framework for LLM-Based Paper Evaluation by Pujun Zheng, Jiacheng Yao, Jinquan Zheng, Chenyang Gu, Guoxiu He, Jiawei Liu, Yong Huang, Tianrui Guo, Wei Lu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-17588.txt.
Introduction
Zheng and co-authors move language-model paper review from solo scores to paired comparisons. Fixed score scales shift across venues and years, so models learn narrow rules instead. Their CNPE framework builds training pairs with a graph-based sampler. Fine-tuning and reinforcement learning then sharpen relative judgment. At test time, pairwise votes merge into one global ranking. Scores rose 21.8 percent over a strong baseline, they report.
Catalog record
| CREATOR | Pujun Zheng, Jiacheng Yao, Jinquan Zheng, Chenyang Gu, Guoxiu He, Jiawei Liu, Yong Huang, Tianrui Guo, Wei Lu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 75363 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.17588
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
From Isolated Scoring to Collaborative Ranking: A Comparison-Native Framework for LLM-Based Paper Evaluation Pujun Zheng, Jiacheng Yao, Jinquan Zheng, Chenyang Gu, Guoxiu He, Jiawei Liu, Yong Huang, Tianrui Guo, Wei Lu Abstract Large language models (LLMs) are currently applied to scientific paper evaluation by assigning an absolute score to each paper independently. However, since score scales vary across conferences, time periods, and evaluation criteria, models trained on absolute scores are prone to fitting narrow, context - specific rules rather than developing robust scholarly judgment. To overcome this limitation, we propose shifting paper evaluation from isolated scoring to collaborative ranking. In particular, we design C omparison- N ative framework for P aper E valuation ( CNPE ), integrating comparison into both data construction and model learning. We first propose a graph-based similarity ranking algorithm to facilitate the sampling of more informative and discriminative paper pairs from a collection. We then enhance relative quality judgment through supervised fine-tuning and reinforcement learning with comparison-based rewards. At inference, the model performs pairwise comparisons over sampled paper pairs and aggregates these preference signals into a global relative quality ranking. Experimental results demonstrate that our framework achieves an average relative improvement of 21.8% over the strong baseline DeepReview-14B, while exhibiting robust generalization to five previously unseen datasets. Code . 1 Introduction Figure 1: Comparison of current methods (top) and our approach (bottom) across data (left), training (middle), and inference (right). Paper evaluation plays a central role in advancing scientific progress Margolis 1967 . Peer review has long served as the primary mechanism for ensuring publication quality and rigor ( Alberts et al. 2008 ) . However, the rapid growth of submissions across disciplines has placed increasing pressure on the peer review system He et al. 2023 , exacerbating issues such as inconsistent evaluations and inherent biases Xue et al. 2023 ; Lee et al. 2013 ; Tomkins et al. 2017 . In recent years, large language models (LLMs) Naveed et al. 2023 have been increasingly applied to paper evaluation, enabling more consistent and scalable review processes ( Du et al. 2024 ; Zhou et al. 2024 ; Zhuang et al. 2025 ; Thakkar et al. 2025 ) . Nevertheless, their growing adoption and societal impact do not eliminate the significant limitations of LLMs ( Latona et al. 2024 ; Liang et al. 2024 ; Zhu et al. 2025a ) . Currently, LLM-based paper evaluation methods primarily assign scores to individual papers (as illustrated at the top of Figure 1 ). As closed knowledge systems, LLMs have inherent limitations in performing the complex reasoning and judgment required for cutting-edge scientific innovation ( Liu and Shah 2023 ; Purkayastha et al. 2025 ) , making them prone to hallucinations and retrieval biases ( Li et al. 2025 ) . Multi-agent frameworks that simulate human peer review workflows cannot fundamentally resolve these issues and may inherit biases present in manual review, such as favoring predictable or mediocre results ( Zhang et al. 2025 ) . Even when models are trained to align with human-assigned absolute scores ( Lu et al. 2024 ; Yu et al. 2024 ; Weng et al. 2025 ; Zhu et al. 2025b ) , performance is constrained by dataset-specific factors: score scales vary across conferences, time periods, and evaluation criteria, causing models to learn context-specific rules rather than generalizable scholarly judgment. Although several recent studies use pairwise or listwise comparisons ( Zhang et al. 2025 ; Zhao et al. 2025 ) , they either do not train model parameters for comparison tasks or still output absolute scores, limiting both performance and generalization Höpner et al. 2025 . Consequently, existing methods are impractical and struggle to produce reliable and consistent rankings when evaluating a set of papers, such as a batch of conference submissions. To this end, we propose a comparison-native framework that uses collaborative data to train the LLM to evaluate and rank papers (as illustrated at the bottom of Figure 1 ). Unlike methods that assign absolute scores to individual papers, our framework reframes the complex task of quantitative reasoning as a comparison problem, better leveraging the reasoning strengths of LLMs Wei et al. 2022 . By learning more reliable preference signals, the model can more clearly capture differences across novelty, significance, and clarity. Furthermore, the comparative logic acquired through this approach is more transferable than that of single-paper scoring, enabling generalization to new paper collections without adapting to specific scoring scales. Specifically, we design two key modules for effective comparison: pair sampling and quality judgment. At the data level, pair sampling selects the most informative paper pairs from all possible combinations to ensure diverse contextual coverage, enhancing comparison accuracy and model generalization. We facilitate this using a graph-based similarity ranking algorithm that prioritizes papers with overlapping research areas. At the model level, quality judgment assesses candidate papers along dimensions such as novelty, significance, and clarity, reliably identifying superior papers and handling fine-grained distinctions. To strengthen judgment capabilities, we construct a dataset of paper pairs annotated with quality preferences and optimize the model via supervised fine-tuning combined with reinforcement learning using verifiable comparison-based rewards. At inference, the model performs pairwise comparisons over sampled paper pairs and aggregates the resulting preference signals into an interpretable global ranking of relative paper quality. Our framework achieves leading performance on the ICLR-2025 dataset for both paper quality ranking and acceptance prediction across all evaluation metrics. With only 7B parameters, it achieves an average relative improvement of 21.8% over the strong baseline DeepReview-14B ( Zhu et al. 2025b ) . Ablation studies validate the effectiveness of our training strategy based on supervised fine-tuning and reinforcement learning, and reveal that jointly applying similarity-based and random sampling enhances performance. The model exhibits strong generalization to five unseen submission datasets from top-tier conferences in 2025, including ICML, NeurIPS, ACL, EMNLP, and NAACL. Our main contributions are as follows: \bullet We propose an LLM-based comparison-native framework that implements paper ranking through data construction and model training. \bullet We propose a graph-based semantic similarity recognition method for paper pair sampling and a comparison-based reward to enhance quality judgment through reinforcement learning. \bullet Our framework achieves leading performance in ranking and acceptance prediction, with robust generalization to previously unseen datasets. 2 Related Work Agent - based Assessment Systems These systems typically combine intelligent agents with general-purpose LLMs to generate review outcomes. Jin et al. 2024 employed an LLM-powered peer-review simulation framework to replicate the review process and identify potential influencing factors. Lu et al. 2024 designed automated review systems to simulate peer evaluation in paper scoring tasks. Other notable efforts include customizing feedback through novel alignment mechanisms with iterative optimization Garg et al. 2025 , implementing tree-structured workflows to enhance performance Chang et al. 2025 , and creating multi-agent frameworks that merge prompt engineering, collaboration, shared memory, and multimodal perception to deliver high-quality reviews Lu et al. 2025 . However, these methods generally rely on LLMs that have not been specifically trained for academic review, which limits their reliability, while the reliance on general-purpose models increases the risk of data leakage. Training-based Review Models These approaches transform open-source LLMs into expert reviewers through domain-specific fine-tuning or reinforcement learning. Yu et al. 2024 developed a domain-specific