UniCBE: An Uniformity-driven Comparing Based Evaluation Framework with Unified Multi-Objective Optimization

Peiwen Yuan, Shaoxiong Feng, Yiwei Li, Xinglin Wang, Yueqi Zhang, Jiayi Shi, Chuyi Tan, Boyuan Pan, Yao Hu, Kan Li · 2025

Read UniCBE: An Uniformity-driven Comparing Based Evaluation Framework with Unified Multi-Objective Optimization by Peiwen Yuan, Shaoxiong Feng, Yiwei Li, Xinglin Wang, Yueqi Zhang, Jiayi Shi, Chuyi Tan, Boyuan Pan, Yao Hu, Kan Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-11454.txt.

Introduction

Peiwen Yuan and co-authors cut in a 2025 study the cost of ranking language models by human preference, head-to-head comparisons being dear and sparse. Their framework, UniCBE, attacks three wastes at once: sampling bias, lopsided uncertainty shrinkage, and noisy updates. Three sampling matrices, each enforcing evenness in one aspect, are merged into one schedule.

On a standard preference benchmark it saved over seventeen percent of the budget while matching human verdicts almost exactly. When new models keep arriving, savings passed half. Even sampling, they show, extracts far more signal from each expensive human judgment.

Catalog record

CREATORPeiwen Yuan, Shaoxiong Feng, Yiwei Li, Xinglin Wang, Yueqi Zhang, Jiayi Shi, Chuyi Tan, Boyuan Pan, Yao Hu, Kan Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-11454

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

UniCBE: An Uniformity-driven Comparing Based Evaluation Framework with Unified Multi-Objective Optimization Peiwen Yuan, Shaoxiong Feng, Yiwei Li, Xinglin Wang, Yueqi Zhang, Jiayi Shi, Chuyi Tan, Boyuan Pan, Yao Hu, Kan Li Abstract Human preference plays a significant role in measuring large language models and guiding them to align with human values. Unfortunately, current comparing-based evaluation (CBE) methods typically focus on a single optimization objective, failing to effectively utilize scarce yet valuable preference signals. To address this, we delve into key factors that can enhance the accuracy, convergence, and scalability of CBE: suppressing sampling bias, balancing descending process of uncertainty, and mitigating updating uncertainty. Following the derived guidelines, we propose UniCBE , a unified uniformity-driven CBE framework which simultaneously optimize these core objectives by constructing and integrating three decoupled sampling probability matrices, each designed to ensure uniformity in specific aspects. We further ablate the optimal tuple sampling and preference aggregation strategies to achieve efficient CBE. On the AlpacaEval benchmark, UniCBE saves over 17% of evaluation budgets while achieving a Pearson correlation with ground truth exceeding 0.995, demonstrating excellent accuracy and convergence. In scenarios where new models are continuously introduced, UniCBE can even save over 50% of evaluation costs, highlighting its improved scalability. 1 Introduction The ongoing evolution of large language models (LLMs) has made it increasingly important to assess their alignment with human preferences ( Dubois et al. 2024 ; Zheng et al. 2023 ) . The preference signals provided by humans are crucial for accurately assessing and guiding models toward safe and reliable AGI ( Ji et al. 2023 ; Jiang et al. 2024 ) . However, the rapid iteration of LLMs in training and application scenarios has created a substantial demand for evaluation, complicating the acquisition of sufficient labor-intensive human preferences ( Chiang et al. 2024 ; Cui et al. 2023 ) . Therefore, exploring the use of precious preference signals for efficient model alignment evaluation is of great significance and requires long-term research. Figure 1: Flowchart of the process for comparing-based evaluation. Current mainstream model evaluation paradigms include scoring-based evaluation (SBE) ( Liu et al. 2023 ; Cai et al. 2023 ) and comparing-based evaluation (CBE) ( Chiang et al. 2024 ; Dubois et al. 2024 ) . The former requires the judge to offer preference scores for individual responses, while the latter needs the judge to establish a preference order among multiple candidate model responses. By directly comparing the responses of different models, Zheng et al. 2023 ; Liu et al. 2024 confirm that CBE can more accurately assess model performance. However, the O(NM^{2}) evaluation overhead limits the practicality of CBE when there are M models to evaluate on N samples ( Qin et al. 2024 ) . To achieve efficient CBE, various methods have been explored Chiang et al. 2024 ; Zhou et al. 2024 ; Dubois et al. 2024 . As shown in Figure 1 , based on existing observational results, these methods iteratively allocate preference budget to the next (models, sample) tuple according to respective optimization objectives. Specific preference aggregation methods (e.g., ELO rating ( Elo & Sloan 1978 ) ) are then applied to predict the model capability scores based on these preference results. Nevertheless, as shown in Table 1 , the optimization objectives of these methods are often singular, failing to simultaneously achieve the accuracy, convergence, and scalability well. We will discuss this in detail in § 2 and conduct experimental validation in § 5.2 . Table 1: Optimization Objectives of widely applied CBE Methods. The number of ’+’ indicates the degree of optimization for the objective, which is discussed in § 2 and measured in Table 2 . Methods Qin et al. 2024 Chiang et al. 2024 Dubois et al. 2024 Ours Random Arena AlpacaEval UniCBE Accuracy + - - ++ Convergence - + - ++ Scalability - - ++ ++ To develop a method that can accurately assess model performance, quickly converge evaluation results, and ensure good scalability when new models are introduced, we theoretically analyze and summarize the following guidelines: • Improving the accuracy of evaluation results relies on completely uniform sampling of tuple combinations, so as to mitigate sampling bias. • Accelerating the convergence process involves ensuring the uniformity of the win rate uncertainty matrix during its descending process to reduce observation variance. • Enhancing scalability requires sufficient budgets being allocated to new added models to ensure the uniform allocation among models, which helps reduce updating uncertainty. Based on these insights, we propose UniCBE , a unified uniformity-driven framework that can achieve CBE with better accuracy, convergence and scalability. In each iteration of the evaluation process, we first establish sampling probability matrices under different optimization objectives respectively based on real-time preference results. Afterwards, we integrate these matrices to obtain a global sampling probability matrix. Furthermore, we explore various tuple sampling strategies and preference aggregation methods to achieve optimal evaluation results. To comprehensively validate the effectiveness and generalizability of UniCBE , we conduct multiple experiments involving various types of judges (LLMs and humans), different benchmarks, varied model sets to be evaluated, diverse scenarios (static and dynamic), and multiple evaluation metrics. The main results indicate that, compared to the random sampling baseline, UniCBE saves over 17% of evaluation budgets when achieving the same assessment accuracy (with a Pearson coefficient exceeding 0.995 with the ground truth), demonstrating significantly better convergence and accuracy than other baselines. Furthermore, in scenarios where new models are continuously introduced, UniCBE can even save over 50% of evaluation costs compared to random sampling, showcasing excellent scalability. 2 Related Work Comparative preference signals have long been used for model training ( Ouyang et al. 2022 ; Touvron et al. 2023 ) and evaluation ( Chiang et al. 2024 ; Yuan et al. 2024 ) . Centered around comparing-based evaluation, we will discuss existing budget allocation strategies and preference aggregation methods below. Budget Allocation Many efforts have been made to explore preference budget allocation approaches. The most naive allocation method is to randomly select (models, sample) tuple for judging each time until the preset preference budget is reached ( Qin et al. 2024 ) . This method ensures a relatively uniform sampling across different tuple combinations in expectation, thereby guaranteeing the accuracy of evaluation results according to our derivation in § 3.2 . Arena ( Chiang et al. 2024 ) aims to sample model pairs proportionally to the variance gradient of win rate at each step, seeking to accelerating the convergence of evaluation by reducing the uncertainty of the observed win rate matrix in a greedy manner. AlpacaEval ( Dubois et al. 2024 ) measures model performance by comparing the models under evaluation with a fixed reference model. Different models under evaluation are expected to be assessed on the same set. Thus, when new models are introduced, preference budget is prioritized for them to stabilize the estimation of their capabilities, thereby achieving good scalability. Despite these methods performing well in their intended objectives, they cannot achieve a balance among accuracy, convergence, and scalability. This makes it imperative to explore better preference budget allocation strategy that can effectively reconcile all these attributes. Preference Aggregation Due to the possibility that the same group of models may exhibit different ranking relationships across

Read the complete text