K-Sort Eval: Efficient Preference Evaluation for Visual Generation via Corrected VLM-as-a-Judge

Zhikai Li, Jiatong Li, Xuewen Liu, Wangbo Zhao, Pan Du, Kaicheng Zhou, Qingyi Gu, Yang You, Zhen Dong, Kurt Keutzer · 2026

Read K-Sort Eval: Efficient Preference Evaluation for Visual Generation via Corrected VLM-as-a-Judge by Zhikai Li, Jiatong Li, Xuewen Liu, Wangbo Zhao, Pan Du, Kaicheng Zhou, Qingyi Gu, Yang You, Zhen Dong, Kurt Keutzer (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-09411.txt.

Introduction

Visual generative models need evaluation that scales with them, but crowdsourced arena voting is slow and costly. Vision-language models offer a cheaper stand-in for human judges, yet their hallucinations and biases misalign them with human taste. Zhikai Li and nine co-authors propose K-Sort Eval, a 2026 framework that corrects and economizes model-as-judge evaluation for visual generation.

The framework curates a dataset from thousands of human votes, each instance ranking several models' outputs. A new model faces free-for-all comparisons against the field, with the vision-language model supplying rankings. Posterior correction adjusts the Bayesian update whenever model predictions disagree with human supervision. Dynamic matching then picks each comparison to balance uncertainty and diversity. Trials match arena results while typically needing under ninety model runs.

Catalog record

CREATORZhikai Li, Jiatong Li, Xuewen Liu, Wangbo Zhao, Pan Du, Kaicheng Zhou, Qingyi Gu, Yang You, Zhen Dong, Kurt Keutzer
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-09411

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

K-Sort Eval: Efficient Preference Evaluation for Visual Generation via Corrected VLM-as-a-Judge Zhikai Li, Jiatong Li, Xuewen Liu, Wangbo Zhao, Pan Du, Kaicheng Zhou, Qingyi Gu, Yang You, Zhen Dong, Kurt Keutzer Abstract The rapid development of visual generative models raises the need for more scalable and human-aligned evaluation methods. While the crowdsourced Arena platforms offer human preference assessments by collecting human votes, they are costly and time-consuming, inherently limiting their scalability. Leveraging vision-language model (VLMs) as substitutes for manual judgments presents a promising solution. However, the inherent hallucinations and biases of VLMs hinder alignment with human preferences, thus compromising evaluation reliability. Additionally, the static evaluation approach lead to low efficiency. In this paper, we propose K-Sort Eval, a reliable and efficient VLM-based evaluation framework that integrates posterior correction and dynamic matching. Specifically, we curate a high-quality dataset from thousands of human votes in K-Sort Arena, with each instance containing the outputs and rankings of K K models. When evaluating a new model, it undergoes ( K K +1)-wise free-for-all comparisons with existing models, and the VLM provide the rankings. To enhance alignment and reliability, we propose a posterior correction method, which adaptively corrects the posterior probability in Bayesian updating based on the consistency between the VLM prediction and human supervision. Moreover, we propose a dynamic matching strategy, which balances uncertainty and diversity to maximize the expected benefit of each comparison, thus ensuring more efficient evaluation. Extensive experiments show that K-Sort Eval delivers evaluation results consistent with K-Sort Arena, typically requiring fewer than 90 model runs, demonstrating both its efficiency and reliability. The dataset and code are publicly available. 1 Introduction Visual generative models have achieved remarkable progress, enabling high-quality outputs in tasks such as text-to-image ( 5 ; 40 ; 43 ) and text-to-video ( 11 ; 14 ; 54 ) generation. This rapid advancement has fueled growing interest in the field, driving the continuous emergence of new models ( 51 ; 52 ; 50 ) and applications ( 10 ; 29 ; 30 ) . However, the evaluation methods fail to keep pace with the model development, struggling to offer a fair and comprehensive assessment of generated outputs. Traditional metrics such as IS ( 44 ) , FID ( 15 ) , and FVD ( 47 ) , while widely used, are criticized for their inability to capture human preference judgements in the real world. In response, several efforts attempt to construct static datasets for human preference evaluation ( 22 ; 48 ; 49 ) . However, these datasets are inherently limited by their closed-ended nature, lack of user interaction, and inability to stay up-to-date ( 32 ; 7 ) . In contrast, the Arena method, which is an open and live benchmark platform, is a more effective approach for human preference evaluation. It captures real human feedback by collecting crowdsourced manual voting on model comparisons, allowing for a better reflection of real-world preferences. This approach is initially used for evaluating large language models (LLMs) ( 7 ) and is later extended to apply to visual generative models ( 21 ; 32 ) and multi-modal models ( 37 ; 8 ) . Notably, for visual generative models, K-Sort Arena ( 32 ) significantly improves the efficiency and reliability of Arena evaluation by incorporating an improved comparison mode, probabilistic modeling and updating, and an optimized model matching strategy. Nevertheless, due to its heavy reliance on human voting, it still faces significant cost and time-consuming challenges. The excessive human involvement can also lead to potential leaderboard illusion issues ( 45 ) , and the potential delays in crowdsourcing may hinder the timely evaluation of new models, thus limiting its scalability. Figure 1: Overview of the proposed K-Sort Eval. First, a high-quality dataset is curated through consistency filtering. When evaluating a new model, we begin with dynamic matching to select the most informative instance. Then, two prompt strategies are employed to effectively guide the VLM and mitigate hallucinations. Finally, Bayesian updating with correction is performed, where the discrepancy between VLM prediction and dataset supervision is treated as observation noise to correct the posterior estimation of model capability. Therefore, leveraging powerful vision-language models (VLMs) to replace manual judgements, known as VLM-as-a-Judge ( 6 ; 35 ) , presents a promising solution. For instance, T2I-CompBench ( 17 ) investigates the potential of VLMs for compositionality evaluation of text-to-image models, and VIEScore ( 24 ) demonstrates that VLMs can provide results that have a certain relevance to human evaluations. However, VLMs are inherently prone to hallucinations, inconsistencies, and biases, raising concerns about their reliability as a trustworthy substitute for human judgements ( 26 ; 12 ) . While certain techniques from LLM-as-a-Judge, such as swapping operation ( 53 ) , rule augmentation ( 2 ) , and multi-agent collaboration ( 28 ) strategies, can be adapted, they fail to addressing these inherent issues, thus still hindering reliable alignment with human preferences. In addition, existing methods follow the static evaluation style, which necessitates processing the entire large-scale dataset, making the efficiency fall short of expectations. In this paper, we propose K-Sort Eval, built on top of K-Sort Arena ( 32 ) , to enable efficient and reliable visual preference evaluation via VLM-as-a-Judge. Specifically, we curate a high-quality dataset from thousands of human votes in K-Sort Arena, where each instance consists of the outputs of K models along with their rankings. First, we use Spearman’s rank correlation coefficient ( 46 ) to align local rankings within each instance with the overall leaderboard, filtering out contaminated votes that significantly deviate from typical preference patterns. Then, Llama Guard ( 19 ) is applied to screen out potentially harmful or offensive user prompts, ultimately resulting in a widely applicable and representative dataset. With the above dataset, the model to be evaluated can form a ( K +1)-wise free-for-all comparisons with the K models in each instance. Notably, we propose optimized algorithms to ensure the reliability and efficiency of the evaluation. ❶ For reliability , we propose a posterior correction method, which utilizes the human preference outcomes in the dataset as supervision to correct the update of model capability. Following K-Sort Arena, we employ probabilistic modeling to represent model capabilities, and use VLM judgements as observations to update the posterior probability via Bayesian inference. Here, we treat the misalignment between VLM results and supervision as observation noise, and thus derive an adaptive correction policy for the posterior probability. ❷ For efficiency , we propose a dynamic matching strategy, which leverages both uncertainty and diversity to promote the comparison of maximum expected gains. It avoids worthless comparisons, enabling the evaluation to be accomplished using only a subset of the dataset, without traversing the entire dataset. The overview of K-Sort Eval is illustrated in Figure 1 . Table 1 compares K-Sort Eval with existing evaluation methods across various categories, highlighting its advantages in scalability, alignment, efficiency, and generalizability. Furthermore, we conduct extensive experiments to validate the effectiveness of K-Sort Eval, and the results show that it achieves results consistent with K-Sort Arena, while requiring fewer than 90 model runs in most cases, demonstrating its strong potential for reliable preference evaluation of generative models. Table 1: Comparison with existing evaluation methods. The proposed K-Sort

Read the complete text