When Less Is More: Binary Feedback Can Outperform Ordinal Comparisons in Ranking Recovery
Shirong Xu, Jingnan Zhang, Junhui Wang · 2025
Read When Less Is More: Binary Feedback Can Outperform Ordinal Comparisons in Ranking Recovery by Shirong Xu, Jingnan Zhang, Junhui Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-01613.txt.
Introduction
Shirong Xu, Jingnan Zhang, and Junhui Wang show that binary feedback can beat ordinal comparisons for ranking recovery in a 2025 statistics paper. Pairwise comparisons drive ranking and preference learning. Ordinal answers seem richer than yes or no answers, so discarding detail looks wasteful. The authors challenge that intuition.
Within a broad parametric model, ordinal data can be treated as a fine grained version of binary data. They prove that binarizing can speed ranking recovery, with an exponentially faster error rate under a counting algorithm. A signal to noise analysis characterizes when the gain is largest. Simulations and a real application support the theory.
Catalog record
| CREATOR | Shirong Xu, Jingnan Zhang, Junhui Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 110 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-01613
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
When Less Is More: Binary Feedback Can Outperform Ordinal Comparisons in Ranking Recovery Shirong Xu, Jingnan Zhang, Junhui Wang Abstract Paired comparison data, where users evaluate items in pairs, play a central role in ranking and preference learning tasks. While ordinal comparison data intuitively offer richer information than binary comparisons, this paper challenges that conventional wisdom. We propose a general parametric framework for modeling ordinal paired comparisons without ties. The model adopts a generalized additive structure, featuring a link function that quantifies the preference difference between two items and a pattern function that governs the distribution over ordinal response levels. This framework encompasses classical binary comparison models as special cases, by treating binary responses as binarized versions of ordinal data. Within this framework, we show that binarizing ordinal data can significantly improve the accuracy of ranking recovery. Specifically, we prove that under the counting algorithm, the ranking error associated with binary comparisons exhibits a faster exponential convergence rate than that of ordinal data. Furthermore, we characterize a substantial performance gap between binary and ordinal data in terms of a signal-to-noise ratio (SNR) determined by the pattern function. We identify the pattern function that minimizes the SNR and maximizes the benefit of binarization. Extensive simulations and a real application on the MovieLens dataset further corroborate our theoretical findings. Keywords: Binarization, Paired Comparison Data, Ranking, Preference Learning, 1 Introduction Paired comparison data arises from evaluating items in pairs and is commonly encountered across various scenarios, including college comparisons ( Caron et al. 2014 ) , product evaluations ( Böckenholt and Dillon 1997 ; Duineveld et al. 2000 ) , sports tournaments ( Buhlmann and Huber 1963 ; Li et al. 2022 ; Jadbabaie et al. 2020 ) , and human feedback in large language models ( Zhu et al. 2023 ; Poddar et al. 2024 ) . Usually, it is commonly assumed that a true parameter vector \bm{\theta}^{\star}=(\theta_{1}^{\star},\ldots,\theta_{n}^{\star}) exists for n items with \theta_{i}^{\star} representing the preference of i -th item. A central problem resolving around paired comparison data is to estimate the ordering of \bm{\theta}^{\star} , and then a complete ranking of n items can be obtained ( Chen et al. 2019 ; Chen et al. 2022a ; Chen et al. 2022b ; Wauthier et al. 2013 ) . Throughout the past century, the literature has seen extensive research dedicated to the development of parametric paired comparison models. One prominent line of work focuses on modeling comparisons as binary outcomes. Specifically, the probability that item i is preferred over item j under a given criterion is modeled as \displaystyle\mathbb{P}\left(i\succ j\right)=F(\theta_{i}^{\star}-\theta_{j}^{\star}), (1) where \succ denotes a ranking relationship under a specific criterion, such as preference. For instance, in the context of large language models, users may be asked to express a binary preference when comparing two textual responses. Similarly, in sports tournaments, two teams compete, and the outcome of the match reflects which team is stronger. The function F(\cdot) can take various forms, such as the logistic function, F(x)=(1+\exp(-x))^{-1} , or the normal cumulative distribution function. These correspond to the Bradley-Terry-Luce (BTL) model ( Bradley and Terry 1952 ) and the Thurstone-Mosteller (TM) model ( Thurstone 1994 ) , respectively. Furthermore, Stern 1990 introduced a model in which binary comparisons arise from the comparison of two gamma-distributed random variables with different scale parameters. This framework includes the BTL and Thurstone-Mosteller models as special cases for different values of the gamma shape parameter. Another line of research aims at modeling non-binary ordinal paired comparisons. This line of research is motivated by scenarios in which items are evaluated with varying degrees of preference. For example, consumers may express a strong preference for one product over another when comparing alternatives. One of the earliest contributions in this area extends the Bradley-Terry-Luce (BTL) model to account for ties in paired comparisons, effectively incorporating three distinct levels of preference ( Glenn and David 1960 ; Rao and Kupper 1967 ; Davidson 1970 ) . Building on this, Agresti 1992 introduces an adjacent-categories logit model that accommodates comparisons with more than three options, while naturally reducing to the BTL model when only two categories are present. Further extensions to continuous paired comparison data are presented in Stern 2011 and Han et al. 2022 . Notably, Han et al. 2022 proposes a general paired comparison framework capable of modeling both continuous and ordinal observations. To clarify the distinction between binary and ordinal comparisons, we present an example in Figure 1 . In this example, the same two users are asked to compare the same pair of items under two differently structured systems. In the first system (Left), users provide binary responses to their comparisons, while in the second system (Right), they offer more detailed, ordinal feedback. In the first case, the preference ordering between the items is ambiguous because the two users give conflicting responses. In contrast, under the second system, item 1 receives more favorable feedback: user 1 expresses a strong preference for item 1, whereas user 2 only slightly prefers item 2. This example illustrates that when a system restricts users to binary choices, valuable information about the strength of preferences may be lost. In other words, the responses in Scenario I are binarized versions of those in Scenario II. Specifically, strongly agree and somewhat disagree are reduced to 1\succ 2 and 2\succ 1 , respectively, resulting in significant information loss. Figure 1 : Two users submit pairwise comparison responses using two distinct sets of system-defined options. In Scenario I (Left), a preference ranking tie occurs. In Scenario II (Right), object 1 receives more favorable feedback. The example in Figure 1 naturally gives rise to the intuition that ordinal comparison data conveys more information about the underlying ground-truth ranking. Consequently, one might expect that using ordinal comparisons would lead to more accurate ranking estimates of items. However, in this paper, we theoretically demonstrate that this intuition does not hold within a broad class of ordinal comparison models. In fact, we show that binarizing ordinal comparison data into binary comparisons can significantly improve the estimation of the ground-truth ranking. To address the question of which type of comparison data is more effective for recovering item rankings, it is crucial to understand the underlying relationship between ordinal and binary comparison data. Specifically, as illustrated in Example 1 , when two different systems elicit responses on the same items from the same group of users, what is the fundamental connection between their binary and ordinal comparison responses? To some extent, binary responses can be viewed as the result of binarizing the corresponding ordinal responses. This raises a natural question: Is there a modeling framework that explicitly captures the binarization process? To address the above questions, we begin by introducing a general class of parametric models designed to capture ordinal outcomes in pairwise comparisons without ties. This focus is motivated by the fact that ties are often absent in many real-world comparison settings, such as sports competitions and consumer preference surveys. The proposed model takes the form of a generalized additive model, consisting of a link function that captures the preference difference between two items and a pattern function that characterizes the distribution over ordinal response levels. The contributions of this