Bounds on Agreement between Subjective and Objective Measurements

Jaden Pieper, Stephen D. Voran, Senior Member, IEEE · 2026

Read Bounds on Agreement between Subjective and Objective Measurements by Jaden Pieper, Stephen D. Voran, Senior Member, IEEE (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-13204.txt.

Introduction

Jaden Pieper and Stephen D. Voran ask how well objective quality scores can ever match human ratings. Subjective test scores carry noise, so a perfect Pearson correlation or zero error is not realistic. From basic assumptions, they derive bounds on PCC and MSE set by vote variance. When variance data are missing, other tests or a vote model can supply them. Their binomial-based BinoVotes model leads to BinoMOS, which tracks 18 real tests closely. The bounds let teams set fair goals for any subjective test.

Catalog record

CREATORJaden Pieper, Stephen D. Voran, Senior Member, IEEE
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55637
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.13204

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Bounds on Agreement between Subjective and Objective Measurements Jaden Pieper, Stephen D. Voran, Senior Member, IEEE Abstract Objective estimators of multimedia quality are typically judged by comparing estimates with subjective “truth data,” most often via Pearson correlation coefficient (PCC) or mean-squared error (MSE). But subjective test results contain noise, so striving for a PCC of 1.0 or an MSE of 0.0 is neither realistic nor repeatable. Numerous efforts have been made to acknowledge and appropriately accommodate subjective test noise in objective-subjective comparisons, typically resulting in new analysis frameworks and figures-of-merit. We take a different approach. By making only the most basic assumptions, we are able to derive bounds on PCC and MSE that can be expected for a subjective test. Consistent with intuition, these bounds are functions of subjective vote variance. When a subjective test includes vote variance information, the calculation of the bounds is straight-forward, and in this case we say the resulting bounds are “fully data-driven.” We also provide two options for calculating bounds in cases where vote variance information is not available. One option is to use vote variance information from other subjective tests that do provide such information, and the second option is to use a model for subjective votes. Thus we introduce a binomial-based model for subjective votes (BinoVotes) that naturally leads to a mean opinion score (MOS) model, named BinoMOS, with multiple unique desirable properties. BinoMOS reproduces the discrete nature of MOS values and its dependence on the number of votes per file. This modeling provides vote variance information required by the derived PCC and MSE bounds and we compare this modeling with data from 18 subjective tests. The modeling yields PCC and MSE bounds that agree very well with those found from the data directly. These results allow one to set expectations for the PCC and MSE that might be achieved for any subjective test, even those where vote variance information is not available. I Introduction Attributes of multimedia signals such as quality, noisiness, and others are often measured in subjective tests. In the most common case, subjects watch sets of video files or listen to sets of audio files and rate the overall quality of each file by voting 1 (bad), 2 (poor), 3 (fair), 4 (good), or 5 (excellent). Mean opinion scores (MOS) are created by averaging responses from multiple subjects for each file. These tests rely on the underlying assumption that each file has a true quality that can be estimated by running well-designed subjective tests. Test trials may use a single stimulus or involve comparisons, and may be conducted in labs or performed by crowd-workers. The International Telecommunication Union has defined specific test protocols for the speech [ 14 , 15 , 16 , 17 ] , audio [ 10 ] , video [ 11 , 21 , 20 ] , multimedia [ 19 , 22 ] , and gaming [ 18 ] domains. No matter how well a subjective test is designed, the results are imperfect, as all measurements are. Regardless, these tests are the gold standard for measuring the quality and other attributes of audio, video, or multimedia signals. They are expensive and time-consuming and this has motivated the continual development of objective estimators that can approximate MOS values without querying human subjects. The development and evaluation of these estimators rely on databases that contain media files and associated subjective test results. The performance of an objective estimator is often measured by comparing its estimates to the corresponding subjective test results using mean-squared error (MSE) and Pearson correlation coefficient (PCC). Root mean-squared error (RMSE) is a common alternative to MSE, and we switch between the two here as is convenient. Results from subjective tests are typically considered “truth data,” but they are in fact noisy measurements that have inherent uncertainty. One source of noise in MOS values is the discrete nature of subjective rating scales. Another is the limited number of subjects that rate each signal. Additional noise comes from per-subject biases and general test design considerations such as the distribution of quality present in the test, and the level of control in the test environment. These inherent noises must be acknowledged when developing and assessing the performance of objective estimators. By accounting for the inherent uncertainty in MOS values we contextualize the degree to which an objective estimator can match this “truth data.” We address this need by deriving mathematical bounds for MSE and PCC that are entirely based on the limitations of estimating quality through MOS. These bounds incorporate the number of votes per file and the distribution of qualities within a test. The bounds are applicable regardless of how votes, and hence MOS, are generated, as long as the MOS values converge towards the true quality. These bounds allow one to decide if an objective estimator has room for improvement, or if it is as good as one could expect given the limitations of the subjective tests being used. As one might expect, these bounds require information about the variance of the individual votes that contribute to the MOS value. We call this the vote variance and it drives the “noise” that obscures the “signal.” When vote variance information is available, these bounds let the data “speak for itself,” without any additional assumptions. But vote variance is not always shared (this is the case for some of the large crowd-sourced subjective tests commonly used for training objective speech quality estimators). In this event, a voting model can provide the variance information needed to calculate the bounds. Thus we also offer a simple and interpretable model for subjective votes. This model is based on the binomial distribution so we have named it BinoVotes. Just as real votes are averaged to produce MOS, we average BinoVotes to produce BinoMOS. We show that BinoVotes and BinoMOS have multiple desirable properties that agree with intuition and that BinoVotes comport with real votes in the sense that the MSE and PCC bounds for the two are quite close. Subjective testing has been carefully studied and the popular five-level rating scale and resulting MOS values have been of particular interest. Their scope and limitations are discussed in detail in [ 35 ] which also provides readers with a large number of citations to additional related work. Seeking to get closer to an underlying truth, researchers have proposed multiple alternatives to simply taking the mean of individual votes. One proposal involves setting up a maximum likelihood estimation problem and then solving it using belief propagation [ 26 ] or alternating projections [ 25 ] . Other proposals include leveraging generalized linear models [ 29 ] , an iterative parameter estimation algorithm [ 37 ] , an approach based on Z-scores [ 43 ] , an entropy-based subjective quality recovery algorithm [ 1 ] , and a regularized maximum-likelihood estimation algorithm [ 36 ] . Since the underlying truth is never knowable in practice, arguments for and against these alternatives to the mean are necessarily based on various simulations and on robustness to specific classes of controlled perturbations to the votes. Distributions of votes have previously been modeled with the alpha-stable distribution [ 7 ] , the Gaussian distribution [ 6 ]

Read the complete text