Are LLM Evaluators Really Narcissists? Sanity Checking Self-Preference Evaluations

Dani Roytburg Carnegie Mellon UniversityMatthew Bozoukov University of California, San DiegoMatthew Nguyen University of VirginiaJou BarzdukasUniversity of VirginiaMackenzie Puig-HallApart ResearchNarmeen OozeerMartian Research · 2026

Read Are LLM Evaluators Really Narcissists? Sanity Checking Self-Preference Evaluations by Dani Roytburg Carnegie Mellon UniversityMatthew Bozoukov University of California, San DiegoMatthew Nguyen University of VirginiaJou BarzdukasUniversity of VirginiaMackenzie Puig-HallApart ResearchNarmeen OozeerMartian Research (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-22548.txt.

Introduction

Dani Roytburg, Matthew Bozoukov, and co-authors sanity-check the claim that model judges favor their own work. Past studies found that models, acting as judges, rate their own outputs higher. Their 2026 preprint notes that this self-love is hard to split from plain test confounds.

A skewed test can fake narcissism where none exists. The team sort which biases the vanity claim truly explains. Clean measures matter: judge bias can warp training and leaderboards at scale. The work is a caution and a cleaner yardstick. It asks the field to measure twice.

Catalog record

CREATORDani Roytburg Carnegie Mellon UniversityMatthew Bozoukov University of California, San DiegoMatthew Nguyen University of VirginiaJou BarzdukasUniversity of VirginiaMackenzie Puig-HallApart ResearchNarmeen OozeerMartian Research
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-22548

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Are LLM Evaluators Really Narcissists? Sanity Checking Self-Preference Evaluations Dani Roytburg Carnegie Mellon UniversityMatthew Bozoukov University of California, San DiegoMatthew Nguyen University of VirginiaJou BarzdukasUniversity of VirginiaMackenzie Puig-HallApart ResearchNarmeen OozeerMartian Research Abstract Recent research has shown that large language models (LLMs) favor their own outputs when acting as judges, undermining the integrity of automated post-training and evaluation workflows. However, it is difficult to disentangle which evaluation biases are explained by narcissism versus general experimental confounds, distorting measurements of self-preference bias. We discover a core methodological confound which could reduce measurement error by 89.6% . Specifically, LLM evaluators may deliver self-preferring verdicts when the judge responds to queries which they completed incorrectly themselves; this would be true regardless of whether one of their responses is their own. To decouple self-preference signals from noisy outputs on hard problems, we introduce an Evaluator Quality Baseline , which compares the probability that a judge incorrectly votes for itself against the probability that it votes for an incorrect response from another model. Evaluating this simple baseline on 37,448 queries, only 51% of initial findings retain statistical significance. Finally, we turn towards characterizing the entropy of “easy” versus “hard” evaluation votes from LLM judges. Our corrective baseline enables future research on self-preference by eliminating noisy data from potential solutions. More widely, this work contributes to the growing body of work on cataloging and isolating judge-bias effects. Figure 1 : (top) Self-preference biases can be decomposed into examples a judge completed correctly (LSP) and incorrectly (ILSP). (bottom) Since bias comes from false positives on incorrect examples, bias measurements should be made by taking the difference against a baseline to measure uncertainty artifacts. Often, differences between a judge evaluating itself and other models disappear against the baseline. 1 Introduction As language models see increasing deployment as evaluators, raters, and proxies for human preference, researchers have observed consistent patterns of unreliability ( 13 ; 21 ) . These patterns are frequently categorized as biases ( 9 ; 40 ) and are often interpreted through anthropomorphic tropes or linked to emerging theory-of-mind frameworks. One notable example comes with the recent fascination with self-preference bias , or narcissism , of LLM evaluators ( 30 ; 26 ) . A suite of recent experimental findings claim that when language models are used to compare responses to a query, the distribution of votes will skew towards outputs generated by that same model. If true, this behavior would pose critical implications for safety and alignment research; this form of situational awareness ( 19 ) risks contaminating post-training, guardrail construction, and monitoring efforts that bootstrap human preferences using LLM evaluators. However, self-preference signals remain confounded by plausible baselines. For instance, early tests did not compare self-preference rates against ground truth accuracy, potentially mistaking a model’s accurate assessment of its own superior performance for illegitimate bias. Recent efforts now control for this by using oracle labels ( 7 ; 39 ) , but still fail to disentangle the relationship between task accuracy and evaluation accuracy ; that is, the ability of a judge model to assign accurate preference ratings regardless of its own skill in a given domain. The capabilities a model exhibits on a given task may inform its preferences on that same task; specifically, one might imagine that an LLM-judge’s distribution of preferences will differ depending on if it could succeed on the query in question. If evaluations on these “harder” queries show greater uncertainty, then a model might choose its own, incorrect response at random. The inability to distinguish between a model’s general evaluative proficiency and an active self-bias undermines the causality of previously established results. We find that evaluator uncertainty produces artifacts in reported self-preference bias. If we replace the judge’s response with an incorrect response from a different model, we can compare the judge’s preference distribution on this “proxy” test against its own, isolating the effect of a judge evaluating itself. Contributions First, we pose an Evaluation Quality Baseline that calibrates self-preference statistics using an output-matched control group. For queries where a judge produced an incorrect or inferior response (cases where self-preference would be illegitimate ), we retrieve responses to that query from other models which were also incorrect/inferior, and elicit from the judge the probability that it votes for this alternative. By collecting probabilities from judgments which do not feature the judge’s own response, there is no longer a “self” to prefer. Taking the difference in probabilities between the original case and this proxy test yields the relative surplus of preference caused by auto-evaluation. We describe an outcome-matching approach to collecting proxies and implement our baseline as a paired t-test, where the test statistic produces this self-preference effect. We then implement this baseline on 9 datasets across 16 models. Finally, we offer a preliminary, entropy-based analysis on the mechanisms which produce evaluator biases, which attempt to capture markers of confidence in pairwise evaluations. We operationalize these proposals on reproductions of four landmark self-preference papers, showing that evaluator uncertainty accounts for an average of 89.6% of measured self-preference , substantially reducing but not eliminating evidence for bias; furthermore, 44% of experiments should show no or negative bias , and 50% report self-preference values that lose statistical significance under the null. Furthermore, we test if inducing chain-of-thought reasoning in the judge reduces bias in the controlled setting, finding limited evidence despite previous claims. We conclude with hypotheses on the mechanisms which produce evaluator noise, including a preliminary analysis on asymmetric entropy conditioned on evaluator correctness. Together, our findings distill self-preference into a clear, causal signal to improve future meta-evaluation, bias and calibration research. Figure 2 : Judge Task Accuracy versus Illegitimate Self-Preference (Sec. 4.2.1 ). With our proposed baseline, reported bias drops substantially. 2 Background 2.1 Self-Preference and Other Biases in LLM Evaluators Literature on LLM evaluation has rapidly expanded in recent years, with numerous surveys cataloging the diverse methodologies and challenges ( 41 ; 34 ; 6 ; 21 ; 20 ; 13 ) . A significant subset of this work focuses on the reliability and biases of LLM evaluators, identifying various failure modes including inconsistency, susceptibility to manipulation, and context sensitivity ( 17 ; 40 ; 9 ) . Among these, self-preference has emerged as a particularly intriguing phenomenon, where models appear to favor their own outputs over those of other models. 30 isolate this behavior using proxy tests on summarization tasks, while 26 observe a similar behavior using judges trained on encoding-based representations. These early approaches faced limitations due to their entanglement of self-preference signals with objective quality. A model which might achieve a 100% win-rate against a significantly weaker counterpart may correctly prefer its own outputs 100% of the time, unfairly earning the title of “narcissist”. To address this confound, 7 , 39 , 22 , and 27 proposed experimental setups based on response evaluations from verifiable tasks like mathematics, code, or multiple-choice responses. For subjective data, 8 utilize “gold labels” from a panel of stronger, third-party language models to emulate human judgment. Both approaches enable “oracle labels” which add a dimension of ground truth to pairwise comparisons. Many of these works find that a judge’s accuracy on a task correlates positively with its accuracy evaluating that same

Read the complete text