Strategic Candidacy in Generative AI Arenas

Chris Hays, Rachel Li, Bailey Flanigan, Manish Raghavan · 2026

Read Strategic Candidacy in Generative AI Arenas by Chris Hays, Rachel Li, Bailey Flanigan, Manish Raghavan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-26891.txt.

Introduction

Hays and co-authors study strategic candidacy in arenas that rank generative AI models. Users vote on pairs of model outputs, and the votes build a ranking. A producer might flood the arena with near-clone models to lift its best rank. The authors show when cloning pays, using theory and simulations from the Arena platform. They propose You-Rank-We-Rank, where producers rank their own models first. That self-ranking corrects the quality estimates. The mechanism is provably near clone-robust and can sharpen overall accuracy.

Catalog record

CREATORChris Hays, Rachel Li, Bailey Flanigan, Manish Raghavan
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE112981
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.26891

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Strategic Candidacy in Generative AI Arenas Chris Hays, Rachel Li, Bailey Flanigan, Manish Raghavan Abstract AI arenas, which rank generative models from pairwise preferences of users, are a popular method for measuring the relative performance of models in the course of their organic use. Because rankings are computed from noisy preferences, there is a concern that model producers can exploit this randomness by submitting many models (e.g., multiple variants of essentially the same model) and thereby artificially improve the rank of their top models. This can lead to degradations in the quality, and therefore the usefulness, of the ranking. In this paper, we begin by establishing, both theoretically and in simulations calibrated to data from the platform Arena (formerly LMArena, Chatbot Arena), conditions under which producers can benefit from submitting clones when their goal is to be ranked highly. We then propose a new mechanism for ranking models from pairwise comparisons, called You-Rank-We-Rank (YRWR). It requires that producers submit rankings over their own models and uses these rankings to correct statistical estimates of model quality. We prove that this mechanism is approximately clone-robust, in the sense that a producer cannot improve their rank much by doing anything other than submitting each of their unique models exactly once. Moreover, to the extent that model producers are able to correctly rank their own models, YRWR improves overall ranking accuracy. In further simulations, we show that indeed the mechanism is approximately clone-robust and quantify improvements to ranking accuracy, even under producer misranking. \svgsetup inkscapelatex=false, 1 Introduction As generative AI models proliferate, there is a need to systematically compare their performance. Such evaluations allow AI users to make informed choices between models, for organizations to make informed procurement decisions, for investors to allocate investments to more promising AI labs, for AI researchers to identify promising model development techniques towards making further progress, and for the research community to focus its efforts on common tasks ( 9 ; 8 ) . This need has motivated the emergence of generative AI arenas , in which people vote on the outputs of pairs of different models. These comparisons are then aggregated into an overall ranking over models in the arena (where higher-ranked models are those that win pairwise comparisons more often). This evaluation approach offers the key advantage (over, e.g., static benchmarks) that evaluation occurs in the course of organic use, and so quality is measured on arguably user-relevant dimensions that might otherwise be difficult to capture. Accordingly, since 2024, many such arenas have emerged ( 7 ; 28 ; 6 ; 17 ; 14 ) : foremost among them currently is Arena (formerly LMArena, Chatbot Arena) ( 7 ) , which we will use as our prototypical example. Although these platforms are new, there is already evidence that their rankings are substantively important: consumers ( 18 ) , investors ( 10 ) , and tech companies themselves ( 1 ) consider generative AI arenas to be a credible and economically relevant signal about the quality of models. As these arenas’ rankings become more important, so grow incentives for companies to try to improve the standing of their models. In recent work, 23 highlight the risks of strategic manipulations: They submit several identical copies (which we’ll call clones ) of the same models to Arena, finding that one is ranked several places above the other. 1 1 1 Arena publishes confidence intervals around estimates of model qualities, and the confidence intervals for the estimated scores for the models in the 23 experiment were overlapping. Such uncertainty quantification strategies would be useful for assigning ties in the ranks between models. However, in this work, we consider the (perhaps more difficult) problem of model ranking mechanisms where ties are not allowed, and indeed, at the time of writing, Arena does not allow ties in the ranks it assigns to models. Intuitively, conditions on these platforms may be favorable to strategic candidacy: rankings are formed through a relatively small number of comparisons (typically tens to hundreds per pair of models on Arena, or thousands to tens of thousands of comparisons per model) and pairwise win margins between models are small (for example, at the time of writing, the 1st-ranked model has a 58% win rate against the 20th-ranked model). In this low-data, tightly competitive regime, small amounts of noise can substantially affect outcomes. Thus, model producers have both the incentive and the ability to improve their arena position by simply submitting identical or near-identical 2 2 2 For example, near-identical submissions might occur if a producer submits multiple, qualitatively similar checkpoints from the same training run. models. With this as a starting point, we study two central questions: 1. When can producers benefit from clones in existing ranking systems, and if they can, by how much? 2. Can we design clone-robust ranking systems for this setting? Our approach and contributions. We begin with a formal model of the generative AI arenas. We assume that voter preferences are generated via the statistical model used for inference in these arenas ( 7 ) , a Bradley-Terry (BT) model. Thus, voter behavior is in some sense ‘‘ideal’’ and the status quo statistical model used for ranking is not misspecified. However, because the number of voter preferences are limited, there is noise in the rankings generated from these preferences. We then analyze the choices of model producers, who we assume have an exogenous set of distinct models (where distinctness of models is not observed by the platform) and may submit multiple copies of the same model to the ranking mechanism. Key to our formulation is the idea that producers may hope to increase the ranking of their top models by exploiting noise in voter preferences. 3 3 3 This is related to but different from the concerns around “hand picked” scores highlighted in 23 . In their work, the focus is on using private testing (where preference data is collected before a model is included in a ranking) to try to exploit selection effects by releasing only the model ranked best during private testing. There is no such notion of private testing in our work: all models submitted to the mechanism are assumed to be released publicly. Instead, in our model, the producers’ incentive to submit multiple models comes from the fact that their utility is determined by their best-performing models, rather than the average performance of their models. In our formalization of model producer incentives, producers derive utility from being ranked first within various subsets of models ("leaderboards"), like all models, all open-weight models, models below a given cost or latency threshold, or other intrinsic features of models on which a model producer wishes to compete. We make two primary contributions, corresponding to the research questions above: Establishing clone-nonrobustness of status quo mechanisms. In Section 3 , we formally prove conditions under which the intuition above is correct: the status-quo mechanism — which fits BT via maximum likelihood estimation and directly reports the implied ranking — indeed rewards producers for submitting model clones. Each clone effectively gives the producer an extra chance of getting “lucky” with their ranking position. We confirm that this problem is significant in simulations calibrated to Arena data: we

Read the complete text