CSP: A Simulator For Multi-Agent Ranking Competitions

Tommy Mordo, Tomer Kordonsky, Haya Nachimovsky, Moshe Tennenholtz, Oren Kurland · 2025

Read CSP: A Simulator For Multi-Agent Ranking Competitions by Tommy Mordo, Tomer Kordonsky, Haya Nachimovsky, Moshe Tennenholtz, Oren Kurland (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-11197.txt.

Introduction

Tommy Mordo, Tomer Kordonsky, Haya Nachimovsky, Moshe Tennenholtz, and Oren Kurland present a 2025 paper on a simulator for ranking competitions where AI models do the competing. In search rankings, authors rewrite content to climb results, a contest older studies ran with students. Language models can now play the author role at scale.

Their simulator is highly configurable and ships with analysis tools for the contests it generates. Document authors, played by language models, react to past rankings and revise, round after round. The team generated several data sets and mined them for patterns: how content drifts, who wins, and what the ranking rewards. Code and data are public for other researchers.

Catalog record

CREATORTommy Mordo, Tomer Kordonsky, Haya Nachimovsky, Moshe Tennenholtz, Oren Kurland
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-11197

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CSP: A Simulator For Multi-Agent Ranking Competitions Tommy Mordo, Tomer Kordonsky, Haya Nachimovsky, Moshe Tennenholtz, Oren Kurland Abstract . In ranking competitions, document authors compete for the highest rankings by modifying their content in response to past rankings. Previous studies focused on human participants, primarily students, in controlled settings. The rise of generative AI, particularly Large Language Models (LLMs), introduces a new paradigm: using LLMs as document authors. This approach addresses scalability constraints in human-based competitions and reflects the growing role of LLM-generated content on the web—a prime example of ranking competition. We introduce a highly configurable ranking competition simulator that leverages LLMs as document authors. It includes analytical tools to examine the resulting datasets. We demonstrate its capabilities by generating multiple datasets and conducting an extensive analysis. Our code and datasets are publicly available for research. 1. Introduction Ranking incentives can drive corpus dynamics in competitive search settings ( Kurland and Tennenholtz, 2022 ) : document authors (publishers) might respond to rankings induced for queries of interest by modifying their documents. The goal of the modification is to improve future ranking. A case in point, in Web search, the documents most highly ranked often attract most clicks ( Joachims et al., 2005 ) . Hence, for queries of commercial intent, high ranks are of utmost importance. This practice of modifying documents to improve their future ranking is often referred to as search engine optimization (SEO) ( Gyöngyi and Garcia-Molina, 2005 ) . Competitive search ( Kurland and Tennenholtz, 2022 ) , which is our focus in this paper, refers to white hat SEO: legitimate modifications that do not hurt content quality and/or the search ecosystem. There are a few recent studies of competitive search. Ben Basat et al. ( Ben-Basat et al., 2017 ) showed using game theoretic analysis that the probability ranking principle (PRP) ( Robertson, 1977 ) is not optimal in competitive search: it leads to reduced topical diversity in the long run. The implication is that the static view of a corpus in most work on ad hoc retrieval falls short given the dynamics driven by ranking incentives. Raifer et al. ( Raifer et al., 2017 ) showed using theoretical and empirical analysis that a prevalent strategy of publishers which leads to an equilibrium is to mimic content in the documents most highly ranked in the past. Indeed, previous rankings are the only signal about the undisclosed ranking function. Goren et al. ( Goren et al., 2021 ) empirically showed that this strategy leads to a publisher herding effect with potentially unwarranted implications; e.g., reducing the volume of content relevant to a query in the corpus. Performing empirical studies of competitive search is an extremely difficult challenge ( Kurland and Tennenholtz, 2022 ) . For example, in the Web setting, multiple factors can affect the changes of Web pages, many of which may not be due to ranking incentives. Hence, while there are observational studies of Web dynamics (e.g., ( Radinsky et al., 2013 ) ), analyzing ranking-incentivized publisher strategies and corpus effects remains an open question. This challenge drove forward a new type of empirical analysis in recent work on competitive search: controlled ranking competitions were held between students ( Raifer et al., 2017 ; Goren et al., 2020 ; Goren et al., 2021 ; Nachimovsky et al., 2024 ) . The students modified documents to have them highly ranked for queries they were assigned to. They competed over a few rounds, in each of which they were shown the last ranking induced by the undisclosed ranking function. The resultant datasets were used for various analyses ( Raifer et al., 2017 ; Goren et al., 2020 ; Goren et al., 2021 ; Wu et al., 2023 ; Nachimovsky et al., 2024 ) . Ranking competitions with humans acting as publishers are valuable for offline analysis of the specific competitions that took place. However, studying new retrieval methods and/or publishers’ document modification strategies, which significantly affect the competition, is practically impossible: each new design choice calls for re-running the competition. Kurland and Tennenholtz ( Kurland and Tennenholtz, 2022 ) mentioned this challenge as a potential barrier to empirical study of competitive search and suggested to run ranking competitions using automated agents. Their call of arms, together with the increasing proliferation over the Web of generated AI content, is the motivation for the work we report here. We present a platform dubbed CSP : a multi-agent platform that uses large language models (LLMs) as publishers (agents) in ranking competitions. The platform allows the execution of large-scale and highly varied ranking competitions where various factors can be controlled: the query for which the competition is held, the ranking function, the LLM and its prompt. The platform also provides a competition Analyzer module that analyzes a single competition using various measures. In addition, the platform includes a Compare module that allows to compare different competitions; e.g., competitions run with different LLMs or different rankers. The CSP platform will be made publicly available upon acceptance of this paper, and is currently available for reviewing purposes at https://github.com/csp-platform/Simulator . To demonstrate the merits of CSP , we used it to run many ranking competitions for sets of queries where we varied the LLM and its prompt and the ranking function. We present analysis of the resultant competition datasets using the Analyzer and Compare modules. For example, we compare the corpus dynamics with that reported in past human-based ranking competitions and reveal that LLM-based agents reduce content diversity in the corpus to a larger extent than humans. Furthermore, the analysis of the competitions shows that the ranking function has less effect on the dynamics than the choice of the LLM which serves as a publisher. 2. Related Work Previous studies addressed theoretical and empirical results. Kurland and Tennenholtz ( Kurland and Tennenholtz, 2022 ) highlight several perspectives in the competitive search ecosystem. From the ranker perspective , the focus is on designing mechanisms that can enhance social welfare (i.e., mitigate herding of publishers and promote fairness among publishers ( Kurland and Tennenholtz, 2022 ) ). The publisher perspective includes benefits to document authors who strategically manipulate their documents to improve the future rankings. The user perspective includes benefit of the user, who interacts with the search engine to satisfy an information need expressed using a query. To empirically explore these perspectives, ranking competitions in various settings have been conducted and analyzed. Ranking competitions . Game theory ( Aumann et al., 1995 ; Fudenberg and Tirole, 1991 ) provides a foundational framework for modeling repeated ranking competitions where document authors do not cooperate. Several studies have used controlled ranking competitions as a tool to study dynamics in competitive settings. Raifer et al. ( Raifer et al., 2017 ) conducted the earliest controlled ranking competitions among students, revealing a key strategy employed by publishers: mimicking the documents most highly ranked in previous rounds. Goren et al. ( Goren et al., 2021 ) demonstrated how search engines could drive predefined and targeted content effects, leading to the herding phenomenon ( Banerjee, 1992 ) . Nachimovsky et al. ( Nachimovsky et al., 2024 ) extended this line of research by conducting ranking competitions where authors competed

Read the complete text