Tournament of Prompts: Evolving LLM Instructions Through Structured Debates and Elo Ratings

Anirudh Nair *, Adi Banerjee *, Laurent Mombaerts, Matthew Hagen, Tarik Borogovac · 2025

Read Tournament of Prompts: Evolving LLM Instructions Through Structured Debates and Elo Ratings by Anirudh Nair *, Adi Banerjee *, Laurent Mombaerts, Matthew Hagen, Tarik Borogovac (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-00178.txt.

Introduction

Anirudh Nair and four co-authors let prompts for language models compete in debates and evolve toward better wording. Their paper appeared in two thousand twenty-five. Writing a good prompt takes skill, trial, and time. The pain is worst for tasks judged by taste, where no clean score exists to chase.

Their framework, DEEVO, pits prompts against each other in set debates. Winners and losers both feed lessons into the next round's wording. Ratings borrowed from chess rank the field and keep weak but diverse prompts alive. Smart crossbreeding mixes the strong parts of rivals rather than splicing at random. Tests beat both hand-written prompts and rival auto-tuners on open and closed tasks. No true answers were used as feedback. The method suits teams that need strong prompts but lack the hours to handcraft them.

Catalog record

CREATORAnirudh Nair *, Adi Banerjee *, Laurent Mombaerts, Matthew Hagen, Tarik Borogovac
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-00178

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Tournament of Prompts: Evolving LLM Instructions Through Structured Debates and Elo Ratings DOI: XXXXXXX.XXXXXXX Conference: August 4–5, 2024; Toronto, Canada; Thanks: * These authors contributed equally to this work CCS: Computing methodologies Artificial intelligence Natural language processing CCS: Computing methodologies Artificial intelligence Knowledge representation and reasoning CCS: Computing methodologies Artificial intelligence Planning and scheduling Anirudh Nair *, Adi Banerjee *, Laurent Mombaerts, Matthew Hagen, Tarik Borogovac Abstract . Prompt engineering represents a critical bottleneck to harness the full potential of Large Language Models (LLMs) for solving complex tasks, as it requires specialized expertise, significant trial-and-error, and manual intervention. This challenge is particularly pronounced for tasks involving subjective quality assessment, where defining explicit optimization objectives becomes fundamentally problematic. Existing automated prompt optimization methods falter in these scenarios, as they typically require well-defined task-specific numerical fitness functions or rely on generic templates that cannot capture the nuanced requirements of complex use cases. We introduce DEEVO (DEbate-driven EVOlutionary prompt optimization), a novel framework that guides prompt evolution through a debate-driven evaluation with an Elo-based selection. Contrary to prior work, DEEVO’s approach enables exploration of the discrete prompt space while preserving semantic coherence through intelligent crossover and strategic mutation operations that incorporate debate-based feedback , combining elements from both successful and unsuccessful prompts based on identified strengths rather than arbitrary splicing. Using Elo ratings as a fitness proxy, DEEVO simultaneously drives improvement and preserves valuable diversity in the prompt population. Experimental results demonstrate that DEEVO significantly outperforms both manual prompt engineering and alternative state-of-the-art optimization approaches on open-ended tasks and close-ended tasks despite using no ground truth feedback. By connecting LLMs’ reasoning capabilities with adaptive optimization, DEEVO represents a significant advancement in prompt optimization research by eliminating the need of predetermined metrics to continuously improve AI systems. 1. Introduction Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse domains, such as literary and professional writing, code generation, and problems requiring logical reasoning. However, their performance towards a specific task remains heavily dependent on the quality of instructions — or prompts — provided to them ( Fernando et al. 2023 ; Guo et al. 2025b ) . The term prompt engineering has become widely used, signaling that prompting has become an important skill for harnessing these models’ full potential. This skill requires specialized expertise attained through learning techniques and significant trial-and-error. Furthermore, when prompts prove insufficient for a task, developers must either implement dedicated post-processing logic or employ fine-tuning strategies to address performance gaps. In that sense, the development of systems executing complex tasks while solely relying on prompt engineering is resource intensive, thus motivating the need for an automated method to optimize prompts. The automated optimization of prompts is especially challenging for tasks where performance or quality is judged subjectively, where ambiguity challenges require resolution, or where managing conflicting contexts is paramount ( Suzgun et al. 2022 ) . In these scenarios, agents are expected to learn to adapt synthesizing different types of information, make judgment calls from different perspectives, and self-ascertain branching and stopping conditions during its iteration process; all in the absence of any quality criteria or scoring functions to quantify success along said criteria. Current approaches to automated discrete prompt optimization fall into two primary categories: gradient-based methods and evolutionary strategies. Gradient-based methods operate on textual gradients defined by means of LLM generated critical feedback (examples of these are Protegi ( Pryzant et al. 2023a ) and TextGrad ( Yuksekgonul et al. 2024 ) ). These methods offer computational efficiency but typically require labeled ground truth data from which to calculate loss; risk task-specific overfitting especially when there is a lack of diversity in the examples; and do not have a mechanism to perform exploration and thus suffer from adaptability issues. Conversely, evolutionary methods (such as EvoPrompt ( Guo et al. 2025b ) and PromptBreeder ( Fernando et al. 2023 ) ) provide broader exploration capabilities but suffer from computational inefficiency due to random search and, crucially, depend on well-defined objective fitness functions — which are often unavailable for subjective tasks. We introduce DEEVO (DEbate-driven EVOlutionary prompt optimization), a novel framework that addresses the challenges of 1) exploration vs exploitation, and 2) lack of labeled ground truth / fitness functions, by guiding prompt evolution through structured debates and Elo-based selection. Exploration vs Exploitation Unlike previous approaches, DEEVO enables systematic exploration of the prompt space through two innovative evolutionary mechanisms. At its core, DEEVO employs multi-agent debate ( Du et al. 2023 ) to guide intelligent crossover, where strengths and weaknesses of parent prompts are identified before strategically combining their effective elements. This process is complemented by targeted prompt mutations that specifically modify instructions to improve task performance. DEEVO’s evolutionary approach selectively incorporates elements from both successful and unsuccessful prompts based on their identified strengths, preserving prompt effectiveness and logical structure while systematically exploring the solution space. Lack of Labeled Ground-Truth Data Unlike prior prompt optimization strategies, DEEVO does not rely on labeled data / fitness functions against ground truth in order to evaluate prompt effectiveness. Instead, it leverages LLM-powered multi-agent debates (MAD) ( Du et al. 2023 ) to evaluate prompt quality without requiring predetermined metrics. By having LLMs critique prompt outputs in a pairwise fashion and determine a winner through structured debates, DEEVO ensures a self-contained evaluation system that can assess quality across diverse tasks — including those with subjective criteria. The evaluation mechanism evolves and improves along with the prompts, incorporating novel ideas as candidate criteria for future evaluation. The resulting debate verdicts serve as a fitness proxy that simultaneously drives improvement and preserves valuable diversity in the prompt population without needing a manually crafted or separately learned objective function for fitness selection. A Modified Elo-Based Selection The Elo rating system ( Elo and Sloan 1978 ) is a robust method to rank entities (in this case, prompts) via pairwise comparisons, where each prompt maintains a numerical rating that dynamically updates based on competition outcomes. This approach has gained significant traction in LLM evaluation frameworks, with numerous benchmark systems adopting Elo-based mechanisms to rank model and prompt performance ( Ameli et al. 2024 ; Chiang et al. 2024 ; Boubdir et al. 2023 ) . Despite its robustness, Elo is particularly limited in its ability to handle newcomer prompt competitors (due to requiring many matchups to reach an accurate skill assessment); and veteran prompt competitors (due to their ratings becoming "sticky" over time, as historical matchups dilute recent performances). For this reason, DEEVO utilizes a modified Elo-selection mechanism that introduces selection quotas for newcomers and veterans; to force the prompt population to always consist of a balanced proportion of newcomer and veteran prompts. This achieves faster calibration for new prompt candidates, better capture current skill levels for veterans, and provide more accurate performance estimates. We demonstrate that DEEVO significantly outperforms both manual prompt engineering and alternative optimization approaches across both open and close ended tasks. By connecting LLMs’ reasoning capabilities with adaptive optimization, DEEVO eliminates the need for developing predetermined metrics to continuously optimize prompts, opening new possibilities for self-improving AI systems across domains where subjective quality assessment is essential. Figure 1. DEEVO. (1) First an initial set of prompts are either provided or generated. (2) Next, each

Read the complete text