Knockout LLM Assessment: Using Large Language Models for Evaluations through Iterative Pairwise Comparisons
Isik Baran Sandan, Tu Anh Dinh, Jan Niehues · 2025
Read Knockout LLM Assessment: Using Large Language Models for Evaluations through Iterative Pairwise Comparisons by Isik Baran Sandan, Tu Anh Dinh, Jan Niehues (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-03785.txt.
Introduction
Isik Baran Sandan, Tu Anh Dinh, and Jan Niehues grade language models by tournament instead of by checklist, in a 2025 paper. Using a model as judge is now standard for scoring translations and exam answers, but single-pass judging lacks a global sense of the field; scores drift without direct comparisons.
Their knockout format pits outputs against each other pairwise, round after round, like a sports bracket. Repeated comparisons let the judge build a consistent ranking as weaker entries fall away. Tested on university exam scoring and machine translation across three judge models, the method raised agreement with human experts by about seven correlation points on average. Tournaments cost more comparisons than single passes, but the paper argues the reliability is worth it where scores carry consequences.
Catalog record
| CREATOR | Isik Baran Sandan, Tu Anh Dinh, Jan Niehues |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 20 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-03785
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Knockout LLM Assessment: Using Large Language Models for Evaluations through Iterative Pairwise Comparisons Isik Baran Sandan, Tu Anh Dinh, Jan Niehues Abstract Large Language Models (LLMs) have shown to be effective evaluators across various domains such as machine translations or the scientific domain. Current LLM-as-a-Judge approaches rely mostly on individual assessments or a single round of pairwise assessments, preventing the judge LLM from developing a global ranking perspective. To address this, we present Knockout Assessment, an LLM-as-a-Judge method using a knockout tournament system with iterative pairwise comparisons. Experiments across three LLMs on two datasets show that knockout assessment improves scoring accuracy, increasing Pearson correlation with expert evaluations by 0.07 on average for university-level exam scoring and machine translation evaluations, aligning LLM assessments more closely with human scoring. 1 Introduction Across various domains, and especially for scientific research, accurate and consistent evaluations are very crucial for informed decision-making. However, the inherent scale and subjectivity make this task very challenging and time-consuming. In recent years, the methodology of “LLM-as-a-Judge” Zheng et al. 2023 has emerged to tackle this challenge, where instead of humans, Large Language Models (LLMs) take the role of the expert to evaluate complex tasks. Using LLMs as evaluators allows us to mimic the abilities of human experts, making evaluations cost-effective and scalable. Although many approaches to LLM-as-a-Judge exist, the most common is individual assessment, in which the evaluation prompt consists of only the question and the corresponding answer, which is to be evaluated Chiang and Lee 2023 . While this approach has already shown to yield good evaluation results next to providing scalability Chiang and Lee 2023 ; Dinh et al. 2024 , it does not consider the relative strength of answers in a set to a given question. The more recent approach of pairwise assessment tries to address this issue by providing two responses to the judge LLM each time, however, it still fails to account for a global ranking perspective, as pairwise comparisons do not analyze how all responses compare to each other in the broader sense. In this paper we present an LLM-as-a-judge method called Knockout Assessment to address this challenge, which can be seen as a variation of the tournament system used by Zheng et al. 2023 , differing in that it makes use of iterative pairwise comparisons. Instead of isolating responses individually or in pairs for evaluation, Knockout Assessment focuses on an iterative process where responses are compared against one another multiple times in a tournament manner. In each round, stronger responses advance to compete against each other in later rounds, allowing us to refine the scores progressively throughout the tournament. This approach allows the judge LLMs to develop a global perspective on responses without requiring all replies to be included in a single prompt, which would otherwise result in an impractically long context length. To summarize, our contributions are as follows: • Knockout Assessment, an LLM-as-a-Judge methodology which makes use of iterative pairwise comparisons in a knockout tournament system for more accurate evaluations • Analysis of Knockout Assessment’s performance compared to individual assessment’s and naive pairwise assessment’s performance on two different datasets concerning scientific evaluation and machine translation evaluation. 2 Related Work Individual Assessment One approach to LLM-as-a-Judge is individual assessment, where the judge LLM is provided with a prompt or a question, the corresponding answer, the scoring criteria and is asked to provide an evaluation such as a grade. Various studies have used this method for tasks such as evaluating story generation, scoring quality of different texts according to different criteria, or grading university-level exams Chiang and Lee 2023 ; Wang et al. 2023 ; Dinh et al. 2024 . Pairwise Assessment Another more recent LLM-as-a-Judge approach is pairwise prompting, in which the LLM judge is provided with two responses to the prompt instead of one. The judge LLM is then asked to evaluate both responses. This has shown to be an effective LLM-as-a-Judge method for ranking documents, as it gives the judge LLM direct comparison points while making evaluations Liusie et al. 2024 . However, this method still does not make the Judge LLM develop a global ranking perspective. Chatbot Arena Approach Zheng et al. 2023 made use of an ELO system in which all possible answer pairs are evaluated against one another. This approach is thus able to make implicit use of a global view over the dataset while assigning scores. However, pairing all possible answers results in a computational time of O(N^{2}) . Sorting Based Approaches To address this inefficiency, Qin et al. 2024 introduced two new methods. First approach uses Heapsort with pairwise comparisons to sort out the possible answers ( O(NlogN) ). Second is a sliding window approach, making use of individual passes in the Bubble Sort algorithm for a constant number K times ( O(N) ). 3 Knockout Assessment We propose using multiple iterative pairwise comparisions instead of individual assessment with Knockout Assessment. In each pairwise assessment, one pairwise ranking prompt similar to the comparative prompt introduced by Liusie et al. 2024 is used. In each prompt, one question and two answers to that question are provided to the judge LLM, which is asked to evaluate both of those answers. We call this a “question-level-match”. The exact prompts we used for our experiments can be found in Appendix A.2 . From the response generated by the judge LLM, the score each individual answer got is parsed and saved to the list of scores for that answer, which keeps track of all the scores an answer got throughout all its question-level-matches. The answer which got the higher score advances to the next round to be matched up against another answer. The order of texts in pairwise rankings has shown to be an influential factor in the LLMs decision making Resnik 2024 , thus we also collected the results with using a debiasing methodology similar to the one introduced by Liusie et al. 2024 , averaging scores from both possible orderings of each answer pair. Debiasing thus results in double the compute-time compared to a regular question-level-match. The main methodology behind our appproach is a knockout tournament system that iteratively uses the question-level matches. In each tournament round, the N available answers to a question are randomly assgined to pairs. Each pair then enters a question-level match, and the higher scoring response advances to the next round. In the case when N is odd, one answer directly advances to the next round. This continues until we reach a tournament round with a single response. Once the tournament ends, the final evaluation score for each answer is computed as the average of all the scores it received throughout the tournament. An example tournament with N = 4 answers is depicted in Figure 1 . The full algorithm is given in A.1 . Figure 1: An example Knockout-Tournament with 4 answers for a question. 4 Experiments Datasets Overall, the datasets we used include the task input, machine outputs, and human-assgined scores on the machine output. The first dataset is SciEx Dinh et al. 2024 , which consists of university exam questions, LLM answers, human grades and LLM grades. Each question is labeled by difficulty and language. SciEx contains 1120 question-answer