The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMs
Nitay Calderon, Roi Reichart, Rotem Dror · 2025
Read The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMs by Nitay Calderon, Roi Reichart, Rotem Dror (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-10970.txt.
Introduction
Nitay Calderon, Roi Reichart, and Rotem Dror propose a statistical test for deciding when a language model may replace human labelers, in this 2025 arXiv paper. Models now grade essays, tag data, and judge other models across many fields, yet no standard check says when that swap is safe. Their Alternative Annotator Test needs only a modest slice of double-labeled examples to give a verdict.
The authors also offer a clear yardstick for comparing machine and human annotators. They ran the procedure on ten datasets with six models and four prompting styles. Closed models sometimes passed as human replacements; the open models tested did not. Prompt style mattered a lot. They urge the field to test before trusting.
Catalog record
| CREATOR | Nitay Calderon, Roi Reichart, Rotem Dror |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 119 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-10970
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMs Nitay Calderon, Roi Reichart, Rotem Dror Abstract The ‘‘LLM-as-an-annotator’’ and ‘‘LLM-as-a-judge’’ paradigms employ Large Language Models (LLMs) as annotators, judges, and evaluators in tasks traditionally performed by humans. LLM annotations are widely used, not only in NLP research but also in fields like medicine, psychology, and social science. Despite their role in shaping study results and insights, there is no standard or rigorous procedure to determine whether LLMs can replace human annotators. In this paper, we propose a novel statistical procedure, the Alternative Annotator Test (alt-test), that requires only a modest subset of annotated examples to justify using LLM annotations. Additionally, we introduce a versatile and interpretable measure for comparing LLM annotators and judges. To demonstrate our procedure, we curated a diverse collection of ten datasets, consisting of language and vision-language tasks, and conducted experiments with six LLMs and four prompting techniques. Our results show that LLMs can sometimes replace humans with closed-source LLMs (such as GPT-4o), outperforming the open-source LLMs we examine, and that prompting techniques yield judges of varying quality. We hope this study encourages more rigorous and reliable practices. 1 1 1 Code for the procedure and datasets are available at: https://github.com/nitaytech/AltTest 1 Introduction The rise of Large Language Models (LLMs) has transformed the field of Natural Language Processing (NLP), bringing unprecedented capabilities in reasoning and generating human-like text ( Kojima et al., 2022 ; Achiam et al., 2023 ; Laskar et al., 2023 ; Yang et al., 2024 ) . Recently, a new trend has emerged where LLMs are employed as annotators and judges across various NLP applications ( Li et al., 2024a ; Tan et al., 2024b ) . One key advantage of LLM-as-an-annotator and LLM-as-a-judge 2 2 2 The term “LLM-as-a-judge” typically refers to LLMs evaluating outputs of other LLMs. It can be viewed as a special case of the broader “LLM-as-an-annotator” paradigm. However, since “LLM-as-a-judge” is more widely used, we adopt it throughout this work to refer more generally to any evaluation, annotation, or labeling of texts (or images) traditionally performed by humans, regardless of the input source. paradigms is the scalability and speed of LLMs. They can quickly annotate large-scale datasets, reducing the time required for tasks traditionally performed by costly human annotators Nasution and Onan (2024) . LLMs also avoid challenges inherent to human factors, such as fatigue and guideline misinterpretation Uma et al. (2021) ; Bartsch et al. (2023) . In certain cases, they even outperform crowd-workers ( Gilardi et al., 2023 ; Nahum et al., 2024 ) . Indeed, LLMs-as-judges are extensively used in research, taking on a pivotal role once filled by humans. They are employed to annotate new datasets ( Gat et al., 2024 ; Tan et al., 2024b ) , or refine existing ones ( Nahum et al., 2024 ; Pavlovic and Poesio, 2024 ) , and commonly serve as evaluators for benchmarking models and methods ( Ahmed et al., 2024 ; Gu et al., 2024 ; Li et al., 2024a ) . LLMs’ influence extends far beyond the NLP field. They annotate papers for literature reviews ( Calderon and Reichart, 2024 ; Joos et al., 2024 ) or extract findings from academic literature ( Khraisha et al., 2024 ; Naik et al., 2024 ) . They are also utilized in cognitive sciences to simulate human subjects ( Aher et al., 2023 ; Shapira et al., 2024 ; Trott, 2024 ) and in social science, researchers leverage LLM annotations to uncover social and cultural insights ( Ventura et al., 2023 ; Ziems et al., 2024 ) . Accordingly, LLMs directly shape the results, findings, and insights of studies and guide the direction of scientific inquiry, prioritization, and innovation. Despite the advantages of the LLM-as-a-judge paradigm, research shows that LLMs amplify biases, leading to unfair or inconsistent judgments ( Ashktorab et al., 2024 ; Chen et al., 2024c ; Ye et al., 2024 ) and that they may struggle with tasks that require deep contextual understanding or domain-specific expertise ( Ravid and Dror, 2023 ; Szymanski et al., 2024 ) . These weaknesses highlight the need for rigorous evaluation and transparency when relying on LLM annotations in research. Yet, many studies employing LLM annotations do not explicitly measure the alignment between LLMs and humans, and those that do typically use traditional measures such as accuracy (% agreements), F1 score, Inter-Annotator-Agreement (IAA) kappas, and correlation ( Li et al., 2024b ) , which have limitations. To start, IAA measures assess agreement among a group of annotators, while we aim to compare the LLM to the group. Other measures frequently rely on majority vote labels, overlooking important nuances that individuals introduce. Moreover, there are no established criteria for making a definitive yes/no decision on whether an LLM can replace humans (e.g., “is an F1 score of 0.6 sufficient?” ). This decision demands statistical rigor, which often lacks in the way researchers apply traditional measures. Finally, they can only evaluate whether an LLM matches human performance (i.e., is bounded by it) but cannot determine whether it provides a better alternative. We argue that to justify using an LLM instead of human annotators, researchers should demonstrate that the LLM offers a better alternative to recruiting human annotators. In other words, when factoring in the cost-benefit and efficiency advantages of LLM annotations, they should be as good as or better than human annotations. In this paper, we propose a statistical procedure to verify this claim, which we call the Alternative Annotator Test , or simply alt-test . This procedure is simple and requires minimal effort to apply; it involves comparing the LLM to a small group of human annotators (at least three) on a modest subset of examples (between 50 and 100). Our procedure is described in § 3 and illustrated in Figure 1 . Once applied, researchers can confidently rely on the LLM’s annotations for their work. In addition, we define a measure for comparing LLM judges called the Average Advantage Probability . This measure is naturally derived from our statistical procedure and represents the probability that the LLM annotations are as good as or better (e.g., by being closer to the majority) than those of a randomly chosen human annotator. It possesses desirable properties that traditional measures lack while maintaining a high correlation with them. It is versatile, supports different types of annotations, and is highly interpretable. We exemplify the application of our procedure with six LLMs and four prompting techniques. To this end, we curate a diverse collection of ten datasets, each with instances annotated by multiple annotators. Our datasets vary in size, annotation types (discrete, continuous, and free-text), number of annotators (3 to 13), and levels of annotator expertise (crowd-workers, skilled annotators, and experts). They encompass a wide range of language tasks, including two vision-language tasks. Our results indicate that in many cases, LLMs can serve as an alternative to human annotators. Specifically, on nine datasets, at least one LLM, with some prompting technique, successfully passed the alt-test. We found that closed-source LLMs (such as GPT-4o and Gemini-1.5) consistently outperform open-source models we examined (like Mistral-v3 and Llama-3.1), and that in-context learning generally