Evaluating the Reliability and Fidelity of Automated Judgment Systems of Large Language Models
Tom Biskupski, Stephan Kleber · 2026
Read Evaluating the Reliability and Fidelity of Automated Judgment Systems of Large Language Models by Tom Biskupski, Stephan Kleber (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-22214.txt.
Introduction
Biskupski and Kleber test how well language models work as automated judges of other models. A judge setup pairs one model with a prompt that holds the scoring criteria. Such judges grade free-form text far faster than human reviewers. The question is whether their grades match human ones. The team tries 37 chat models with 5 judge prompts, plus a second-level judge and 5 fine-tuned models. Tasks span eight judgment categories with human-made ground truth. With a good prompt, judge scores track human scores closely.
Catalog record
| CREATOR | Tom Biskupski, Stephan Kleber |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58015 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.22214
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Evaluating the Reliability and Fidelity of Automated Judgment Systems of Large Language Models Thanks: The authors like to thank Saba Fost, Mercedes-Benz AG, and Johannes Maucher, Hochschule der Medien Stuttgart, for supporting this project. Tom Biskupski, Stephan Kleber Abstract A Large Language Model (LLM) as judge evaluates the quality of victim Machine Learning (ML) models, specifically LLMs , by analyzing their outputs. An LLM as judge is the combination of one model and one specifically engineered judge prompt that contains the criteria for the analysis. The resulting automation of the analysis scales up the complex evaluation of the victim models’ free-form text outputs by faster and more consistent judgments compared to human reviewers. Thus, quality and security assessments of LLMs can cover a wide range of the victim models’ use cases. Being a comparably new technique, LLMs as judges lack a thorough investigation for their reliability and agreement to human judgment. Our work evaluates the applicability of LLMs as automated quality assessors of victim LLMs . We test the efficacy of 37 differently sized conversational LLMs in combination with 5 different judge prompts, the concept of a second-level judge, and 5 models fine-tuned for the task as assessors. As assessment objective, we curate datasets for eight different categories of judgment tasks and the corresponding ground-truth labels based on human assessments. Our empirical results show a high correlation of LLMs as judges with human assessments, when combined with a suitable prompt, in particular for GPT-4o, several open-source models with ⩾ \geqslant 32B parameters, and a few smaller models like Qwen2.5 14B. I Introduction The widespread deployment of LLMs as chatbots [ 35 ] , virtual assistants in phones and cars [ 6 ] , and to control cyber-physical systems (CPSs) [ 48 ] , e. g., in robotics [ 32 ] make them a powerful and ubiquitous tool. This raises the need to assure their reliability, in particular facing increased interest of malicious actors to exploit vulnerabilities introduced by the usage of LLMs . Due to the non-deterministic nature of LLMs , a formal verification is equally infeasible as fixed testing schemes that cannot deal with the free-form text outputs with static rule sets for tests [ 53 ] . However, accurately evaluating capabilities, limitations, and vulnerabilities of LLMs is crucial, especially in public applications and safety-critical CPSs . Conventional evaluation techniques used for Natural Language Processing (NLP) are, e. g., BLEU [ 30 ] , METEOR [ 4 ] , and embedding based metrics like BERTScore [ 50 ] and MoverScore [ 52 ] . However, they are limited to word similarity and semantic distances and cannot capture the language complexity of LLM -generated texts. In particular, these evaluation techniques do not capture human preferences in the assessment of LLM outputs [ 53 ] . While evaluations using human experts are the gold standard for accurate assessments of LLM outputs, these are prohibitively costly and time-consuming. The rapid development of a growing variety of LLMs render any human evaluation strategy inefficient as manual assessments cannot keep pace with the number of tests that need to be conducted for a reasonable test coverage. judge prompt prompt LLM A output instruction p LLM B verdict explanation Evaluation Target/Victim Model Judge Fig. 1: Concept of an LLM as a judge As a solution, [ 53 ] formally introduced the LLM -as-a-judge approach: A powerful LLM is paired with judging instructions as a prompt to automate the evaluation of other LLMs ’ outputs as Figure 1 illustrates. The approach offers a scalable and cost-effective evaluation method while promising high alignment with human judgments. Being a comparably new technique, LLMs as judges lack thorough investigations for their reliability and agreement to human judgment depending on different judge configurations. Specifically the application of LLMs as judges for detecting vulnerabilities in LLMs was hence largely unexplored. Thus, we conduct a comprehensive analysis of multiple judge configurations consisting of different LLMs paired with different judge prompts to determine the most accurate and reliable model/prompt combinations. In addition, we identify further strengths and limitations of the LLM -as-a-judge approach. ? ? ? ? ? prompts ! ! ! ! ! outputs ✔ ✔ ✔ ✔ ✔ ground truth ? ? ? ? ? prompts Victim LLM ! ! ! ! ! outputs output instruction judge prompts human evaluator Judge LLM verdict explanation evaluate evaluate Fig. 2: Our evaluation workflow: 1. generating datasets (blue), 2. judging datasets (orange), 3. evaluate judge verdicts (green). Contributions We assess 37 differently sized conversational LLMs in combination with 5 different judge-prompt designs for their ability to provide well-formatted outputs that align with human judgments. For the best-performing combinations of 12 models and 3 prompts we conduct an thorough reliability evaluation. Finally, we assess the explanation quality of 3 selected LLMs by inspecting judges’ justification of their verdicts. Moreover, we test the efficacy of the concept of a second-level judge and investigate 5 models specifically fine-tuned for the task as judge and filter. As assessment objective for the LLM -as-a-judge, we curate eight datasets sourced from the publicly available benchmarks CategoricalHarmfulQA [ 5 ] , SimpleSafetyTests [ 43 ] , TruthfulQA [ 22 ] , and JailBreakV-28k [ 25 ] , which focus on LLM quality assessment and adversarial testing. We augment these subsets of publicly available benchmarks with self-created datasets that typically contain static question-answer pairs as input for the judge and ground-truth labels of the corresponding expected judgment. Our findings demonstrate that the LLM -as-a-judge approach is a viable alternative to human evaluations for basic quality and security assessments. LLMs with \geqslant 32B parameters, GPT-4o , and a few smaller models, e. g., QwQ , show strong alignment with human judgments. Which kind of prompt used as judging instructions is most effective highly depends on the model it is used with. Further research should assess the effectiveness on more complex scenarios and analyze the explainability and reliability in more depth. We focus on the reliability and validity of LLMs as judges—that is, their consistency and agreement with human assessments. While using adversarial inputs as benchmark datasets, our study does not investigate adversarial robustness or resistance to malicious attacks of the judge, but rather the soundness and fidelity of the automated LLM-based evaluation. II Related Work TABLE I: Overview of evaluation datasets. Dataset type is either a victim’s undesired (u) output or its correctness (c) Dataset Size Type Judge Task: Detecting content that… Source Victim Model(s) Harmful Behavior 113 u encourages or could lead to physical or mental harm. [ 5 ] , [ 43 ] Dolphin 3 Illegal Activity 86 u promotes or provides instructions for illegal behavior. [ 5 ] , [ 43 ] Dolphin 3, DeepSeek-R1 7B Personal Information 58 u can be considered personal or sensitive. GPT-4o, [ 8 ] GPT-4o Brand Harm 75 u could harm a brand’s reputation. Custom GPT-3.5 Malicious Code/Text 50 u instructs for malicious code or phishing. [ 5 ] DeepSeek-R1 7B Jailbreak Attacks 34 u indicates a successful jailbreak attack. [ 25 ] Mistral 7B, GPT-4o, Gemini 1.5 Pro Similarity 50 c is semantic similar between two short statements. Custom — Truthfulness 68 c is factually correct based on a reference. [ 22 ]