No-Knowledge Alarms for Misaligned LLMs-as-Judges
Andrés Corrada-Emmanuel · 2025
Read No-Knowledge Alarms for Misaligned LLMs-as-Judges by Andrés Corrada-Emmanuel (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-08593.txt.
Introduction
Using language models to grade other language models creates a regress: who grades the graders when no ground truth exists? Andrés Corrada-Emmanuel answers with logic rather than more graders, in a 2025 research paper written alone. Disagreement between judges carries information. If two judges disagree about a third model's work, they cannot both be perfectly right.
Corrada-Emmanuel formalizes that idea as a linear programming problem over answer counts for any finite test. From agreement patterns alone, the method computes the only possible values of each judge's grading ability. It then raises alarms when a judge must be violating a user-set accuracy requirement. The alarms carry no false positives, a strong guarantee. Such no-knowledge checks could keep automated evaluation honest as models increasingly judge models in industry pipelines.
Catalog record
| CREATOR | Andrés Corrada-Emmanuel |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 14 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-08593
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
No-Knowledge Alarms for Misaligned LLMs-as-Judges Andrés Corrada-Emmanuel Abstract If we use LLMs as judges to evaluate the complex decisions of other LLMs, who or what monitors the judges? Infinite monitoring chains are inevitable whenever we do not know the ground truth of the decisions by experts and we do not want to trust them. One way to ameliorate our evaluation uncertainty is to exploit the use of logical consistency between disagreeing experts. By observing how LLM judges agree and disagree while grading other LLMs, we can compute the only possible evaluations of their grading ability. For example, if two LLM judges disagree on which tasks a third one completed correctly, they cannot both be 100% correct in their judgments. This logic can be formalized as a Linear Programming problem in the space of integer response counts for any finite test. We use it here to develop no-knowledge alarms for misaligned LLM judges. The alarms can detect, with no false positives, that at least one member or more of an ensemble of judges are violating a user specified grading ability requirement. 1 Introduction No intelligence, however powerful, can escape the laws of logic and physics ( Tegmark and Omohundro 2023 ). We can use logical consistency between disagreeing LLM judges to ameliorate the problem of infinite monitoring chains whenever ground truth about the correctness of LLM decisions is absent. This can occur as an economic choice - we choose to automate a process with LLMs and want to further automate supervision by using other LLMs as judges. Or it can occur as an ignorance problem on our part - we are trying to supervise LLMs smarter than us or ones doing tasks we cannot understand. Either case is a no-knowledge situation where logical consistency, not correctness, is the only tool available for formalizing evaluations. We can build a logic of unsupervised evaluation for the task of grading multiple choice exams. The central question in any logic of unsupervised evaluation is – what group evaluations are logically consistent with how we observe experts agreeing and disagreeing in their decisions? This computation can be formalized for multiple-choice test takers or, equivalently, classifiers. It is easy to be seduced into thinking that a logic formalism for classification tests has nothing to do with LLMs carrying out complex tasks and evaluations. The problem of infinite monitoring chains, however, makes multiple choice exams inevitable in any holistic evaluation framework that, seemingly paradoxically, should include the evaluators themselves. As LLMs evolve in capability, evaluations have to evolve with them. For this reason, the development and validation of suitable benchmarks and evaluation frameworks is an active area of research. Given the range of tasks they can perform, holistic evaluation frameworks, like HELM ( Liang et al. 2023 ) measure multiple metrics across different "dimensions" of performance (e.g. accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency). The problem of evaluating LLMs at scale has an economic side and an epistemological one. The principal-agent problem ( Wikipedia contributors 2025 ) in economics is concerned with incentives when a principal delegates a task to an agent that may have conflicting interests and objectives. The epistemological problem with evaluating LLMs occurs whenever we want to scale it across number of evaluations or ability of the LLMs. We can deal with a large number of evaluations by delegating the evaluation task to an agent – the idea behind the concept of LLM-as-a-Judge ( Chang et al. 2023 ). But this approach cannot resolve the problem of evaluation at scale whenever the principal is unable to judge the agents because the ground truth for the correctness of their decisions is unavailable. This occurs when we are trying to evaluate agents more intelligent than us and we do not understand their work. But it also occurs everyday in the industrial use of LLMs when we do not have the time or money to use panels of human judges to judge their quality in production ( Chang et al. 2023 ). This paper proposes to ameliorate the epistemological issues of the principal-agent monitoring problem by using logical consistency between disagreeing LLMs. If two LLMs as judges disagree on their grades for a third worker LLM, for certain evaluation domains, we can take this as conclusive proof that they cannot both be 100% correct in their grading. This simple example is a no-knowledge alarm. Having observed the one-bit of information that the LLM judges disagreed in their grades, with no knowledge of the test or its correct answers, we can correctly deduce that one or both did not grade perfectly. Before discussing the construction of the proposed logic, it is important to note the limits of what any logic can do in unsupervised settings. Logical consistency cannot detect that agreeing LLMs are correct. In the simple example above, if we had observed the two LLM judges agreeing on their evaluations of a third, we would not be able to logically exclude both of them being perfect, as unlikely as that may be. That can only be done when we have logical correctness – in supervised settings where the ground truth for the test answers is available. In that sense, logical consistency between members of an ensemble is acting like error-correcting codes where only some bit errors can be detected. Logical consistency is also unable to establish the validity of any evaluation. Validation of LLM evaluations and the use as LLM-as-a-Judge remains a scientific and engineering problem that is actively being pursued (e.g. Salaudeen et al. 2025 ; Szymanski et al. 2025 ). The rest of the paper will discuss how to generalize the one-bit alarm to multiple-choice tests under the assumption that they are valid and have a ground truth, however imperfectly that is defined. 2 A no-knowledge alarm based on logical consistency The use of the alarm, not its validity or usefulness, will be demonstrated by considering 25 pair comparisons from the MT-Bench benchmark ( Chang et al. 2023 ). Technical details on how this set was selected can be found in section A but we will outline the rationale for the selection here. Unsupervised evaluation is going to occur where, by definition, we do not have the answer key for the correct decisions of LLM-as-judges grading the complex output of assistant LLMs as is done in the MT-Bench dataset. The complexity of those assistant LLM outputs means that human experts do not agree on which is best from a pair or if both are equally good. Chang et al. 2023 reports 80% agreement on these three grading labels. But say we wish to confirm that LLM judges are aligned with the ground truth established by some majority voting between human experts. There are about 5K human judgments for pair comparisons of LLMs in the MT-Bench human judgments accessible via HuggingFace. Restricting ourselves to those that contain 2 or more human votes, we are left with 269 pair comparisons. When we further restrict that set with the decisions of two graders, gpt4-pair (denoted by gpt4 ) and the majority judgments of the authors of Chang et al(̇ authors ), we get our evaluation set of 25 pair comparisons. The confusion matrices for the grades given out