LLM-as-a-Judge is Bad, Based on AI Attempting the Exam Qualifying for the Member of the Polish National Board of Appeal
Michał Karp, Anna Kubaszewska, Magdalena Król, Robert Król, Aleksander Smywiński-Pohl, Mateusz Szymański, Witold Wydmański · 2025
Read LLM-as-a-Judge is Bad, Based on AI Attempting the Exam Qualifying for the Member of the Polish National Board of Appeal by Michał Karp, Anna Kubaszewska, Magdalena Król, Robert Król, Aleksander Smywiński-Pohl, Mateusz Szymański, Witold Wydmański (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-04205.txt.
Introduction
Michal Karp is the lead author among seven authors testing whether language models can pass Poland's qualifying examination for the National Appeal Chamber, and whether model judges can grade such attempts. The exam joins a multiple-choice test on public procurement law with a written judgment, and a hybrid retrieval pipeline supported the candidate models.
Several models, tested closed-book and with retrieval support, scored satisfactorily on the knowledge test. None passed the practical written part, however, and model-as-judge evaluations often diverged from the official committee. Hallucinated citations, weak legal argument, and shaky logic were named limits. Automatic judging of high-stakes legal writing looks premature on this evidence.
Catalog record
| CREATOR | Michał Karp, Anna Kubaszewska, Magdalena Król, Robert Król, Aleksander Smywiński-Pohl, Mateusz Szymański, Witold Wydmański |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 119 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-04205
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LLM-as-a-Judge is Bad, Based on AI Attempting the Exam Qualifying for the Member of the Polish National Board of Appeal Michał Karp, Anna Kubaszewska, Magdalena Król, Robert Król, Aleksander Smywiński-Pohl, Mateusz Szymański, Witold Wydmański Abstract This study provides an empirical assessment of whether current large language models (LLMs) can pass the official qualifying examination for membership in Poland’s National Appeal Chamber (Krajowa Izba Odwoławcza). The authors examine two related ideas: using LLM as actual exam candidates and applying the ’LLM-as-a-judge’ approach, in which model-generated answers are automatically evaluated by other models. The paper describes the structure of the exam, which includes a multiple-choice knowledge test on public procurement law and a written judgment, and presents the hybrid information recovery and extraction pipeline built to support the models. Several LLMs (including GPT-4.1, Claude 4 Sonnet and Bielik-11B-v2.6) were tested in closed-book and various Retrieval-Augmented Generation settings. The results show that although the models achieved satisfactory scores in the knowledge test, none met the passing threshold in the practical written part, and the evaluations of the ’LLM-as-a-judge’ often diverged from the judgments of the official examining committee. The authors highlight key limitations: susceptibility to hallucinations, incorrect citation of legal provisions, weaknesses in logical argumentation, and the need for close collaboration between legal experts and technical teams. The findings indicate that, despite rapid technological progress, current LLMs cannot yet replace human judges or independent examiners in Polish public procurement adjudication. keywords Large Language Models (LLMs), legal reasoning, public procurement law, National Appeal Chamber (Poland), legal exam evaluation, Retrieval-Augmented Generation (RAG), LLM-as-a-judge 1 Introduction With the growing popularity of Large Language Models (LLMs) it is observed that people want these Artificial Intelligence (AI) models to accomplish more sophisticated tasks. One of the areas that is particularly well suited for taking advantage of them is law. This domain of human endeavor is dominated by text and thus is particularly well suited for receiving help from LLMs. Inspired by this observation, in this research we want to find out if an LLM can pass an exam for a judge in the Polish National Appeal Chamber (pol. Krajowa Izba Odwoławcza, NAC ), verifying its competence in a well defined testing procedure where theoretical knowledge of provisions has to be combined with practical ability to apply these laws in a legal case. On the other hand, the term LLM-as-a-judge 7 , functions as a name for the procedure when an LLM is applied to judge if a text generated by this or another LLM is valid vis-a-vis certain reference answer. The idea to follow such a procedure stems from two facts: it is impossible to apply well-defined metrics such as precision and recall to a generated content and that human evaluation of generated content is time consuming and costly. One solution is to fed the LLM-as-a-judge with the generated content and the reference content and ask it, how much the generated content reflects the features of the reference answer. In this research we show that combining these two approaches in the context of law may lead to unexpected an unwanted outcomes. In the rest of the paper we introduce the exam for judges in the Polish National Appeal Chamber, how this exam was approached with the use of LLMs and what were the outcomes of the preliminary experiments we have conducted to evaluate the probability of the models passing the exam. Then we describe the actual exam taken by the LLMs in the same regime as the human participants and compare the assessment made by the real examining committee with the assessment made by LLM-as-a-judge system. We show that although the metrics achieved by our models in the preliminary experiments were pretty satisfactory, the final exam was not passed by any of the models. 1.1 Research Motivation and Organizational Impact The Public Procurement Office, which provides operational support to the NAC, initiated this research to evaluate the extent to which Large Language Model (LLM) technology could support the work of Chamber adjudicators in various capacities including charge synthesis, documentation searches, identification of precedential rulings in analogous factual circumstances, legal provision analysis for identifying adjudicatory trends, and general knowledge assessment within the specialized field of public procurement law. The experimental results will inform management decisions regarding the scope of further implementation of this technology within the institutional framework. The potential positive impact of successful examination performance by AI systems on the organization extends beyond technological validation. Such outcomes could demonstrate the feasibility of AI-assisted legal research tools, potentially enhancing the efficiency and consistency of adjudicatory processes while maintaining the high standards of legal analysis required in public procurement dispute resolution. This technological advancement could contribute to more streamlined case processing and improved accessibility of legal reasoning patterns, ultimately serving the broader goal of ensuring fair and efficient public procurement procedures in Poland. 2 Exam for a Judge in the National Appeal Chamber 2.1 National Appeal Chamber The National Appeal Chamber is a specialized adjudicatory body established within the Polish public procurement system, operating as the primary instance for resolving procurement-related disputes. The Chamber serves as the competent authority for hearing appeals submitted in public procurement proceedings and replaced the previous system of arbitration panels that had operated since 2007. 2.2 Scope of Competence and Legal Framework The Chamber’s primary responsibilities encompass: (1) hearing appeals in cases referenced in Article 513 of the Public Procurement Law ( Prawo Zamówień Publicznych ) 23 ; (2) reviewing applications for lifting contract conclusion prohibitions as specified in Article 578(1) of the Public Procurement Law; and (3) adopting resolutions containing opinions on contracting authorities’ objections to the results of preliminary and ad-hoc controls conducted by the President of the Office. The legal foundation for the Chamber’s operations is established in Articles 473-492 of the Public Procurement Law. The Chamber comprises no more than 100 members, appointed by the minister responsible for economic affairs from among candidates meeting statutory requirements who achieved the best results in qualification proceedings. Members maintain independence in their adjudication and are bound exclusively by applicable legal provisions. 2.3 Qualification Process Structure for Chamber membership The recruitment process for Chamber membership follows a rigorous three-stage assessment protocol regulated by the Regulation of the Prime Minister of February 22, 2021, regarding the procedure for conducting qualification and supplementary qualification proceedings for candidates for National Appeal Chamber membership 24 . The examination structure comprises: (1) a knowledge assessment consisting of 50 single-choice questions testing theoretical understanding of public procurement and civil law, worth a maximum of 100 points; (2) a written practical examination requiring legal analysis of a factual case study within a 120-minute timeframe, also worth 100 points; and (3) a qualification interview evaluating personal characteristics and professional competencies, worth up to 30 points. 2.4 The 2025 Recruitment Cycle The most recent qualification proceeding was announced on March 7, 2025, by the Minister of Development and Technology, acting pursuant to Article 477(6) of the Public Procurement Law and Section 18(1) of the Prime Minister’s Regulation of February 22, 2021. The recruitment cycle established a limit of 13 positions to be filled through the qualification process, with application submissions accepted until March 3, 2025. The qualification requirements mandate candidates to possess: Polish citizenship;