LEXam :Benchmarking Legal Reasoning on 340 Law Exams
Yu Fan Jingwei Ni Jakob Merane Yang Tian, Yoan Hermstrüwer Yinya Huang Mubashara Akhtar Etienne Salimbeni, Florian Geering Oliver Dreyer Daniel Brunner Markus Leippold, Mrinmaya Sachan Alexander Stremitzer Christoph Engel, Elliott Ash Joel Niklaus · 2025
Read LEXam :Benchmarking Legal Reasoning on 340 Law Exams by Yu Fan Jingwei Ni Jakob Merane Yang Tian, Yoan Hermstrüwer Yinya Huang Mubashara Akhtar Etienne Salimbeni, Florian Geering Oliver Dreyer Daniel Brunner Markus Leippold, Mrinmaya Sachan Alexander Stremitzer Christoph Engel, Elliott Ash Joel Niklaus (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-12864.txt.
Introduction
Yu Fan and co-authors published this paper in 2025. It introduces LEXam, a benchmark that tests language models on real law school exams. Long legal reasoning, where an answer must spot issues, recall rules, and apply them step by step, remains a weak spot for these models.
The set holds 7,537 questions from 340 exams across 116 courses, in English and German. It mixes open written problems with multiple choice items. For the open problems, marking guides spell out the reasoning path a good answer should take. Current models handled choice questions far better than the open ones, which demand structured multi-step work. The authors grade model reasoning with a panel of judge models, checked against human experts, and the grades lined up well. They offer the setup as a scalable way to score legal reasoning.
Catalog record
| CREATOR | Yu Fan Jingwei Ni Jakob Merane Yang Tian, Yoan Hermstrüwer Yinya Huang Mubashara Akhtar Etienne Salimbeni, Florian Geering Oliver Dreyer Daniel Brunner Markus Leippold, Mrinmaya Sachan Alexander Stremitzer Christoph Engel, Elliott Ash Joel Niklaus |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 101 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-12864
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LEXam :Benchmarking Legal Reasoning on 340 Law Exams Yu Fan Jingwei Ni Jakob Merane Yang Tian, Yoan Hermstrüwer Yinya Huang Mubashara Akhtar Etienne Salimbeni, Florian Geering Oliver Dreyer Daniel Brunner Markus Leippold, Mrinmaya Sachan Alexander Stremitzer Christoph Engel, Elliott Ash Joel Niklaus Abstract Long-form legal reasoning remains a key challenge for large language models (LLMs) in spite of recent advances in test-time scaling. To address this, we introduce LEXam , a novel benchmark derived from 340 law exams spanning 116 law school courses across a range of subjects and degree levels. The dataset comprises 7,537 law exam questions in English and German. It includes both long-form, open-ended questions and multiple-choice questions with varying numbers of options. Besides reference answers, the open questions are also accompanied by explicit guidance outlining the expected legal reasoning approach such as issue spotting, rule recall, or rule application. Our evaluation on both open-ended and multiple-choice questions present significant challenges for current LLMs; in particular, they notably struggle with open questions that require structured, multi-step legal reasoning. Moreover, our results underscore the effectiveness of the dataset in differentiating between models with varying capabilities. Deploying an ensemble LLM-as-a-Judge paradigm with rigorous human expert validation, we demonstrate how model-generated reasoning steps can be evaluated consistently and accurately, closely aligning with human expert assessments. Our evaluation setup provides a scalable method to assess legal reasoning quality beyond simple accuracy metrics. Project page: https://lexam-benchmark.github.io/ . 1 Introduction Legal reasoning is a critical frontier for large language models (LLMs) specifically and artificial intelligence (AI) at large, requiring specialized domain knowledge and advanced reasoning abilities such as precedent interpretation, statutory analysis, and legal inference. Despite progress in general reasoning, legal reasoning remains difficult and under-assessed in NLP research. Moreover, the legal domain is inherently high-stakes and a failure to thoroughly examine the capabilities and limitations of models could lead to serious real-world consequences ( Kleinberg et al., 2018 ; Medvedeva and Mcbride, 2023 ; Mahari et al., 2023 ) . As LLMs are used increasingly for various legal tasks, such as legal judgment prediction ( Wang et al., 2024a ; Shui et al., 2023 ; Jiang and Yang, 2023 ; Deng et al., 2023 ) , legal summarization ( Bauer et al., 2023 ; Ash et al., 2024 ) , legal interpretation ( Savelka et al., 2023 ; Engel and McAdams, 2024 ; Dugac and Altwicker, 2025 ; Luo et al., 2025 ) , legal case retrieval ( Ma et al., 2024 ; Zhou et al., 2023 ; Mahari et al., 2024 ) , legal argument classification ( Thalken et al., 2023 ; Chlapanis et al., 2024 ) , and contract drafting ( Lam et al., 2023 ) , it is imperative to assess and understand their capabilities and limitations. However, LLM benchmarking on deterministic tasks has so far mainly concentrated in STEM domains, such as math Olympiad problems ( Zheng et al., 2022 ; Liu et al., 2023 ; Tsoukalas et al., 2024 ; Ren et al., 2025 ) and physics problems ( Feng et al., 2025 ; Zhang et al., 2025 ; Chow et al., 2025 ) . Reasoning models such as OpenAI o3 ( OpenAI, 2025c ) and DeepSeek-R1 ( DeepSeek-AI, 2025a ) have demonstrated impressive performance in tasks dominated by deduction and established scientific rules. Nevertheless, their abilities in more nuanced and context-dependent domains, such as the law, remain poorly understood ( Engel et al., 2025 ; Niklaus et al., 2025 ) . Legal reasoning requires not only rigorous deductive and inductive logic but also their application to complex practical scenarios, often involving ill-defined problems. This domain offers an ideal testbed for non-formal reasoning, yet it remains underexplored in the context of LLM performance. In contrast, STEM benchmarks allow for straightforward validation—for example, by checking final numerical answers or employing formal verifiers. Adopting this outcome-based evaluation paradigm, prior legal reasoning benchmarks ( Guha et al., 2023 ; Fei et al., 2023 ; Chlapanis et al., 2024 ; Zheng et al., 2025 ) have primarily assessed the correctness of a model’s final outputs, without explicitly evaluating the intermediate steps necessary for reaching a legally sound conclusion. Consequently, these benchmarks often overlook the reasoning process itself, leaving it unclear why LLMs fail to reach correct conclusions in some cases. This limitation raises concerns of tangible harm when LLM-based systems are deployed in practice. Furthermore, while jurisdictions differ in their authoritative sources and treatment of precedent, they share a common foundation in legal reasoning, though with notable distinctions between common law and continental law systems. Common law systems (e.g. the United States, United Kingdom, and Australia) emphasize case precedent and judge-made law, whereas continental law systems (prevalent in most of Europe, Latin America, and parts of Asia) rely primarily on comprehensive legal codes and statutes. Across traditions, legal reasoning involves identifying the legal issue from the facts, determining the relevant rules, and applying those rules to the case. It also depends on diverse linguistic expressions that vary across systems. Prior studies ( Ryan et al., 2024 ; Jin et al., 2025 ; Fan et al., 2025 ; Tian et al., 2026 ) show that LLMs exhibit cultural biases tied to language use. Evaluating LLMs on multilingual legal data is therefore essential to align them with real-world needs. Figure 1: Process for creating LEXam , a comprehensive legal reasoning benchmark derived from real law school exams. Created through careful expert extraction and curation, LEXam contains 2,841 open-ended and 4,696 multiple-choice questions (MCQs), each with detailed domain metadata. Open-ended questions support both process- and outcome-based evaluation by LLMs-as-a-Judge and human judges, while MCQs provide clear, outcome-based assessments. To address these gaps, we introduce LEXam , a multilingual legal reasoning benchmark designed to assess the process-based and outcome-based correctness of the answers generated by LLMs, as illustrated in Figure 1 . Based on exams in Swiss, European, and international law, LEXam provides a comprehensive framework for evaluating the legal reasoning capabilities of LLMs through problem sets available in both English and German. The dataset comprises 7,537 law exam questions in English and German. It includes both long-form, open-ended questions and multiple-choice questions (MCQs) with varying numbers of options. Each open question is paired with reference answers and clear normative guidance, outlining the expected legal reasoning chain. This allows for thorough benchmarking of an LLM’s legal reasoning capabilities beyond merely evaluating final answers. Questions are sourced from 340 law exams across 116 courses at the Bachelor’s and Master’s levels taught by leading law professors at the University of Zurich Faculty of Law, one of Europe’s largest and most prestigious law schools (see Appendix B for further details). As a result, LEXam spans a wide range of legal areas both in domestic and international law, thus providing a robust benchmark for the evaluation of legal reasoning skills (see Table 6 for more details). Experimental results show that state-of-the-art (SOTA) models struggle with multi-step reasoning and applying legal rules and principles to novel scenarios. LLMs particularly struggle with open questions requiring structured legal reasoning. By designing and deploying an ensemble LLM-as-a-Judge evaluation framework, we demonstrate that model-generated reasoning steps can be evaluated consistently and accurately, closely aligning with human expert assessments. This approach provides a scalable method