Evaluation of Large Language Models in Legal Applications: Challenges, Methods, and Future Directions
Yiran Hu, Huanghai Liu, Chong Wang, Kunran Li, Tien-Hsuan Wu, Haitao Li, Xinran Xu, Siqing Huo, Weihang Su, Ning Zheng, Siyuan Zheng, Qingyao Ai, Yun Liu, Renqun Bian, Yiqun Liu, Charles L.A. Clarke, Weixing Shen, Ben Kao · 2026
Read Evaluation of Large Language Models in Legal Applications: Challenges, Methods, and Future Directions by Yiran Hu, Huanghai Liu, Chong Wang, Kunran Li, Tien-Hsuan Wu, Haitao Li, Xinran Xu, Siqing Huo, Weihang Su, Ning Zheng, Siyuan Zheng, Qingyao Ai, Yun Liu, Renqun Bian, Yiqun Liu, Charles L.A. Clarke, Weixing Shen, Ben Kao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-15267.txt.
Introduction
Yiran Hu, Huanghai Liu, and a large team survey how language models are judged for legal work. Their 2026 arXiv paper deals with a fast-growing use of AI. Models now support judges, aid legal practice, and serve the public with legal information.
Evaluating such models is its own problem. The authors review the challenges that legal uses raise for testing. They sort the methods used so far to grade legal performance. They then set out future directions, so the next tests can be harder to game and closer to real legal needs.
Catalog record
| CREATOR | Yiran Hu, Huanghai Liu, Chong Wang, Kunran Li, Tien-Hsuan Wu, Haitao Li, Xinran Xu, Siqing Huo, Weihang Su, Ning Zheng, Siyuan Zheng, Qingyao Ai, Yun Liu, Renqun Bian, Yiqun Liu, Charles L.A. Clarke, Weixing Shen, Ben Kao |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-15267
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Evaluation of Large Language Models in Legal Applications: Challenges, Methods, and Future Directions Yiran Hu, Huanghai Liu, Chong Wang, Kunran Li, Tien-Hsuan Wu, Haitao Li, Xinran Xu, Siqing Huo, Weihang Su, Ning Zheng, Siyuan Zheng, Qingyao Ai, Yun Liu, Renqun Bian, Yiqun Liu, Charles L.A. Clarke, Weixing Shen, Ben Kao Abstract Large language models (LLMs) are being increasingly integrated into legal applications, including judicial decision support, legal practice assistance, and public-facing legal services. While LLMs show strong potential in handling legal knowledge and tasks, their deployment in real-world legal settings raises critical concerns beyond surface-level accuracy, involving the soundness of legal reasoning processes and trustworthy issues such as fairness and reliability. Systematic evaluation of LLM performance in legal tasks has therefore become essential for their responsible adoption. This survey identifies key challenges in evaluating LLMs for legal tasks grounded in real-world legal practice. We analyze the major difficulties involved in assessing LLM performance in the legal domain, including outcome correctness, reasoning reliability, and trustworthiness. Building on these challenges, we review and categorize existing evaluation methods and benchmarks according to their task design, datasets, and evaluation metrics. We further discuss the extent to which current approaches address these challenges, highlight their limitations, and outline future research directions toward more realistic, reliable, and legally grounded evaluation frameworks for LLMs in legal domains. 1 Introduction As large language models (LLMs) continue to advance in capability, they are increasingly being applied to the legal domain 77 ; 44 . Typical applications include assisting laypersons in understanding legal issues, supporting lawyers in legal practice, and providing auxiliary assistance to judges in judicial decision-making 76 ; 44 ; 24 ; 15 . This raises a fundamental question: Are LLMs truly qualified to enter the legal domain, and which types of legal tasks can they reliably perform? Previous studies 39 ; 23 have shown that LLMs are able to pass legal examinations, and some works 87 ; 52 further demonstrate that LLMs can achieve high performance in legal judgment prediction tasks. However, due to the intrinsic complexity of real-world legal scenarios and the demanding nature of legal reasoning, evaluating LLMs solely based on standardized exam-style questions or prediction accuracy is insufficient to comprehensively assess their performance in real legal applications. Consequently, a growing number of benchmarks 21 ; 27 ; 45 have been proposed to evaluate LLMs from multiple perspectives in legal tasks. Evaluating LLMs in legal tasks is significantly more challenging than conducting general-purpose evaluations 32 . Existing studies 31 ; 49 show that LLMs are increasingly being applied in real-world court settings. In real legal applications, LLMs face many issues beyond whether an answer is correct. Evaluation should not focus solely on the final result, but also consider the reasoning process and system-level constraints. The aim of evaluating LLMs in the legal domain is to systematically and automatically assess their capabilities across multiple dimensions, reflecting not only whether models arrive at correct outcomes, but also how those results are produced and under what constraints they remain valid. From the process perspective, addressing legal problems requires strong logical reasoning abilities; a model may produce a legally correct answer while relying on flawed or invalid reasoning steps. For example, when an LLM serves as a judicial assistant, a key factor in whether the public can be convinced lies in its reasoning process, such as whether it cites authentic and correct legal provisions and whether it performs logically sound inferences based on both legal rules and factual circumstances. From the constraint perspective, LLMs need to handle legal cases fairly and safely. For instance, an LLM should not exhibit discrimination based on gender or geographic origin when making judgments. Given that legal decisions directly affect individuals’ rights and social welfare, ensuring fairness, reliability, and safety is a prerequisite for real-world adoption 81 . Legal applications operate under strict normative and social constraints: even when both the final result and the reasoning appear correct, a model may still be unsuitable for deployment if it exhibits unfair bias, lacks robustness, or fails under adversarial or distributional shifts. In summary, when LLMs are applied to real-world legal settings, they need to be evaluated across the entire pipeline (Result, Process, Constraint). These three stages correspond to three evaluation dimensions: Output Accuracy (Result), Legal Reasoning (Process), and Trustworthiness (Constraint) . Only through effective evaluation along these three dimensions can models be applied more reliably in practice. Based on these three challenges, we categorize existing benchmarks and select representative benchmarks for analysis. Figure 1: Overview of the proposed framework. Outcome Accuracy aims to assess a model’s legal knowledge and its overall task performance. Existing datasets are mainly constructed around legal examinations 45 ; 20 and judicial decision prediction tasks 87 , with evaluation metrics such as accuracy for multiple-choice questions, as well as ROUGE and BERT-based similarity measures for open-ended legal responses. Trustworthiness evaluates the potential risks introduced by LLMs when performing legal tasks 81 . Current studies 31 ; 30 mainly investigate ethical concerns, fairness, and robustness, often using counterfactual analysis and adversarial attacks. Legal Reasoning measures whether the reasoning process employed by the model aligns with legal reasoning principles. Existing work 1 typically relies on fine-grained legal reasoning rubrics annotated by human experts to score the model’s reasoning steps. Despite the progress made by existing research, many challenges remain unresolved. For instance, trustworthiness evaluations still lack coverage of critical dimensions such as privacy risks and toxicity in legal contexts. Evaluations of logical inference heavily rely on manually annotated rubrics, making them difficult to scale to large datasets. Moreover, certain task-specific issues, like legal hallucination, have not yet been systematically evaluated. These limitations indicate that current evaluation frameworks for LLMs in legal tasks are far from complete, leaving substantial room for further exploration. As Figure 1 shows, in this survey, we summarize existing studies on evaluating LLMs in the legal domain, identify key challenges, and outline promising future research directions. The contributions of this survey are threefold: 1. We identify major challenges in applying LLMs to the legal domain and discuss potential solutions. 2. We systematically summarize and categorize existing evaluation methods and benchmarks for legal LLMs. For details, see https://github.com/THUYRan/Evaluation-of-LLMs-in-Legal-Applications 3. We align current evaluation approaches with outstanding challenges, highlighting open problems and future research directions. 2 Challenges: Risks and Complexity of LLM Applications in Law A comprehensive, scientific, and reliable evaluation of LLM performance in legal scenarios needs to start from their specific applications. A prerequisite for meaningful LLM evaluation is to understand where and how LLMs are being deployed in legal workflows, and how these deployments affect legal decision-making and everyday life. By systematically examining the legal tasks to which LLMs are applied, researchers can better identify the unique challenges involved in evaluating LLMs within the legal domain. In this section, we review representative applications of LLMs across three user groups — judges, lawyers, and the general public. Although these applications differ in function, authority, and risk profile, they illustrate the breadth of LLM adoption in legal contexts and highlight why legal AI evaluation needs to be carried out across the entire pipeline. 2.1 Applications to Judges LLMs are increasingly incorporated into judicial workflows, supporting tasks such as case triage, legal