LegalEval-Q: A New Benchmark for The Quality Evaluation of LLM-Generated Legal Text
Li Yunhan, Wu Gengshen · 2025
Read LegalEval-Q: A New Benchmark for The Quality Evaluation of LLM-Generated Legal Text by Li Yunhan, Wu Gengshen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-24826.txt.
Introduction
Li Yunhan and Wu Gengshen build a benchmark for judging the quality of legal text written by AI models. Most tests of legal AI check facts alone and skip how clear or sound the text is. Their method scores text on clarity, flow, and use of legal terms. They train a scoring model, write a set of legal questions, and grade forty-nine large language models.
Three results stand out from the grading. Quality stops rising much after fourteen billion parameters, with small gains at far larger sizes. Build choices like compression and context length had little effect. Models trained to reason beat plain models across the board. Their ranking names one model family as the best mix of quality and cost.
Catalog record
| CREATOR | Li Yunhan, Wu Gengshen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-24826
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LegalEval-Q: A New Benchmark for The Quality Evaluation of LLM-Generated Legal Text Li Yunhan, Wu Gengshen Abstract As large language models (LLMs) are increasingly used in legal applications, current evaluation benchmarks tend to focus mainly on factual accuracy while largely neglecting important linguistic quality aspects such as clarity, coherence, and terminology. To address this gap, we propose three steps: First, we develop a regression model to evaluate the quality of legal texts based on clarity, coherence, and terminology. Second, we create a specialized set of legal questions. Third, we analyze 49 LLMs using this evaluation framework. Our analysis identifies three key findings: First, model quality levels off at 14 billion parameters, with only a marginal improvement of 2.7% noted at 72 billion parameters. Second, engineering choices such as quantization and context length have a negligible impact, as indicated by statistical significance thresholds above 0.016. Third, reasoning models consistently outperform base architectures. A significant outcome of our research is the release of a ranking list and Pareto analysis, which highlight the Qwen3 series as the optimal choice for cost-performance tradeoffs. This work not only establishes standardized evaluation protocols for legal LLMs but also uncovers fundamental limitations in current training data refinement approaches. Code and models are available at: https://github.com/lyxx3rd/LegalEval-Q . 1 Introduction The rapid advancement of LLMs has significantly transformed natural language processing, with their widespread deployments across specialized domains such as finance Lu et al. 2023 ; Xie et al. 2024 ; Zhang et al. 2023 , healthcare Dai et al. 2023 ; Fei et al. 2023 ; Guha et al. 2023 , and legal practice Dada et al. 2024 ; Wang et al. 2023a ; Li et al. 2023 . Current evaluation benchmarks primarily assess three key capabilities: domain-specific knowledge mastery Lu et al. 2023 ; Dai et al. 2023 ; Dada et al. 2024 , extended context processing Kamradt 2023 ; Zhao et al. 2024 , and reasoning capability Hendrycks et al. 2021 . However, these benchmarks systematically neglect an essential dimension—textual quality. This oversight leads to a critical paradox: LLMs may generate factually correct outputs that nevertheless exhibit suboptimal linguistic characteristics including awkward phrasing, stylistic inconsistencies, or impaired readability. Establishing robust textual quality assessment methodologies therefore represents a pressing need for both academic research and industrial applications, particularly in specialized domains. In the legal domain, two unresolved challenges persist. First, existing frameworks lack standardized quantitative metrics for evaluating the nuanced aspects of legal text quality. Second, the relationship between model parameters—such as scale and architectural configurations—and resultant textual quality remains poorly understood. These gaps hinder model selection and optimization processes in legal applications. Our study presents three significant contributions: 1) We have developed a specialized benchmark for evaluating the quality of legal texts. This benchmark introduces a multidimensional assessment model that measures critical quality attributes, including clarity, coherence, and terminological precision. 2) We have constructed a comprehensive validation set that includes legal questions from various subdomains. This set is designed to systematically assess the quality of responses generated by models across different legal contexts. 3) Through rigorous empirical analysis, we demonstrate how key characteristics of models—particularly their size and fine-tuning strategies—affect the quality of the text outputs. Our framework not only provides practitioners with actionable insights for selecting LLMs in legal applications but also establishes standardized evaluation protocols for future research in domain-specific text quality assessment. 2 Related Work 2.1 Traditional Natural Language Generation Evaluation With the proliferation of LLMs across diverse domains, establishing rigorous evaluation methodologies has become critical. Current benchmarks predominantly focus on task-oriented or reasoning-based assessments through verifiable instructions. The IFEval framework Zhou et al. 2023 exemplifies this approach by quantifying instruction-following accuracy via objective metrics. For complex reasoning, the BIG-Bench Hard suite Suzgun et al. 2023 evaluates multi-step problem-solving, while the MATH dataset Hendrycks et al. 2021 provides a standardized testbed for mathematical deduction. For domain-specific challenges, benchmarks adopt specialized designs: GPTA Rein et al. 2024 , which curates expert-level questions in STEM fields, and MuSR Sprague et al. 2024 targets soft reasoning in narrative contexts. Knowledge-intensive evaluations have evolved from MMLU Hendrycks et al. 2020 to its advanced variant MMLU-Pro Wang et al. 2024 , incorporating adversarial noise to better probe reasoning robustness. For non-English contexts, C-Eval Huang et al. 2023 simultaneously measures Chinese linguistic competence and cross-domain knowledge application. While these methods excel at quantifying task completion, such as solution accuracy and instruction adherence, structured reasoning, they reveal fundamental limitations when applied to open-domain text quality assessment. 2.2 Early Exploration of Text Quality Assessment Current evaluation of generated text remains predominantly reliant on similarity-based metrics Kasai et al. 2021 , which can be categorized into two paradigms: (1) lexical overlap-based methods such as ROUGE Lin 2004 and BLEU Papineni et al. 2002 that compute n-gram matches between generated and reference texts, and (2) contextualized embedding-based approaches Zhang et al. 2019 ; Zhao et al. 2019 ; Clark et al. 2019 that measure semantic similarity using deep language models. Despite their prevalent use in more than 60% of recent natural language generation(NLG) studies Kasai et al. 2021 , these metrics exhibit fundamental limitations. First, they fail to capture essential quality dimensions including content coherence Reiter and Belz 2009 and syntactic correctness Stent et al. 2005 . Second, as NLG models advance, surface-level feature matching becomes increasingly inadequate for discriminating system capabilities Gehrmann et al. 2023 . Most problematically, this similarity-based paradigm introduces circular dependency on reference texts, inherently constraining the assessment of creative or open-ended generation where optimal outputs may diverge from provided references. 2.3 LLM-driven Text Quality Evaluators To advance fine-grained evaluation in NLG, researchers have developed both specialized and unified assessment frameworks. Dimension-specific evaluators now target critical quality aspects such as summarization consistency Kryscinski and McCann 2021 ; Wang et al. 2020 and dialogue coherence Dziri et al. 2019 ; Huang et al. 2020 , while unified approaches employ diverse methodologies including adaptive prompting Yuan et al. 2021 , model ensembles Mehri and Eskenazi 2020 , and dynamic metric integration Scialom et al. 2021 to generate comprehensive quality assessments. While these methods demonstrate versatility, their lack of explainability remains a significant limitation, often reducing complex evaluations to simplistic summative metrics Li et al. 2025 . In response, the field has witnessed three key developments in transparent evaluation architectures: (1) rationale-enhanced systems like G-EVAL Liu et al. 2023 and UNIEVAL Li et al. 2025 that combine scoring with interpretable reasoning chains; (2) reference-free paradigms including GPTScore’s probability-based quantification Fu et al. 2023 and PandaLM’s efficient comparative assessment Wang et al. 2023b ; and (3) robust evaluation frameworks such as JudgeLM Zhu et al. 2023 designed for adversarial testing scenarios. These innovations, while significantly improving evaluation transparency, have inadvertently created new challenges regarding computational efficiency and scoring flexibility. The current landscape reveals two critical limitations: prohibitive computational demands from LLM-dependent architectures Liu et al. 2023 ; Fu et al. 2023 , and inflexible scoring mechanisms that either oversimplify quality distinctions or depend on rigid weighting schemes Li et al. 2025 . Our proposed solution introduces a novel decoupled evaluation framework that strategically combines a fine-tuned quality assessor with a regression-based scoring model, achieving both computational efficiency and nuanced quality analysis while maintaining