Evaluating Legal Reasoning Traces with Legal Issue Tree Rubrics

Jinu Lee, Kyoung-Woon On, Simeng Han, Arman Cohan, Julia Hockenmaier · 2025

Read Evaluating Legal Reasoning Traces with Legal Issue Tree Rubrics by Jinu Lee, Kyoung-Woon On, Simeng Han, Arman Cohan, Julia Hockenmaier (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-01020.txt.

Introduction

Jinu Lee and four coauthors introduce LEGIT in a 2025 paper on judging AI legal reasoning. Language models now produce reasoning traces, step-by-step chains of thought, in expert fields like law. Checking the quality of a trace is essential for credibility, yet the reasoning is too complex for simple answer keys.

LEGIT is a large expert-level dataset for legal reasoning, built around legal issue trees. Those trees break a case into its issues and sub-issues, forming a rubric for grading a trace. With many thousands of instances, the dataset lets researchers measure whether a model's stated reasons are sound, not just its final answer.

Catalog record

CREATORJinu Lee, Kyoung-Woon On, Simeng Han, Arman Cohan, Julia Hockenmaier
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-01020

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Evaluating Legal Reasoning Traces with Legal Issue Tree Rubrics Jinu Lee, Kyoung-Woon On, Simeng Han, Arman Cohan, Julia Hockenmaier Abstract Evaluating the quality of LLM-generated reasoning traces in expert domains ( e.g. , law) is essential for ensuring credibility and explainability, yet remains challenging due to the inherent complexity of such reasoning tasks. We introduce LEGIT (LEGal Issue Trees), a novel large-scale (24K instances) expert-level legal reasoning dataset with an emphasis on reasoning trace evaluation. We convert court judgments into hierarchical trees of opposing parties’ arguments and the court’s conclusions, which serve as rubrics for evaluating the issue coverage and correctness of the reasoning traces. We verify the reliability of these rubrics via human expert annotations and comparison with coarse, less informative rubrics. Using the LEGIT dataset, we show that (1) LLMs’ legal reasoning ability is seriously affected by both legal issue coverage and correctness, and that (2) retrieval-augmented generation (RAG) and RL with rubrics bring complementary benefits for legal reasoning abilities, where RAG improves overall reasoning capability, whereas RL improves correctness albeit with reduced coverage. 1 Introduction Large language models (LLMs) can solve complex reasoning problems by generating an intermediate reasoning trace (" Chain-of-thoughts ") before outputting the final answer 58 ; 12 . Evaluating the quality of these reasoning traces is crucial for understanding as well as improving the reasoning ability of LLMs, for example, through selecting the best responses or reinforcement learning ( 30 ; 60 ; 15 ; 31 ) . However, evaluating reasoning traces for expert-level tasks, e.g. , law and medicine, requires substantial domain expertise that even advanced LLMs do not yet fully possess ( 41 ; 57 ) . Moreover, automatic evaluators often lack an understanding of domain-specific and nuanced desirability shaping human judgments ( 27 ; 52 ) . For instance, when asked to predict the outcome of a court judgment, a good reasoning trace should not only avoid logical and factual errors but also exhaustively identify and analyze legal issues ( 62 ) . This underscores the need for a more sophisticated evaluation strategy for expert-level, domain-specific reasoning traces. We choose law as our testbed, as its textual and logical nature enables objective evaluation of reasoning traces compared to domains that rely on probabilistic and data-driven reasoning, e.g. , medicine and finance. Figure 1: Overview of the LEGIT dataset and task. Facts and issue trees are extracted from real-world court judgments to serve as inputs and rubrics for the LEGIT task. See Appendix A for another example. In this paper, we introduce LEGIT ( LEGal Issue Trees , Figure 1 ), a large-scale Korean legal judgment prediction (LJP) benchmark with an additional emphasis on reasoning trace evaluation . In contrast to other LJP datasets where the task is simply to predict the court’s final order given basic facts ( 7 ) , LEGIT includes additional rubrics for reasoning traces that are extracted from court judgments. These rubrics can be used to assess whether the reasoning traces fully cover the issues addressed during the trial ( issue coverage ) and if the reasoning about those issues is correct ( issue correctness ). We argue that they enable reliable LLM-as-a-judge evaluations of legal reasoning traces that are comparable to licensed lawyers and that these rubrics facilitate principled and detailed analyses of the legal reasoning ability of LLMs. Using LEGIT, we evaluate the legal reasoning ability of different LLMs, and discover that even state-of-the-art LLMs are not fully capable of complex legal judgment prediction. We identify and analyze two major error types: either failing to identify relevant issues ( decomposition error ) or failing to reason correctly about facts ( deduction error ). Finally, we explore how retrieval augmented generation (RAG) and reinforcement learning (RL) affect legal reasoning performance. These two approaches show complementary benefits: RAG improves both issue coverage and correctness, while RL maximizes correctness at the expense of issue coverage. 2 Background 2.1 Evaluating reasoning traces While reasoning traces dramatically increase LLM performance on complex tasks ( 58 ) , they might contain factual or logical errors even if the final answer is correct ( 30 ) . Such errors can be critical in high-stakes fields such as law and medicine, if end-users rely on LLM responses for important decision-making ( 41 ) . Evaluating reasoning traces can help reduce this risk by filtering out incorrect responses or training the model via reinforcement learning ( 31 ) . LLMs can be prompted to evaluate reasoning traces by finding errors or assessing the overall quality ( "LLM-as-a-judge" ) ( 60 ) . Despite its simplicity, LLM-as-a-judge achieves strong performance in error detection and best response selection in math and programming ( 18 ; 28 ) . However, using generic evaluation criteria might not be sufficient to capture deeper insights into what constitutes a good reasoning trace. Defining desirability is particularly challenging in expert domains, where one must evaluate both precise factual knowledge and deeper domain-specific insights. To incorporate diverse, fine-grained forms of desirability in reasoning traces, recent works introduced instance-specific rubrics ( 13 ; 45 ) . For instance, BigGen-Bench ( 27 ) includes handcrafted rubrics tailored for individual problems that evaluate knowledge and problem-solving strategies. While such rubrics allow LLM judges to assess diverse criteria beyond correctness with high human-LLM agreement, the human effort required to define and validate rubrics hinders scaling this approach beyond evaluation-only benchmarks. By contrast, LEGIT automatically extracts high-quality rubrics from court judgments, resulting in scalable and expert-aligned evaluation of reasoning traces. 2.2 Legal Judgment Prediction (LJP) Legal judgment prediction (LJP) is the task of predicting the court’s final order given the facts and claims of a case ( 7 ) . It is a representative legal reasoning task that simultaneously assesses an LLM’s legal knowledge (statutes, case laws) and reasoning ability (applying laws to given facts) ( 24 ; 19 ) . While traditional reasoning tasks like math often use final answer correctness as the sole criterion ( 39 ; 8 ) , issue coverage and correctness are also critical in LJP ( 62 ) . Imagine a case where the beneficiary of an insurance policy (plaintiff) sues the insurance company (defendant) for payment, and the ground-truth order is to dismiss the case (Figure 1 ). Even if two reasoning traces both predict that the case should be dismissed, if one thoroughly considers the contractual conditions of an insured accident ( e.g. , whether the accident was external) when the other does not, legal experts will find the former more useful in practice. Unfortunately, existing benchmarks account only for the final order prediction accuracy, and ignore both issue coverage and correctness ( 2 ; 20 ; 19 ; 1 ) . Current LJP benchmarks also fail to cover the full range of legal cases. As LJP was traditionally considered a classification/regression task, many works have focused on either criminal cases with a continuous spectrum of sentencing labels ( 20 ; 19 ) , or a minimal set of cases with clear binary labels ( 2 ; 1 ) . To the best of our knowledge, no LJP dataset has covered arbitrary case types observed in civil and administrative cases, which account for up to 84.1% of total

Read the complete text