CALRK-Bench: Evaluating Context-Aware Legal Reasoning in Korean Law

JiHyeok Jung, Taeyoung Yoon, Hyunsouk Cho [email protected]@[email protected] · 2026

Read CALRK-Bench: Evaluating Context-Aware Legal Reasoning in Korean Law by JiHyeok Jung, Taeyoung Yoon, Hyunsouk Cho [email protected]@[email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-26332.txt.

Introduction

Jung, Yoon, and Cho present CALRK-Bench, a test of legal reasoning built on Korean law. Legal judgment needs more than rule use; the setting around a rule matters too. The bench checks whether models can spot when a norm holds in time, judge if case facts suffice, and explain shifts in verdicts. Its cases come from precedents and consultation records, checked by legal experts. Even recent large models score low on all three tasks.

Catalog record

CREATORJiHyeok Jung, Taeyoung Yoon, Hyunsouk Cho [email protected]@[email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49647
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.26332

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CALRK-Bench: Evaluating Context-Aware Legal Reasoning in Korean Law JiHyeok Jung, Taeyoung Yoon, Hyunsouk Cho [email protected]@[email protected] Abstract Legal reasoning requires not only the application of legal rules but also an understanding of the context in which those rules operate. However, existing legal benchmarks primarily evaluate rule application under the assumption of fixed norms, and thus fail to capture situations where legal judgments shift or where multiple norms interact. In this work, we propose CALRK-Bench, a context-aware legal reasoning benchmark based on the legal system in Korean. CALRK-Bench evaluates whether models can identify the temporal validity of legal norms, determine whether sufficient legal information is available for a given case, and understand the reasons behind shifts in legal judgments. The dataset is constructed from legal precedents and legal consultation records, and is validated by legal experts. Experimental results show that even recent large language models consistently exhibit low performance on these three tasks. CALRK-Bench provides a new stress test for evaluating context-aware legal reasoning rather than simple memorization of legal knowledge. Our code is available at https://github.com/jhCOR/CALRKBench . 1 Introduction Legal reasoning is the process of deriving judicial conclusions by interpreting and applying legal norms based on established facts 9 . With the advancement of Large Language Models (LLMs), such reasoning structures have increasingly been utilized in various legal tasks such as legal judgment prediction, statutory interpretation, and case analysis 2 ; 9 ; 3 ; 20 . LLMs demonstrate a certain level of competence in rule-application-oriented tasks when both the factual circumstances and the applicable legal provisions are explicitly provided or assumed to be fixed 1 . Figure 1: Overview of the Benchmark. Many legal benchmarks have already evaluated tasks such as statutory interpretation, legal question answering, and judgment prediction. Although they take different forms on the surface, most studies formalize legal reasoning as the process of applying a given rule to given facts, which reflects the IRAC (Issue–Rule–Application–Conclusion) structure well 19 . In particular, previous studies have mainly focused on knowledge recall ability 10 ; 17 or the reproduction of judgment outcomes 14 . However, such studies cannot reflect situations in which laws are amended or repealed, or situations in which the governing norm remains the same but judicial decisions change. These changes are typically treated as contextual factors because they are not directly observable from the immediate facts of a case. These concepts are also well explained in Hart’s classical legal theory 12 . Hart defined law not as a mere collection of rules governing conduct, but as a combination of primary rules governing conduct and secondary rules governing the validity and change of rules. In particular, the rule of recognition and the rule of change determine which norms constitute valid law and when legal norms are created or modified. These systemic characteristics show that legal judgment is not merely the application of rules but a contextual determination that considers which norms are valid at a given time and how their legal effects interact. To address this gap, we propose CALRK-Bench (Context-Aware Legal Reasoning in Korean Law) , a benchmark that conceptualizes legal judgment as a multi-stage process prior to rule application and evaluates whether models can (i) identify legally meaningful changes ( Type-JSA ), (ii) reason about temporal validity and scope of application ( Type-TCR ), and (iii) assess the sufficiency of legal information ( Type-ISR ) prior to judgment. However, these capabilities have rarely been evaluated in IRAC-centered benchmarks. To enable reliable evaluation of these capabilities, the benchmark is constructed based on the Korean legal system, which provides practical advantages for such analysis. As a civil law system, Korean legal norms are primarily articulated in written statutes, making the temporal validity and applicability of norms relatively clear and amenable to rule-based modeling. By contrast, common-law systems rely more heavily on precedent-based doctrinal development, where legal change often emerges gradually through judicial interpretation, making doctrinal shifts harder to formalize into precise evaluation criteria. CALRK consists of 100, 279, and 100 stress-test questions for Type-TCR , Type-ISR , and Type-JSA , respectively. Considering the inherent rarity of clearly classifiable doctrinal change cases, the benchmark prioritizes rigor over scale and focuses on collecting high-quality instances grounded in real legal sources. The dataset is constructed from legal precedents and consultation records, with carefully designed problem instances that reflect realistic legal reasoning scenarios. In particular, for information sufficiency tasks, we employ a hard negative mining procedure to construct legally plausible but misleading distractors based on co-citation patterns in precedents and conflicting legal conclusions. Such co-cited provisions reflect real legal scenarios in which competing arguments rely on different statutory grounds for the same case, requiring the model to determine which legal basis is actually governing and whether the provided information is sufficient for reaching a judgment. The resulting dataset, while emphasizing quality over scale, is comparable in size to prior legal benchmarks such as GoldCoin 8 and the NitiBench-Tax subset of NitiBench 2 . Experimental results show that even state-of-the-art LLMs exhibit systematic failures across all three tasks. Models struggle to reason about the temporal validity of norms, fail to recognize when available information is insufficient, and show biases in attributing the causes of judgment shifts. These findings highlight fundamental limitations of current LLMs in context-aware legal reasoning, which CALRK-Bench is designed to evaluate. 2 Related Work 2.1 General Legal Benchmarks The expansion of Large Language Models (LLMs) into the legal domain has produced numerous evaluation benchmarks and datasets. Representative efforts such as MMLU, LexGLUE, LegalBench, and LEXAM 6 ; 21 ; 13 ; 9 cover tasks including statute interpretation, legal QA, document classification, and judgment prediction, establishing the foundation for systematic legal reasoning evaluation. Despite differences in scope, most assume a fixed and predetermined governing rule rooted in the IRAC (Issue–Rule–Application–Conclusion) framework. Evaluation, therefore, measures rule application accuracy rather than whether the normative framework itself remains valid or applicable. 2.2 Temporal Context Benchmarks Several studies have examined how large language models handle the temporal aspects of legal information. LawShift 11 shows that LLMs often fail to properly adjust their judgments when legal norms change due to statutory revisions. Other studies address temporality more indirectly. For example, LexTime 4 analyzes the temporal ordering of events in legal texts, while MultiEURLEX 5 highlights temporal concept drift in the distribution of legal labels. 2.3 Sufficient Context Benchmarks In real-world legal reasoning, various legal and factual information may be involved, and in such situations, it may be necessary to recognize that additional information is required before reaching a conclusion. Some studies have addressed this issue indirectly. For example, previous work analyzes split-vote decisions of the European Court of Human Rights (ECHR) and measures disagreement among judges and case difficulty, evaluating how well models recognize such uncertainty 25 . However, these studies focus on uncertainty awareness or calibration rather than directly evaluating whether the provided legal information is sufficient for making a judgment. 2.4 Shift Context Benchmarks Recent studies have also explored disagreement and change in legal judgments. For example, previous work investigates whether LLMs can identify overruling relationships between judicial precedents in long legal documents 26 . The results show that even when the overruling is explicitly stated, models often rely on shallow heuristics and struggle

Read the complete text