AnnoCaseLaw: Legal Annotation Benchmark of Civil Negligence
Rilton Franzone, Magnus Sesodia, Alina Petrova, John Armour, Thomas Lukasiewicz, Oana-Maria Camburu, Puneet K. Dokania, Philip Torr, Christian Schroeder de Witt, Fabio J. Fehr · 2025
Read AnnoCaseLaw: Legal Annotation Benchmark of Civil Negligence by Rilton Franzone, Magnus Sesodia, Alina Petrova, John Armour, Thomas Lukasiewicz, Oana-Maria Camburu, Puneet K. Dokania, Philip Torr, Christian Schroeder de Witt, Fabio J. Fehr (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-00128.txt.
Introduction
Rilton Franzone and co-authors built a benchmark for legal annotation in two thousand twenty-five. The task is marking parts of court opinions in United States civil negligence cases. Experts must tell apart the reasoning that supports a ruling from other legal talk in the same text.
The benchmark holds four hundred seventy-one opinions with more than twelve thousand expert highlights in five classes. The team tested large, small, and law-focused language models against those marks. Models did best on facts and outcomes, and worst on applying law to facts. Top paid models led there, while one open model matched them at far lower cost.
Catalog record
| CREATOR | Rilton Franzone, Magnus Sesodia, Alina Petrova, John Armour, Thomas Lukasiewicz, Oana-Maria Camburu, Puneet K. Dokania, Philip Torr, Christian Schroeder de Witt, Fabio J. Fehr |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-00128
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AnnoCaseLaw: Legal Annotation Benchmark of Civil Negligence Rilton Franzone, Magnus Sesodia, Alina Petrova, John Armour, Thomas Lukasiewicz, Oana-Maria Camburu, Puneet K. Dokania, Philip Torr, Christian Schroeder de Witt, Fabio J. Fehr Abstract Language models offer a way to scale costly, labour-intensive expert annotation, but their ability to reproduce expert judgments remains uncertain. This is particularly challenging in law, where annotators must distinguish the reasoning supporting judicial decisions from other legally relevant discussion. We present AnnoCaseLaw 1 1 1 Benchmark Code as a densely annotated benchmark comprising 471 471 U.S. civil-negligence court opinions with more than 12,000 12,000 expert highlights across five annotation categories. We evaluate proprietary frontier, open-weight, legal-specialised, and decision models on their ability to reproduce these annotations. Models perform better on Facts and Outcome than on Application of Law to Facts, Procedural History, and Relevant Precedents. On Application of Law to Facts, proprietary frontier models achieve the strongest balance of reasoning coverage and precision, while the open-weight model Gemma 4 achieves comparable Macro-F1 to proprietary frontier models at a fraction of the estimated inference cost. Our findings highlight the importance of evaluating annotation coverage and cost alongside aggregate scores when assessing the feasibility of automated legal annotation. † † footnotetext: ∗ Equal first authorship; † equal senior authorship. 1 Introduction Figure 1: AnnoCaseLaw expert annotations. Five categories illustrated in St. Mary’s Hospital v. Bynum (1978). Sentence numbers refer to the full opinion; bracketed ellipses mark omitted text. Unhighlighted text was not annotated. High-quality data annotation is essential for developing and evaluating AI systems, yet remains costly and labour-intensive, particularly in specialised domains. Language models offer a way to scale annotation, with recent studies demonstrating potential cost and efficiency advantages over human annotation ( Tan et al., 2024 ; Parfenova et al., 2025 ; Gu et al., 2026 ) . However, models still struggle with legal annotations requiring nuanced understanding, such as identifying complex legal references ( Schepers et al., 2025 ) . Expert legal annotations remain scarce, and some existing datasets rely on keyword-based annotation methods ( Huang et al., 2024 ; Huang, 2025 ) . This raises the question: can language models reliably reproduce expert legal annotations? Identifying the reasoning supporting a judicial decision is particularly demanding, as models may confuse legally worded passages with those that actually support an outcome ( Kalamkar et al., 2022 ; Cheng et al., 2026 ) . Civil negligence provides a useful setting for studying this problem, as cases with different factual circumstances share a common legal framework, including duty of care and breach. Appellate courts are particularly suitable for studying legal reasoning because they review lower-court decisions against an established factual record, drawing on legal arguments and precedent. Their opinions provide a rich source of evidence on how legal principles are applied to facts. We present AnnoCaseLaw (Figure 1 ), a benchmark comprising 471 U.S. civil-negligence court opinions with more than 12,000 expert-annotated highlights. These highlights identify legally significant passages across five categories: Application of Law to Facts (ALF), Procedural History, Relevant Precedents, Facts, and Outcome. Related benchmarks are discussed in Appendix A . Our main contributions are: • Expert-annotated legal benchmark. We present AnnoCaseLaw, a densely annotated benchmark capturing legally significant passages, including the application of law to facts and relevant precedents, for systematic evaluation of legal annotation reproduction (Section 2 ). • Comprehensive model evaluation. We compare proprietary frontier, open-weight, legal-specialised, and decision models across five annotation categories. Models perform better on Facts and Outcome than on more nuanced categories, including ALF, Procedural History, and Relevant Precedents (Table 1 ); even frontier models miss about a quarter of expert-marked ALF passages (Figure 2 ). • Annotation quality and cost trade-offs. We evaluate annotation performance alongside estimated inference cost. Proprietary frontier models achieve the strongest balance of precision and recall on ALF, while Gemma 4 achieves comparable Macro-F1 to proprietary frontier models at a fraction of the estimated inference cost (Figures 2 and 3 ). 2 AnnoCaseLaw Benchmark Version note. An earlier version presented AnnoCaseLaw ( Sesodia et al., 2025 ) primarily in the context of legal judgment prediction, with automated case annotation as one of three tasks. This version reframes the work around reproducing expert legal annotations, retaining the underlying corpus and expert annotations while introducing a revised task formulation and new model evaluations and analyses. 2.1 Dataset Construction The court opinions were retrieved from the Caselaw Access Project ( Harvard Law School Library, 2018 ) . The dataset includes 471 U.S. civil-negligence cases from three states: Arkansas, Illinois, and New Mexico. Cases were identified from the repository and checked for relevance by the annotators. It includes decisions from state appeals courts, state supreme courts, and the Illinois Claims Court, a specialised tribunal handling negligence claims against the state. Further details on jurisdictional coverage, opinion lengths, and annotation statistics are provided in Appendix B . 2.2 Expert Annotation Protocol The annotations were produced by three legally trained annotators (final-year law students) under the supervision of an experienced legal scholar, following a shared annotation protocol. Each opinion was annotated by one student. The full annotation instructions are provided in Appendix E.1 . The protocol distinguishes five categories of passages in judicial opinions, illustrated in Figure 1 . Application of Law to Facts (ALF) , Procedural History (PH) , and Relevant Precedents capture specific aspects of the court’s legal analysis, covering only passages concerning duty of care, breach of duty, and contributory negligence, three central issues in civil negligence law. ALF identifies passages explaining how the court applies the law to the facts, selecting only the reasoning necessary to explain its decision. PH records the proceedings, including earlier courts’ decisions and reasons, and the standard the deciding court uses to review them. Relevant Precedents identifies citations to earlier cases and other legal sources directly relevant to the court’s reasoning, excluding those used only as background; only cited cases are evaluated in this category (Section 2.3 ). Facts captures events and circumstances before litigation, including facts the court accepts or assumes to be true. Facts are marked comprehensively, including within the court’s reasoning. Outcome identifies passages stating the present court’s disposition, such as affirming, reversing, remanding, or dismissing. Group Model ALF PH Precedents Facts Outcome Macro-F1 Proprietary frontier Claude Opus 5 54.3 \pm 2.1 53.1 \pm 2.1 62.8 \pm 2.8 79.2 \pm 1.6 77.1 \pm 1.9 65.3 \pm 1.4 GPT-5.6 Sol 55.1 \pm 2.2 40.8 \pm 2.0 61.7 \pm 2.9 71.0 \pm 1.7 72.9 \pm 2.2 60.3 \pm 1.4 Open, > 1T Qwen3.8 Max 2.4T/95B 52.5 \pm 2.1 48.0 \pm 2.1 58.0 \pm 3.0 69.3 \pm 1.7 77.2 \pm 2.1 61.0 \pm 1.3 Kimi K3 2.8T 42.5 \pm 1.9 46.1 \pm 2.1 61.9 \pm 2.8 73.8 \pm 1.8 69.5 \pm 2.2 58.8 \pm 1.4 DeepSeek V4 Pro 1.6T/49B 38.4 \pm 1.9 41.0 \pm 2.3 54.8 \pm 3.0 69.7 \pm 2.0 79.3 \pm 2.1 56.6 \pm 1.4 Open, 31–54B Gemma 4 31B 50.7 \pm 2.0 56.9 \pm 2.2 60.6 \pm 2.9 74.8 \pm 1.7 79.9 \pm 1.9 64.6 \pm 1.2 Thomson 1.0 Small † 35B/3B 41.0 \pm 2.0 51.7 \pm 2.7 58.5 \pm 3.3 74.9 \pm 1.7 66.5 \pm 2.7 58.5 \pm 1.5 Snowdon 1.1 Small 35B/3B 33.7 \pm 1.8 49.5 \pm