Not ready for the bench: LLM legal interpretation is unstable and out of step with human judgments

Abhishek Purushothama, Junghyun Min, Brandon Waldon, Nathan Schneider · 2025

Read Not ready for the bench: LLM legal interpretation is unstable and out of step with human judgments by Abhishek Purushothama, Junghyun Min, Brandon Waldon, Nathan Schneider (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-25356.txt.

Introduction

Abhishek Purushothama, Junghyun Min, Brandon Waldon, and Nathan Schneider examine legal meaning. Their 2025 paper questions language models in court work. United States judges often ask about ordinary meaning. That means how ordinary speakers understand legal text. Scholars recently proposed using large language models there. Models might reveal how people normally use words.

The authors argue models need rigorous evaluation first. Pretraining on huge data may not be enough. Their controlled experiments found troubling robustness failures. Slight changes in question format altered conclusions wildly. Interested parties could exploit that vulnerability in litigation. They compared model answers with human judgment data. Models correlated only moderately with human responses there. That level falls short given legal stakes. Criminal law and civil rights cases raise concerns. Models should not be treated as authoritative, they conclude.

Catalog record

CREATORAbhishek Purushothama, Junghyun Min, Brandon Waldon, Nathan Schneider
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-25356

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Not ready for the bench: LLM legal interpretation is unstable and out of step with human judgments Abhishek Purushothama, Junghyun Min, Brandon Waldon, Nathan Schneider Abstract Legal interpretation frequently involves assessing how a legal text, as understood by an ‘ordinary’ speaker of the language, applies to the set of facts characterizing a legal dispute in the U.S. judicial system. Recent scholarship has proposed that legal practitioners add large language models (LLMs) to their interpretive toolkit. This work offers an empirical argument against LLM interpretation as recently practiced by legal scholars and federal judges . Our investigation in English shows that models do not provide stable interpretive judgments: varying the question format can lead the model to wildly different conclusions. Moreover, the models show weak to moderate correlation with human judgment, with large variance across model and question variant, suggesting that it is dangerous to give much credence to the conclusions produced by generative AI. We publicly release our code to facilitate future work 1 1 1 https://github.com/bwaldon/llms-legal-interp . 1 Introduction Legal decisions often come down to the interpretation of written text (e.g., in a statute or contract). Often, legal interpretation is straightforward. At times, however, the text is imprecise or ambiguous, leading to disputes about how to apply a written provision to a particular set of circumstances. In the U.S. legal system, judges often place considerable weight on what they deem the ‘ordinary meaning’ (meaning as would be understood by ordinary speakers of American English) of a legal term ( Slocum 2015 ) . 2 2 2 In this paper, we are officially agnostic as to whether ‘ordinary meaning’ is a useful legal analytical construct (or even a coherent linguistic concept). Our focus is on whether LLMs are useful for pursuing interpretative methodologies that are purportedly grounded in ‘ordinary meaning.’ One recent case ( Snell v. United Specialty Insurance Co. ) required interpretation of the term ‘landscaping’ concerning an in-ground trampoline (see fig. 1 ). A second case ( U.S. v. Deleon ) turned on the interpretation of the phrase ‘physically restrained’ and whether the phrase describes indirectly restricting movement by threatening someone with a gun. How can a judge ascertain ‘ordinary meaning’ of a legal text? Often, the judge will deploy armchair intuition buttressed by hypotheticals and dictionaries ( Krishnakumar 2024 ) . On occasion, judges reference corpora Solan and Gales 2017 ; Gries and Slocum 2017 and surveys Tobia 2024 . John is a contractor with insurance that covers property loss, damage, or personal injury claims that arise due to his ‘landscaping’ work. John is employed by a family, the Smiths, to install an in-ground trampoline in the family’s backyard. A few years after John completes the project, the Smiths successfully sue John for injuries that their daughter sustained while playing on the trampoline. John files a claim with his insurance company to recover losses incurred from the lawsuit. Considering just how “landscaping” would be understood by ordinary speakers of English, is John covered by the insurance—yes or no? Figure 1: A legal interpretation scenario represented as QA task with binary questions. The example is based on the case Snell v. United Specialty Insurance Co. and constructed in the style of our task. Enter LLMs. The advent of LLMs has produced considerable excitement in the legal field, and textual interpretation is no exception. Hoffman and Arbel 2024 opined that LLMs are good resources for ascertaining ordinary meaning. Federal judicial opinions ( Newsom 2024a ; Newsom 2024b ) have included ‘direct queries’ to LLMs on the ordinary meaning of ‘landscaping’ and ‘physically restrained’ 3 3 3 In Snell and Deleon , Judge Kevin Newsom posed the two following direct queries, respectively: “ What is the ordinary meaning of landscaping? ” and “ What is the ordinary meaning of physically restrained? ”. See Waldon et al. 2025 for a detailed discussion of the direct query method. on the grounds that these models supposedly capture and reflect patterns of ordinary language use. These developments have been met with critical scholarship assessing the benefits, risks, and impact of these technologies from the standpoint of both legal theory and practice. Waldon et al. 2025 and Choi 2025 demonstrated that LLMs’ interpretive judgments are highly sensitive to how queries about the meaning are phrased and suggest that LLMs are not reliable tools for legal interpretation; however, the studies were limited in scope, testing only a few models and interpretive questions 4 4 4 Waldon et al. 2025 utilized two scenarios, and Choi 2025 utilized five. . In this work, we investigate the stability of LLM-based legal interpretation methods, with a focus on LLMs’ ability to reflect ‘ordinary meaning’ as understood by human native speakers. In light of recent research ( section 2 ), our hypotheses are as follows: 1. LLM judgments are highly sensitive to subtle manipulations in how ordinary meaning queries are posed to the model. 2. LLM judgments are poorly correlated to human judgment. To test our hypotheses, we create queries based on a previously developed set of 138 scenarios that assess linguistic interpretation in a variety of hypothetical insurance contract disputes Waldon et al. 2023 , and query 14 open-weight models (and one closed-weight model) across 9 systematic question variants ( section 3 ). In line with our hypotheses, we find in section 4 that LLM judgments are inconsistent across model family and size, as well as across subtle choices in question phrasing. The outputs of some instruction-tuned models and some model scales are correlated to human judgment only in some question variants; moreover, neither decoded tokens nor probability distributions offer a reliable source of human-like ‘ordinary meaning’ judgment. Though LLMs possess undeniably fluent generation capabilities, our results add to a growing chorus of skepticism about LLMs as tools for legal interpretation ( section 5 ). 2 Background In the current era of generative AI enthusiasm, some scholars have expressed optimism that LLMs will revolutionize legal interpretation. Multiple variations of this idea have been put forward. One argument is that LLMs need not be perfectly accurate, but as long as they are ‘good enough’, they will produce interpretive outcomes that are more consistent and more affordable to the masses than the legal status quo ( Hoffman and Arbel 2024 , writing about contract interpretation) . This, of course, presumes that LLM judgments are consistent. Hoffman and Arbel 2024 offer a small-scale case study of robustness with 5 scenarios and 20 machine-generated paraphrases of each prompt. This study was expanded by Choi 2025 , who generated 2000 paraphrases for each of the 5 scenarios, and found models to be unreliable with prompt variation, in ways that differ across models, training methods, and output processing methods. While this result is suggestive, we will take the study of prompt variation a step further, with more rigorous testing that covers more scenarios and uses tightly controlled prompt variants (instead of LLM paraphrasing) to ensure meaning preservation. A second line of reasoning holds that for difficult interpretive problems, for investigating ordinary meaning in English, LLMs are potentially better than humans because they have been trained on so much ordinary English data (we can call this the ‘omniscience’ argument). Judge Kevin Newsom in his opinion in the

Read the complete text