Do LLMs Truly “Understand” When a Precedent Is Overruled?

Li Zhang \orcid 0000-0003-0375-1793, Jaromir Savelka \orcid 0000-0002-3674-5456, Kevin Ashley \orcid 0000-0002-3674-5456 · 2025

Read Do LLMs Truly “Understand” When a Precedent Is Overruled? by Li Zhang \orcid 0000-0003-0375-1793, Jaromir Savelka \orcid 0000-0002-3674-5456, Kevin Ashley \orcid 0000-0002-3674-5456 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-20941.txt.

Description

Li Zhang leads two co-authors on whether large language models truly grasp when a legal precedent has been overruled, a core skill in common law reasoning. Long context models promise help with real opinions, yet most tests use simple synthetic tasks far from courtroom documents.

Overruling links are common inside judicial opinions and carry high stakes, so they form a focused test of long document understanding. The study probes whether models track which rulings still stand after later courts reverse them across a full text. Weak tracking would warn lawyers that fluent summaries can hide dead law presented as live authority.

Catalog record

CREATORLi Zhang \orcid 0000-0003-0375-1793, Jaromir Savelka \orcid 0000-0002-3674-5456, Kevin Ashley \orcid 0000-0002-3674-5456
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-20941

Browse the full library · About the Tower of Babel