LegalCiteBench: Evaluating Citation Reliability in Legal Language Models

Sijia Chen, Hang Yin, Shunfan Zhou · 2026

Read LegalCiteBench: Evaluating Citation Reliability in Legal Language Models by Sijia Chen, Hang Yin, Shunfan Zhou (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-10186.txt.

Description

Chen, Yin and Zhou introduce LegalCiteBench, a benchmark that tests whether legal language models cite real cases correctly. Incorrect citations and invented precedents can cause serious professional harm, yet existing legal benchmarks stress statutory reasoning and question answering rather than citation itself. The central failure mode is closed-book: asked for authorities without outside sources, models return plausible-looking but wrong citations.

Built from one thousand real United States judicial opinions held by the Case Law Access Project, the benchmark holds about twenty-four thousand instances. It spans five tasks: retrieval, completion, error detection, case matching, and verification with correction. Across twenty-one models, exact citation recovery stays brutally hard, with even the best below seven out of one hundred on retrieval and completion. Misleading answer rates exceed ninety-four percent for twenty of twenty-one models on retrieval-heavy tasks, and scale plus legal pretraining barely help.

Catalog record

CREATORSijia Chen, Hang Yin, Shunfan Zhou
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE71 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.10186

Browse the full library · About the Tower of Babel