A Reasoning-Focused Legal Retrieval Benchmark

Lucia Zheng, Neel Guha, Javokhir Arifov, Sarah Zhang, Michal Skreta, Christopher D. Manning, Peter Henderson, Daniel E. Ho · 2025

Read A Reasoning-Focused Legal Retrieval Benchmark by Lucia Zheng, Neel Guha, Javokhir Arifov, Sarah Zhang, Michal Skreta, Christopher D. Manning, Peter Henderson, Daniel E. Ho (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-03970.txt.

Introduction

Lucia Zheng and co-authors build in 2025 a test set for law search tools that must reason, not just match words. Law teams now pair large models with search to lift scores and grit. No true to life test set yet caught how hard real law search is.

Two tasks form the set: bar exam queries and housing law queries. Both were built by mark steps that mimic real law desk work. Top search pipes of the day were run on the set. Scores show law search with reasoning stays a hard task, which sets a clear bar for next builds.

Catalog record

CREATORLucia Zheng, Neel Guha, Javokhir Arifov, Sarah Zhang, Michal Skreta, Christopher D. Manning, Peter Henderson, Daniel E. Ho
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE88 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-03970

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Reasoning-Focused Legal Retrieval Benchmark Lucia Zheng, Neel Guha, Javokhir Arifov, Sarah Zhang, Michal Skreta, Christopher D. Manning, Peter Henderson, Daniel E. Ho Abstract . As the legal community increasingly examines the use of large language models (LLMs) for various legal applications, legal AI developers have turned to retrieval-augmented LLMs (“RAG” systems) to improve system performance and robustness. An obstacle to the development of specialized RAG systems is the lack of realistic legal RAG benchmarks which capture the complexity of both legal retrieval and downstream legal question-answering. To address this, we introduce two novel legal RAG benchmarks: Bar Exam QA and Housing Statute QA. Our tasks correspond to real-world legal research tasks, and were produced through annotation processes which resemble legal research. We describe the construction of these benchmarks and the performance of existing retriever pipelines. Our results suggest that legal RAG remains a challenging application, thus motivating future research. DOI: 10.1145/3709025.3712219 Conference: 4th ACM Symposium on Computer Science and Law; March 25-27, 2025; Munich, Germany ISBN: 979-8-4007-1421-4/2025/03 CCS: Computing methodologies Natural language processing CCS: Applied computing Law 1. Introduction There is significant excitement towards using large language model (LLM) tools to improve the quality and cost of legal services ( Ma et al., 2024 ) . Already, lawyers across the world have begun incorporating LLMs into legal practice, and applying them towards a range of tasks: answering questions about the law in various jurisdictions, identifying potential legal issues in client cases, drafting agreements, and more ( Bommasani et al., 2021 ) . Applying LLMs towards legal tasks requires resolving distinctive challenges posed by the legal domain. For instance, legal tasks are often fact-intensive ( Guha et al., 2024 ) and LLMs have a tendency to produce factually-ungrounded statements (“hallucinations”) ( Dahl et al., 2024 ) . The law is also constantly changing—through new statues or judicial opinions—and the knowledge contained in LLM parameters is static ( Lazaridou et al., 2021 ) . To address these challenges, legal AI developers have begun deploying “RAG” systems, where LLMs are augmented with retrievers over corpora of case law, statutes, and other legal documents ( Magesh et al., 2024 ) . Given a lawyer query, the retriever fetches documents from the corpora relevant to the query, and requires the LLM to answer the query with respect to the retrieved documents. However, a significant bottleneck in the development of legal RAG systems is the lack of realistic English legal open-domain question answer benchmarks. In particular, existing benchmarks suffer from one or more of the following weaknesses. (1) First, they fail to represent tasks where where the query and relevant document have little lexical overlap, and identifying the relevant document requires multi-hop or analogical reasoning. In practice this setting is ubiquitous. Producing the legal cases relevant to a client’s factual circumstances, for instance, requires extracting higher-order legal issues and identifying other cases which present those issues—even if the specific factual descriptions are quite different. (2) Second, existing benchmarks are often exclusively retrieval benchmarks, and do not contain paired question-answers to evaluate downstream reasoning based on the retrieved information ( Goebel et al., 2024 ; Hou et al., 2024 ; T.y.s.s. et al., 2024 ; Mahari et al., 2023 ; Chalkidis et al., 2021 ) . As a result, they do not capture the downstream impacts of improvements in retrievers. (3) Finally, benchmarks rely on query-document distributions extracted from datasets built for other purposes, where queries do not correspond to the types of questions lawyers might actually ask ( Saad-Falcon et al., 2024 ; Hou et al., 2024 ; Goebel et al., 2024 ) . To address this gap, we introduce two new benchmark datasets for evaluating retrieval-augmented LLMs: Bar Exam QA and Housing Statute QA. These datasets address the deficiencies discussed above. In Bar Exam QA, queries correspond to reasoning-intensive Bar Exam hypothetical fact-patterns, and documents correspond to judicial opinion passages necessary for answering the hypothetical. In Housing Statute QA, queries correspond to practically useful questions about housing law, and documents correspond to statutes from different jurisdictions. Our datasets provide ~10K labeled, paired query, gold passage, answer examples for training and evaluating language models on legal retrieval and retrieval-augmented downstream QA tasks. The gold passages for these datasets are hand-annotated and validated by law students and researchers, through annotation processes modeled off of a lawyer’s legal research process. Concretely, our work makes three contributions. First, we describe the construction of these datasets (Section 3 ) and compare them to existing benchmarks (Section 4 ). We show that relative to existing benchmarks, ours captures query-document distributions where the lexical similarity between the query and document is low. Second, we benchmark existing state-of-the-art retrieval pipelines on these datasets and find that because of the low lexical similarity, common retrieval methods like BM25 struggle (Section 5 ). Third, we present results of a simple heuristic to verify that our benchmarks accurately measure improved legal reasoning in retrieval (Section 5.3 ). Specifically, we describe a law-inspired query expansion strategy with generative reasoning roll-outs. We find that this approach improves performance on our datasets. Our work suggests that developers of retrieval-augmented legal LLM products may need to go further than simple retrievers to improve the performance of their approaches. In particular, they may need to ensure that retrievers can also be legal reasoners too, either through query expansion or increased embedding model capacities. 2. Related Works 2.1. Open-Domain QA Datasets BEIR ( Thakur et al., 2021 ) is a widely used information retrieval (IR) benchmark, which consists of 18 datasets across 9 task types. Our datasets are most similar to the question answering tasks in BEIR, Natural Questions (NQ) ( Kwiatkowski et al., 2019 ) , HotpotQA ( Yang et al., 2018 ) , and FiQA-2018 ( Maia et al., 2018 ) . For this style of task, the retrieved passage is used as context to help the model on downstream question answering. For NQ and HotpotQA, the answer is an extractive span of the context passage. For Bar Exam QA and Housing Statute QA, the answer is in a multiple-choice format. Past works have also discussed several limitations of existing general IR benchmarks: a skew towards web/search-engine style retrieval tasks, tasks with low lexical and syntactic variance between queries and gold passages, tasks with short query lengths, and propose new datasets to address these challenges ( Joshi et al., 2017 ; Thakur et al., 2021 ) . Most similar to our tasks, BIRCO ( Wang et al., 2024 ) and BRIGHT ( Su et al., 2024 ) introduce new benchmark IR tasks with more complex task objectives and that require greater reasoning capabilities to solve. Compared to BEIR ( Thakur et al., 2021 ) , these tasks have significantly lower lexical similarities between queries and gold passages and longer query lengths. BIRCO and BRIGHT include reasoning tasks in the natural sciences, computer science, and theorem-based mathematics, but neither contains legal reasoning tasks that share similar types of deductive reasoning processes to mathematical reasoning tasks. 2.2. Legal Information Retrieval Datasets Early work on retrieval of statutory law focus on building systems using lexical matching and extensive annotation of semantic features ( Grabmair et al., 2015 ; Šavelka and Ashley, 2022

Read the complete text