Legal-DC: Benchmarking Retrieval-Augmented Generation for Legal Documents
Yaocong Li, Qiang Lan, Leihan Zhang, Le Zhang · 2026
Read Legal-DC: Benchmarking Retrieval-Augmented Generation for Legal Documents by Yaocong Li, Qiang Lan, Leihan Zhang, Le Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-11772.txt.
Introduction
Li and co-authors build Legal-DC, a benchmark for retrieval-augmented generation on Chinese legal texts. It holds 480 legal documents and 2,475 question-answer pairs, each tied to clause-level references. They also present LegRAG, which pairs clause-boundary indexing with a dual-path self-reflection step to keep clauses whole and answers sharp. Automated LLM scoring rounds out the work. LegRAG beats prior best methods by 1.3% to 5.6% on key metrics, the authors report.
Catalog record
| CREATOR | Yaocong Li, Qiang Lan, Leihan Zhang, Le Zhang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48192 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.11772
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Legal-DC: Benchmarking Retrieval-Augmented Generation for Legal Documents Yaocong Li, Qiang Lan, Leihan Zhang, Le Zhang Abstract Retrieval-Augmented Generation (RAG) has emerged as a promising technology for legal document consultation, yet its application in Chinese legal scenarios faces two key limitations: existing benchmarks lack specialized support for joint retriever-generator evaluation, and mainstream RAG systems often fail to accommodate the structured nature of legal provisions. To address these gaps, this study advances two core contributions: First, we constructed the Legal-DC benchmark dataset, comprising 480 legal documents (covering areas such as market regulation and contract management) and 2,475 refined question-answer pairs—each annotated with clause-level references—filling the gap for specialized evaluation resources in Chinese legal RAG. Second, we propose the LegRAG framework, which integrates legal adaptive indexing (clause-boundary segmentation) with a dual-path self-reflection mechanism to ensure clause integrity while enhancing answer accuracy. Third, we introduce automated evaluation methods for large language models to meet the high-reliability demands of legal retrieval scenarios. LegRAG outperforms existing state-of-the-art methods by 1.3% to 5.6% across key evaluation metrics. This research provides a specialized benchmark, practical framework, and empirical insights to advance the development of Chinese legal RAG systems. Our code and data are available at https://github.com/legal-dc/Legal-DC . a School of Economics and Management, Beijing University of Posts and Telecommunications, No. 10 Xitucheng Road, Haidian District, Beijing, 100876, China b College of Computing, Beijing Information Science and Technology University, No. 12 Xiaoying East Road, Haidian District, Beijing, 100085, China † These authors contributed equally to this work ∗ Corresponding author: [email protected] Keywords: Retrieval-Augmented Generation, Benchmarking, Legal Document Consultation, Triple Retrieval 1 Introduction Large Language Models (LLMs) have demonstrated exceptional capabilities in dealing with professional document Q&A problems in medicine, finance, and law. Individuals can use domain-specific corpora to fine-tune the models and enhance their vertical domain expertise. However, LLMs still face issues like hallucinations [ 10 ] and outdated corpora. The Retrieval-Augmented Generation (RAG) approach [ 12 ] , which retrieves and integrates relevant information from reliable external sources, effectively mitigates the issues. Consequently, the application of RAG has rapidly evolved in areas like medicine [ 11 ] , finance [ 23 ] , and law [ 4 ] . A complete RAG consists of a corpus, a retriever, and a generator, with more sophisticated retrievers including components such as embedding models and reranking models [ 6 ] . In the RAG process, retrieved documents are crucial for accurate answers, especially in professional fields. Poor quality documents hinder LLMs from providing satisfactory responses. The generator and retriever together determine the quality of the final answer. In legal document consultation, large language models (LLMs) must strictly adhere to statutory provisions when answering questions, which highlights the fundamental role of accurately retrieving relevant legal documents and contents in retrieval-augmented generation (RAG) systems. However, existing legal RAG benchmarks mainly focus on answer generation and lack dedicated evaluation of retrievers, largely due to the absence of fine-grained annotations that explicitly link questions to their corresponding reference contents within legal documents. To address this limitation, we construct a benchmark dataset for legal document consultation in the context of market business activities in China, named Legal-DC, which enables joint evaluation of both retrievers and generators in RAG systems. Legal-DC consists of 2,475 question–answer–document entries derived from 480 legal documents, including laws, administrative regulations, and departmental rules governing market entities in China, with two rounds of manual annotation conducted to ensure annotation reliability. Furthermore, we propose a simple yet effective benchmark framework based on a classic RAG architecture that adopts a parallel document processing strategy integrating chunk-based semantic segmentation and article-level structural segmentation, allowing legal documents to be represented and retrieved at multiple granularities simultaneously. Experimental results on Legal-DC demonstrate that this parallel chunk–article processing strategy, together with reranking models, significantly improves the performance of legal RAG systems, while also revealing that LLMs are sensitive to noisy retrieved documents, which may lead to incorrect legal references in generated answers. In summary, our contributions are as follows: (1) We construct a benchmark dataset (Legal-DC) for comprehensively evaluating the retrievers and generators of Chinese legal RAG systems, and develop a toolkit that facilitates the combination and testing of different components (including retrievers and generators) to aid in searching for the optimal combination. (2) We propose a concise and efficient benchmark methodology for legal document consultation RAG, demonstrating superior capabilities compared to existing mainstream RAG systems and platforms. This includes approaches such as entity retrieval, reflection, and multi-way recall, whose experimental results provide crucial insights for enhancing RAG system performance. 2 Related Work 2.1 Retrieval-Augmented Generation RAG [ 12 ] is a technology that combines information retrieval with generative artificial intelligence. It retrieves relevant information from an extensive document database before generating the content. The information is used to guide the generation process and enhance the accuracy and relevance of the answer. RAG effectively mitigates the hallucination issues that may arise in LLMs when generating content, accelerates the speed of knowledge updates, and strengthens the traceability of content generation. The paradigm of RAG can be divided into three categories: Naive RAG, Advanced RAG, and Modular RAG [ 6 ] . The commonality among these types is that they comprise a retriever and a generator. Consequently, most researchers focus on improving the performance of RAG from these two aspects, such as fine-tuning the embedding model in the retriever [ 2 , 24 ] , mixing multiple retrieval methods for multi-path recall [ 19 ] , reranking [ 25 ] , etc., and fine-tuning the LLM in the generator, as well as using different thought chain methods for different scenarios [ 16 , 20 , 21 ] . As the RAG method continues to be optimized, various benchmarks are also being refined, such as general benchmarks CRUD-RAG [ 14 ] , ARES [ 18 ] , RAGAS [ 5 ] , and some domain-specific RAG benchmarks, such as FinanceBench [ 9 ] , LegalBench [ 7 ] , MedExpQA [ 1 ] , MIRAGE [ 22 ] . Currently, there is no comprehensive benchmark for RAG testing in the Chinese legal Q&A domain. Thus, this paper introduces a Chinese legal RAG benchmark called Legal-DC, which thoroughly assesses both the retriever and the generator. 2.2 LLMs on Legal Domains Researchers initially used LLMs with specific prompts to evaluate contract legal issues and compared their performance to junior lawyers. Studies showed that LLMs were feasible for contract review and more efficient, providing faster and more accurate results at a lower cost [ 15 ] . Subsequently, researchers began to fine-tune LLMs with instructions to focus on specific domains, such as SaulLM-7B [ 3 ] , a large language model with 7 billion parameters designed to understand and generate legal text. Nevertheless, the issues of hallucinations [ 10 ] and outdated corpus remain severe concerns for the rigorous legal field, which the RAG method has effectively mitigated. Lawyer LLaMA [ 8 ] is a legal RAG framework developed based on the fine-tuning of legal documents with the LLaMA model, while ChatLaw [ 4 ] is a RAG framework based on the Ziya-LLaMA-13B model, which has been fine-tuned on legal documents. LLMs rely on accurate legal documents for reliable answers in the legal field, but