Enhancing Legal LLMs through Metadata-Enriched RAG Pipelines and Direct Preference Optimization

Suyash Maniyar, Deepali Singh, Rohith Kumar Reddy · 2026

Read Enhancing Legal LLMs through Metadata-Enriched RAG Pipelines and Direct Preference Optimization by Suyash Maniyar, Deepali Singh, Rohith Kumar Reddy (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-19251.txt.

Introduction

Maniyar and co-authors make small legal language models safer and better grounded in long documents. Such models often invent clauses or cite false precedents. The team traces errors to poor search in repetitive legal text and to answers given with too little context. Their fix enriches search with metadata and mixes sources. A preference tuning step then teaches the model to refuse when context is thin. The pair of changes lifts grounding and trust in local legal use.

Catalog record

CREATORSuyash Maniyar, Deepali Singh, Rohith Kumar Reddy
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37623
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.19251

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Enhancing Legal LLMs through Metadata-Enriched RAG Pipelines and Direct Preference Optimization Suyash Maniyar, Deepali Singh, Rohith Kumar Reddy Abstract 1 Abstract Large Language Models (LLMs) perform well in short contexts but degrade on long legal documents, often producing hallucinations such as incorrect clauses or precedents. In the legal domain, where precision is critical, such errors undermine reliability and trust. Retrieval Augmented Generation (RAG) helps ground outputs but remains limited in legal settings, especially with small, locally deployed models required for data privacy. We identify two failure modes: retrieval errors due to lexical redundancy in legal corpora, and decoding errors where models generate answers despite insufficient context. To address this, we propose Metadata Enriched Hybrid RAG to improve document level retrieval, and apply Direct Preference Optimization (DPO) to enforce safe refusal when context is inadequate. Together, these methods improve grounding, reliability, and safety in legal language models. 2 Related work 2.1 Hallucination in Legal Domain Recent years have seen the emergence of legal-specific large language models (LLMs) such as LawGPT 6 , LexiLaw 1 , and Lawyer LLaMA 3 , which are trained on extensive legal datasets to improve domain understanding. Despite these efforts, hallucination, where models generate factually incorrect or ungrounded legal information remains a significant issue. Prior work has explored retrieval-augmented generation (RAG) and preference optimization techniques such as DPO and SimPO to improve factuality. However, these methods are not specifically designed to handle the nuanced types of hallucinations common in legal contexts, such as incorrect statute citations or fabricated legal provisions. The work in 2 builds on these ideas by introducing a customized benchmark, LegalHallBench, and a fine-tuning strategy that combines SFT with iterative DPO focused on hard samples to reduce hallucinations and improve factual accuracy in legal QA. This work mainly focuses on Chinese legal data. 2.2 RAG in Legal Domain Along with fine-tuning models for specific domains, Retrieval-Augmented Generation (RAG) has been introduced to improve model accuracy and reduce hallucinations by providing relevant context alongside the query. Different strategies for retrieving and injecting context have led to various RAG architectures tailored to different types of data. In the legal domain, the frequent cross-referencing within documents makes it challenging to design effective RAG approaches 7 . In 4 , the authors benchmark several RAG strategies on question-answering tasks, including legal datasets. However, most existing RAG systems applied to legal documents suffer from Document-Level Retrieval Mismatch (DRM), where the retriever selects chunks from incorrect source documents due to high structural and lexical similarity in legal corpora 5 . Most previous work focuses either on improving the retriever or on downstream tasks such as question answering. In contrast, our work looks at both. We also try to understand how improving RAG retrievers, especially through correct and relevant metadata injection and better chunking strategies, which are not fully explored in prior work which affects overall RAG performance. 3 Dataset We leveraged three carefully selected datasets: PrivacyQA , MAUD , and Australian Legal QA . 3.1 RAG Dataset PrivacyQA consists of 194 question-answer pairs across 453 snippets extracted from 7 major technology company privacy policies. This presents dense, user-focused legal language that challenges models to generate accurate, understandable responses. MAUD , a contract analysis dataset, contains 1,676 question-answer pairs across 2,839 snippets from 150 unique merger and acquisition agreements. This requires models to handle long-range dependencies, hierarchical clauses, and complex cross-references. The Australian Legal QA dataset was split into training and test sets to ensure fair and unbiased evaluation of retrieval and question-answering performance. The documents for this dataset were programmatically collected using a scraping script from federal and state court repositories, ensuring authentic legal text coverage. Collectively, these datasets expose models to diverse legal contexts, long and complex documents, and high-stakes factual precision requirements. This makes it suitable benchmark for retrieval-augmented and alignment-focused methods in legal natural language processing. The legal domain presents unique computational challenges that distinguish it from general-purpose NLP tasks. Legal documents feature extreme length (contracts often exceed 50,000 words), specialized terminology with precise semantic boundaries, and complex cross-referential structures where meaning depends on distant contextual elements. Additionally, legal question-answering demands absolute factual accuracy—incorrect party names, dates, or procedural details. Each dataset follows a similar JSON structure, where queries are paired with relevant text snippets extracted from source documents, including file paths, text spans, and answer segments. An illustrative example from the MAUD dataset is shown below: Listing 1: MAUD Dataset JSON Structure Example ⬇ { " query ": " What is the Type of Consideration in the acquisition ?", " snippets ": [ { " file_path ": " maud / Michaels_Companies_Apollo . txt ", " span ": [5284, 5913], " answer ": "... at a price per share of $22 .00 ( the Offer Price ), net to the holder ..." }, { " file_path ": " maud / Michaels_Companies_Apollo . txt ", " span ": [86031, 86699], " answer ": " Each Share shall be converted into the right to receive ..." } ] } 3.1.1 RAG Data Preprocessing For PrivacyQA and MAUD dataset, train-test splits were readily available. These were then extracted and converted into chunks using recursive chunking strategy. For the Australian Legal QA dataset, the released test split does not provide explicit span annotations linking each question to the corresponding evidence segments within the source judgments. However, span-level ground truth is required for evaluating retrieval quality using chunk-based metrics that rely on positional overlap between retrieved text and gold evidence. To address this limitation, we reconstructed span annotations by re-aligning each answer with its originating judgment document. Specifically, we fetched the full-text judgments from the source URLs provided in the dataset, applied the same recursive chunking procedure used during indexing, and located the minimal contiguous character span that best matched the gold answer text. These reconstructed spans were then used as ground-truth evidence positions for retrieval evaluation. This enabled fair and consistent assessment of chunk-level recall and overlap-based retrieval metrics across all datasets. 3.2 DPO Dataset The dataset used for DPO is derived from Australian legal question-answering (QA) dataset. The dataset was divided into a training set and a test set, with 1918 samples used for training and 150 samples used for testing and 50 samples as validation set. The total number of training samples used for DPO training was 1918 \times 2, to account for two sets of data, questions with correct context and questions with incorrect context. The Australian legal QA dataset has this information for each QA pair that we leveraged in synthesising DPO dataset and RAG experiments, it also includes other information which is not particularly useful for us. { "Question", "document URL", "Context": "<Text excerpt from doc to answer>", "Document MetaData", "Answer" } 3.2.1 DPO Data Creation For DPO training, the dataset was organized into two sets: Set 1 (Correct Context): In this set, the correct context corresponding to each question is provided in the input prompt. The model is expected to answer based only on the given context. If the context allows the answer to be derived, the model is expected to generate the ground truth (GT) answer. If the context is insufficient, the model should respond with “Given context is not sufficient to answer.” Set

Read the complete text