LegalRAG: A Hybrid RAG System for Multilingual Legal Information Retrieval

Muhammad Rafsan Kabir 1 * , Rafeed Mohammad Sultan 1 * , Fuad Rahman 2 , Mohammad Ruhul Amin 3 , Sifat Momen 1 , Nabeel Mohammed 1 , Shafin Rahman 1 · 2025

Read LegalRAG: A Hybrid RAG System for Multilingual Legal Information Retrieval by Muhammad Rafsan Kabir 1 * , Rafeed Mohammad Sultan 1 * , Fuad Rahman 2 , Mohammad Ruhul Amin 3 , Sifat Momen 1 , Nabeel Mohammed 1 , Shafin Rahman 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-16121.txt.

Introduction

Muhammad Rafsan Kabir and co-authors build a question-answering system for multilingual legal documents in this 2025 paper. Language technology has barely touched legal and regulatory work. Their test bed is the Bangladesh Police Gazettes, written in both English and Bangla. The system pairs search with generation, a design called retrieval augmented generation. Retrieved passages ground the answers the model writes. An advanced variant of that design sharpens retrieval further.

Better retrieval means answers cite the right rule more often. Bilingual records are common in government but hard to search by hand. The framework aims at fast, precise answers from such archives. Staff could query regulations in either language. The work is a systems paper with a working prototype. It targets low-resource language settings often ignored by legal tech.

Catalog record

CREATORMuhammad Rafsan Kabir 1 * , Rafeed Mohammad Sultan 1 * , Fuad Rahman 2 , Mohammad Ruhul Amin 3 , Sifat Momen 1 , Nabeel Mohammed 1 , Shafin Rahman 1
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-16121

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LegalRAG: A Hybrid RAG System for Multilingual Legal Information Retrieval Muhammad Rafsan Kabir 1 * , Rafeed Mohammad Sultan 1 * , Fuad Rahman 2 , Mohammad Ruhul Amin 3 , Sifat Momen 1 , Nabeel Mohammed 1 , Shafin Rahman 1 Abstract Natural Language Processing (NLP) and computational linguistic techniques are increasingly being applied across various domains, yet their use in legal and regulatory tasks remains limited. To address this gap, we develop an efficient bilingual question-answering framework for regulatory documents, specifically the Bangladesh Police Gazettes, which contain both English and Bangla text. Our approach employs modern Retrieval Augmented Generation (RAG) pipelines to enhance information retrieval and response generation. In addition to conventional RAG pipelines, we propose an advanced RAG-based approach that improves retrieval performance, leading to more precise answers. This system enables efficient searching for specific government legal notices, making legal information more accessible. We evaluate both our proposed and conventional RAG systems on a diverse test set on Bangladesh Police Gazettes, demonstrating that our approach consistently outperforms existing methods across all evaluation metrics. 1 {muhammad.kabir, rafeed.sultan, sifat.momen, nabeel.mohammed, shafin.rahman}@northsouth.edu 2 [email protected], 3 [email protected] * Equal Contribution I Introduction In recent years, natural language processing (NLP) techniques have been widely applied in various domains, including healthcare, education, and governance, enabling the automated analysis and processing of large-scale textual data [ 1 ] . One particularly significant area of research is the application of NLP techniques to government documents, specifically legal and regulatory texts such as government gazettes [ 2 ] . These documents contain critical legal information and government notices, but are often vast, unstructured, and linguistically diverse, making them both valuable and challenging for NLP applications. The effective application of NLP techniques to these documents is crucial to improve the accessibility of information and the policy-making process [ 3 ] . Although previous research has explored NLP applications in the processing of government regulatory documents, most efforts have focused on high-resource languages such as English, with techniques primarily used for document preprocessing [ 4 , 5 ] . In addition, some studies have examined financial regulations, such as those related to trading decisions. However, large language models (LLMs) in these methodologies often exhibit biases, and limited post-processing techniques have been used to address these issues [ 6 ] . Despite some advancements, little effort has been made to extend such methodologies to low-resource languages like Bangla, where the lack of annotated datasets and specialized approaches presents significant challenges. This gap highlights the need for robust NLP solutions tailored to legal and regulatory documents in low-resource settings. Fig. 1 : A sample response generated by the conventional RAG pipeline (Vanilla RAG) and our proposed pipeline (Advanced RAG) for a given user query. Our proposed advanced RAG pipeline improves the response by retrieving an additional relevant text chunk for the LLM while eliminating an irrelevant one retrieved by the Vanilla RAG pipeline. The orange symbols indicate the English translation of the Bangla texts. To address these challenges, this work proposes a Retrieval-Augmented Generation (RAG) pipeline [ 7 ] designed specifically for low-resource languages, focusing on Bangla, to process complex government regulatory documents. Given that Bangladeshi government gazettes, such as Bangladesh Police Gazettes, contain text in both Bangla and English, a multilingual RAG pipeline is essential to effectively handle the bilingual nature of these documents and retrieve context-specific information. As a first step, we implement a conventional multilingual RAG pipeline for a Question Answering (QA) task on Bangladesh Police Gazettes to facilitate the retrieval of precise legal notices and government information. However, because of the low resource nature, complexity, and unstructured format of these gazettes, the conventional RAG pipeline often struggles with retrieval accuracy and response generation. To overcome these limitations, we introduce a novel and enhanced multilingual RAG framework specifically designed for document-centric question-answering. Our proposed approach integrates an additional LLM that performs relevance checks on retrieved text fragments and refines the user query when necessary. This ensures that only the most relevant information is passed on to the final LLM responsible for generating responses, as shown in Figure 1 . To evaluate the effectiveness of our approach, we curated a small yet diverse test set consisting of 168 question-answer pairs based on Bangladesh Police Gazettes. The performance of the system was assessed using two key criteria: human evaluation of generated responses and semantic similarity between generated outputs and ground-truth responses. Experimental results demonstrate that our proposed framework significantly outperforms the conventional RAG pipeline, highlighting its effectiveness in processing bilingual regulatory documents and improving QA accuracy. Contributions: To summarize, the major contributions of this work are as follows: • Introduction of an advanced multilingual RAG framework specialized for low-resource languages, specifically Bangla, to improve question-answering on complex bilingual legal documents. • Unlike the conventional RAG pipeline, our approach incorporates an additional LLM for relevance checking and query refinement, ensuring that only the most relevant texts are passed to the generative model, thereby improving accuracy. • A diverse Q&A test set is curated based on Bangladesh Police Gazettes to assess the performance of the system. The evaluation, conducted using human judgment and semantic similarity metrics, demonstrates that the proposed framework outperforms the conventional RAG pipeline in handling bilingual regulatory texts. II Related Works NLP in Legal Documents: Natural language processing (NLP) has been widely employed in various critical domains, including the legal domain [ 8 ] . Zhong et al. [ 9 ] provide an overview of legal artificial intelligence (LegalAI), specifically focusing on NLP techniques in the legal field. They highlight how NLP can be applied to legal tasks such as document analysis [ 10 ] and information retrieval [ 11 ] . The work by Chouhan and Gertz [ 12 ] contributes to legal NLP by drafting definitions for legislative documents using Retrieval-Augmented Generation (RAG) techniques [ 7 ] . Their primary focus is on a collection of European (EU) documents. The LegalBench-RAG paper by Pipitone and Alami [ 13 ] introduces a RAG benchmark [ 14 ] for the legal domain to evaluate the effectiveness of RAG-based systems. Despite these advancements in legal NLP, significant gaps still persist, especially in the domain of low-resource legal documents. Hence, our work contributes to this domain by primarily focusing on low-resource legal texts in Bangla. Retrieval Augmented Generation: While pre-trained LLMs have demonstrated remarkable abilities in storing factual knowledge and achieving SOTA results across various NLP tasks, their capacity to effectively access and manipulate knowledge remains limited. Retrieval-Augmented Generation (RAG), introduced by Lewis et al. [ 7 ] , enhances question-answering (QA) tasks by generating more specific, diverse, and factual responses. However, most RAG-based research has primarily focused on English applications, leaving a gap in studies addressing low-resource languages. Chirkova et al. [ 15 ] introduced the concept of multilingual RAG, exploring its effectiveness across different languages. Li et al. [ 16 ] highlighted the transformative impact of recent LLMs, such as Llama [ 17 ] and GPT [ 18 ] , on multilingual NLP. Their work proposed a cross-lingual retriever that maps Bangla text into a shared embedding space using cross-lingual prompts, demonstrating that high-resource language prompts can improve performance for low-resource languages. Based on these advancements, we employ

Read the complete text