LEMUR: A Corpus for Robust Fine-Tuning of Multilingual Law Embedding Models for Retrieval

Narges Baba Ahmadi, Jan Strich, Martin Semmann, Chris Biemann · 2026

Read LEMUR: A Corpus for Robust Fine-Tuning of Multilingual Law Embedding Models for Retrieval by Narges Baba Ahmadi, Jan Strich, Martin Semmann, Chris Biemann (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-09570.txt.

Introduction

Language models more and more mediate legal information, yet many-language legal search suffers from shaky search and a lack of domain-adapted open vector models. Narges Baba Ahmadi and three co-authors present LEMUR, a 2026 large-scale many-language corpus of EU green laws built from nearly twenty-five thousand official EUR-Lex PDF files across twenty-five tongues. PDF sources add noise through flawed pull-out.

The team gauges switch truth with a Lexical Content Score against source HTML versions. Fine-tuning three leading many-language vector models with contrast goals in single- and two-tongue settings lifts top-k search over strong rivals again and again, with the largest gains for low-resource tongues. Cross-tongue tests show gains moving to unseen tongues, hinting fine-tuning builds free-standing content pictures rather than tongue-specific cues. Code and data are shared.

Catalog record

CREATORNarges Baba Ahmadi, Jan Strich, Martin Semmann, Chris Biemann
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-09570

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LEMUR: A Corpus for Robust Fine-Tuning of Multilingual Law Embedding Models for Retrieval Narges Baba Ahmadi, Jan Strich, Martin Semmann, Chris Biemann Abstract Large language models (LLMs) are increasingly used to access legal information. Yet, their deployment in multilingual legal settings is constrained by unreliable retrieval and the lack of domain-adapted, open-embedding models. In particular, existing multilingual legal corpora are not designed for semantic retrieval, and PDF-based legislative sources introduce substantial noise due to imperfect text extraction. To address these challenges, we introduce LEMUR, a large-scale multilingual corpus of EU environmental legislation constructed from 24,953 official EUR-Lex PDF documents covering 25 languages. We quantify the fidelity of PDF-to-text conversion by measuring lexical consistency against authoritative HTML versions using the Lexical Content Score (LCS). Building on LEMUR, we fine-tune three state-of-the-art multilingual embedding models using contrastive objectives in both monolingual and bilingual settings, reflecting realistic legal-retrieval scenarios. Experiments across low- and high-resource languages demonstrate that legal-domain fine-tuning consistently improves Top-k retrieval accuracy relative to strong baselines, with particularly pronounced gains for low-resource languages. Cross-lingual evaluations show that these improvements transfer to unseen languages, indicating that fine-tuning primarily enhances language-independent, content-level legal representations rather than language-specific cues. We publish code 1 1 1 GitHub Repository and data 2 2 2 Hugging Face Dataset . 1 Introduction LLMs are transforming legal work and research by making access to legal knowledge, automated document review, and case law summarization significantly faster and easier 34 . However, the deployment of these models in legal practice is often hindered by "hallucinations" and a lack of grounding in authoritative legal sources 23 ; 19 . To mitigate these risks, Retrieval-Augmented Generation (RAG) has become the de facto standard architecture, ensuring that model outputs are anchored in verifiable primary documents 13 . While RAG relies on the combination of an LLM for generation and embedding models for retrieval, its success is fundamentally dependent on the retrieval setup and the embedding model used for the vector database 9 . While these models are typically general-purpose, fine-tuning them on domain-specific data consistently yields superior performance compared to existing specialized models 25 , particularly in law, where text often contains archaic terminology, complex syntactic structures, or polysemy 1 . Nevertheless, these models are untrained in the legal domain and primarily monolingual 3 ; 4 or proprietary 28 . While multilingual datasets in law already exist, the data are typically formatted for pretraining 12 ; 20 or for classification of legal documents 2 , leaving a void for high-quality benchmarks dedicated to cross-lingual semantic retrieval. Furthermore, legal corpora are often stored in PDF format, which can introduce inaccuracies when converted to text for effective search due to multi-column layouts and nested tables. This ’extraction gap’ affects data integrity in RAG systems, as downstream embedding models are forced to process corrupted or misaligned tokens. To address these gaps, our contributions are: • Multilingual Dataset (LEMUR): We introduce a L aw E uropean MU ltilingual R etrieval corpus ( LEMUR ), which consists of 25k EU legal PDFs in 25 languages, designed for training embedding models on legal text. • The Lexical Content Score (LCS): We systematically analyze PDF-to-text conversion quality by measuring content consistency across twenty-five languages. • Legal Embedding Fine-Tuning: We fine-tune three SOTA embedding models on five languages for a legal document retrieval task and evaluate them in monolingual, bilingual, and cross-lingual settings. 2 Related Work Multilingual Legal Corpora. Research on multilingual legal corpora has produced both supervised benchmarks ( 3 ; 4 ; 34 ; 18 ) and large-scale pretraining resources ( 20 ; 8 ) . 2 introduces MultiEURLEX , a multilingual multi-label dataset of EU legislation in 23 languages for legal document classification, while 4 proposes LexGLUE , a suite of English legal NLU benchmarks that has become a standard evaluation protocol for legal language models. Beyond EU legislation, 34 presents CaseHOLD, a multiple-choice benchmark comprising more than 53,000 U.S. case-law holdings, and 18 introduces LeCaRD, a large-scale case-retrieval dataset for the Chinese criminal law system with expert-designed relevance criteria. Our work contributes to this line of research by constructing a new multilingual EU law dataset directly from official legislative PDFs and targeting downstream embedding-model fine-tuning across multiple European languages, thereby bridging large-scale pretraining corpora and task-specific benchmarks in an EU legislative setting. Embedding Models and Legal Retrieval. Recent work shows that structure-aware models such as SAILER ( 14 ) and DELTA ( 15 ) capture section-level or structural dependencies to improve legal case retrieval, while SM-BERT-CR ( 29 ) and ReaKase-8B ( 24 ) incorporate supporting-relation modeling and reasoning-driven representations. For multilingual and cross-lingual settings, LexCLiPR ( 27 ) enables paragraph-level retrieval across ECtHR judgments, showing that off-the-shelf multilingual encoders struggle without domain-adaptive training. Domain-specific pretraining consistently improves legal NLP tasks. 16 shows that in-domain pretraining and long-document handling benefit legal classification, while 6 demonstrates gains from adapting BERT to legal NER over BiLSTM–CRF baselines. More broadly, 25 and BloombergGPT ( 31 ) provide evidence that domain-adapted embeddings remain essential despite strong general-purpose LLMs. Although these studies focused mainly on monolingual legal data or non-legal domains, they have not systematically studied cross-lingual retrieval for EU legislation. We address this gap by introducing a multilingual EU law corpus and evaluating fine-tuned embedding models for monolingual and cross-language retrieval on EUR-Lex texts. 3 LEMUR We construct LEMUR from official documents published on EUR-Lex 3 3 3 https://eur-lex.europa.eu/homepage.html . Section 3.1 details how the source documents are identified, selected, and collected from the EUR-Lex repository. Section 3.2 then explains the process of converting the original PDF files into a structured and machine-readable text format, and Section 3.3 describes the subsequent preprocessing steps, including the construction of high-quality query–document pairs used in our experiments. An overview of the data preparation process from EUR-Lex PDFs to structured JSONL is shown in Figure 3 . 3.1 Document Collection To build a focused corpus, we gathered all legal acts listed under Category 15 (Environment, consumers and health protection), Subcategory 10 (Environment), across all available publication years. This yielded 1,174 distinct legal acts from 1961–2025 . Because each act is available in 25 official EU languages, the collection comprises a total of 24,953 PDF documents and results in 461k pages. Figure 2 summarizes the number of records per language. Coverage is highest for languages with long-standing EU membership (e.g., German, Dutch, English, Italian), and lower for countries with more recent membership (e.g., Croatia or Bulgaria). Figure 1: Average Content Score similarity per year (5-year bins) for the five languages used in our experiments 3.2 PDF–to–Text Conversion In the original source of EUR-Lex the dataset is available in PDF and HTML format. Previous datasets 2 used the HTML version, but we found that tables were not converted correctly. Therefore, we tested multiple PDF–to–text services (Docling 17 , Unstructured 26 , PyMuPDF 7 ) but found that the best results were obtained by converting all PDFs into structured JSONL files using olmOCR ( 21 ) . On average, documents contain approximately 19 pages, with approximately 403 tokens per page, yielding roughly 7,781 tokens per document. These values indicate that LEMUR consists primarily of long-form legislative text, making it well-suited for evaluating embedding models on long-document and multilingual retrieval

Read the complete text