The Lucie-7B LLM and the Lucie Training Dataset: Open resources for multilingual language generation

Olivier Gouvert (1), Julie Hunter (1), Jérôme Louradour (1), Christophe Cerisara (2), Evan Dufraisse (2), Yaya Sy (2), Laura Rivière (3), Jean-Pierre Lorré (4), OpenLLM-France community · 2025

Read The Lucie-7B LLM and the Lucie Training Dataset: Open resources for multilingual language generation by Olivier Gouvert (1), Julie Hunter (1), Jérôme Louradour (1), Christophe Cerisara (2), Evan Dufraisse (2), Yaya Sy (2), Laura Rivière (3), Jean-Pierre Lorré (4), OpenLLM-France community (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-12294.txt.

Introduction

Olivier Gouvert and co-authors present in 2025 an open language model and training data set centered on French. Most large model data lean heavily toward English. Their collection draws on web sources plus French cultural heritage documents to fill that gap.

Other European languages, including English, Spanish, German, and Italian, are also included. The data set limits copyrighted material and is shared in the form used for training. The model, Lucie-7B, trains on equal shares of French and English. Two instruction-tuned versions are released, with results the authors call promising against leading models.

Catalog record

CREATOROlivier Gouvert (1), Julie Hunter (1), Jérôme Louradour (1), Christophe Cerisara (2), Evan Dufraisse (2), Yaya Sy (2), Laura Rivière (3), Jean-Pierre Lorré (4), OpenLLM-France community
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE111 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-12294

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Lucie-7B LLM and the Lucie Training Dataset: Open resources for multilingual language generation Olivier Gouvert (1), Julie Hunter (1), Jérôme Louradour (1), Christophe Cerisara (2), Evan Dufraisse (2), Yaya Sy (2), Laura Rivière (3), Jean-Pierre Lorré (4), OpenLLM-France community Abstract We present both the Lucie Training Dataset and the Lucie-7B foundation model, open resources created by the OpenLLM-France community. 1 1 1 The OpenLLM France Community is a community of people interested in all aspects of the development of open language models and is organized mainly around the French section of a Discord server now called OpenLLM Europe. The Lucie Training Dataset is a multilingual collection of textual corpora centered around French, and designed to offset the anglo-centric biases found in many datasets for large language model pretraining. Its French data is pulled not only from the traditional web data sources, but also from French cultural heritage documents, filling an important gap in modern datasets. Beyond French, which makes up the largest share of the data, we added documents to support several other European languages, including English, Spanish, German, and Italian. Apart from its value as a resource for French language and culture, an important feature of this dataset is that it prioritizes data rights by minimizing copyrighted material. In addition, building on the philosophy of past open projects, it is redistributed 2 2 2 A single dataset, accounting for 0.2% of the final dataset, could not be redistributed as explained in Section 3 but its content is described on Hugging Face. in the form used for training and its processing is described on Hugging Face and GitHub. The Lucie-7B foundation model is trained on equal amounts of data in French and English---roughly 33% for each---in an effort to better represent cultural aspects of French-speaking communities. We also describe two instruction fine-tuned versions of Lucie-7B, Lucie-7B-Instruct-v1.1 and Lucie-7B-Instruct-human-data, which we release as demonstrations of the foundation model in use. These models achieve promising results compared to state-of-the-art models, demonstrating that an open approach prioritizing data rights can still deliver strong performance. We see these models as an initial step toward developing more performant, and aligned models in the near future. The Lucie-7B resources, like the dataset, are all open. Model weights for Lucie-7B and the Lucie instruct models, along with intermediate checkpoints for the former, are likewise published on Hugging Face, while model training and data preparation code is freely available on GitHub. This makes Lucie-7B one of the first OSI compliant language models according to the new OSI definition. 3 3 3 Open Source AI Definition https://opensource.org/ai/open-source-ai-definition Keywords Large Language Models, Multilingual Language Models, Open Data, Pretraining, Textual Dataset 1 Introduction When we set out to create Lucie-7B, one of our principal goals was to offset the English-centered bias found in the training sets of most mainstream large language models (LLMs) such as Meta’s Llama models. Poor representation of a language can lead not only to reduced language modeling performance for that language, but what’s more, a model that has seen scant data from a certain language is likely to have received scant information about cultures associated with that language. When faced with a question about history, religion, social practices or even everyday activities like cooking, the model is more likely to provide an answer suited to the language on which it is predominantly trained, which can lead to unsatisfying results for members of other language communities. For these reasons, together with the fact that the Lucie team was based in France, we decided early on that Lucie-7B would see at least as much French data as English. A second priority was to ensure that our data collection practices were respectful of the rights of the data producers. This includes abiding by copyright and intellectual property restrictions as well as eliminating personally identifying information where possible. An immediate consequence of this choice was that we needed to be careful in using data collected from the web. Initially, our plans were to rely only on sources, such as Wikipedia, that are well known to provide clean, high-quality data. The reality of the situation, however, is that the vast majority of LLM training data comes from the web and this data provides a level of quantity and diversity that is hard to reach through other means. We ultimately decided to gently relax our prohibition of general web data, though limiting it remained a guiding principle throughout our data collection process. Finally, building on the open-source philosophy of LINAGORA, the founding partner of the OpenLLM France community, a third central motivation for our project was to further growing efforts to shed light on LLM development. In fact, the name “Lucie” is derived from the Latin word for “light,” reflecting our desire make every aspect of our process visible to the public. From the beginning, we committed to sharing all information about our training data, including publication of the training datasets where possible, model weights, evaluation results, and all code used for training and data preparation. What we did not set out to do was to create an all-purpose chat assistant in the style of ChatGPT. Not only is ChatGPT allegedly hundreds of times larger than Lucie, but it, like all of the best performing models nowadays, has undergone multiple phases of fine-tuning, including alignment through methods like Reinforcement Learning with Human Feedback (RLHF) or Direct Preference Optimization (DPO), with large amounts of high-quality data. Collecting the data needed for these methods was beyond our resources in terms of time and human labor for this particular project. In the context of a recently begun research grant that carries the same name as the OpenLLM-France community, 4 4 4 We gratefully acknowledge support from the BPI funded project OpenLLM-France. however, we are working with a larger, dedicated group of partners to build aligned versions of Lucie-7B. The resources described in this paper are thus the first (major) steps in a larger research program. We go ahead and release them in the spirit of open-source collaboration so that those who wish can begin to use the resources for their own purposes. This paper describes in detail the Lucie Training Dataset and the Lucie-7B models, including the foundation model and the two associated instruction-tuned models released as demonstrations. The Lucie Training Dataset, most of which has been republished on Hugging Face, 5 5 5 https://huggingface.co/datasets/OpenLLM-France/Lucie-Training-Dataset is a collection of texts from diverse sources in French, English, Spanish, German and Italian as well as various programming languages. The majority of its data comes from French; it is in fact one of the biggest collections of French text data that has been preprocessed for LLM training. We provide further details in Section 3 . Lucie-7B, while proficient in all of the natural languages from the Lucie Training Dataset, is designed to be particularly competent at generating French and English: roughly one third of its training data is in English and another third is in French. It is presented in Section 4 , where we describe tokenizer training as well as the three stages of pretraining carried out to create the final model. Model weights, including weights for regular intermediate checkpoints, are available on Hugging Face. 6 6

Read the complete text