Pantagruel: Unified Self-Supervised Encoders for French Text and Speech
2026
Read Pantagruel: Unified Self-Supervised Encoders for French Text and Speech (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-05911.txt.
Introduction
Phuong-Hang Le and 29 co-authors release Pantagruel in 2026, a family of self-supervised encoder models for French text and speech. French lacks the scale of English resources, so strong native foundation models matter. Pantagruel differs from predecessors by learning contextualized representations in feature space rather than predicting tokens or speech units directly.
Separate models were pretrained on large French corpora, including Wikipedia and web text for writing, and radio and television archive audio for speech. One new corpus alone contributes one hundred thousand hours of broadcast audio. Across French benchmarks for both modalities, Pantagruel matched or beat strong baselines such as CamemBERT and FlauBERT. A shared architecture handles either input type, supporting multimodal French understanding.
Catalog record
| YEAR | 2026 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-05911
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Pantagruel: Unified Self-Supervised Encoders for French Text and Speech Abstract We release Pantagruel models, a new family of self-supervised encoder models for French text and speech. Instead of predicting modality-tailored targets such as textual tokens or speech units, Pantagruel learns contextualized target representations in the feature space, allowing modality-specific encoders to capture linguistic and acoustic regularities more effectively. Separate models are pre-trained on large-scale French corpora, including Wikipedia, OSCAR and CroissantLLM for text, together with MultilingualLibriSpeech, LeBenchmark, and INA-100k for speech. INA-100k is a newly introduced 100 000-hours corpus of French audio derived from the archives of the Institut National de l’Audiovisuel (INA), the national repository of French radio and television broadcasts, providing highly diverse audio data. We evaluate Pantagruel across a broad range of downstream tasks spanning both modalities, including those from the standard French benchmarks such as FLUE or LeBenchmark. Across these tasks, Pantagruel models show competitive or superior performance compared to strong French baselines such as CamemBERT, FlauBERT, and LeBenchmark 2.0, while maintaining a shared architecture that can seamlessly handle either speech or text inputs. These results confirm the effectiveness of feature-space self-supervised objectives for French representation learning and highlight Pantagruel as a robust foundation for multimodal speech–text understanding. Keywords: self-supervised learning, JEPA, data2vec, French language models, speech and text encoders, multimodal representation learning, joint-embedding predictive architecture, predictive modeling Pantagruel: Unified Self-Supervised Encoders for French Text and Speech Phuong-Hang Le 1 , Valentin Pelloin 2 , Arnault Chatelain 4 , Maryem Bouziane 3 , Mohammed Ghennai 1 , Qianwen Guan 5 , Kirill Milintsevich 2 , Salima Mdhaffar 3 , Aidan Mannion 1 , Nils Defauw 6 , Shuyue Gu 5 , Alexandre Audibert 1 , Marco Dinarelli 1 , Yannick Estève 3 , Lorraine Goeuriot 1 , Steffen Lalande 2 , Nicolas Hervé 2 , Maximin Coavoux 1 , François Portet 1 , Étienne Ollion 4 , Marie Candito 5 , Maxime Peyrard 1 , Solange Rossato 1 , Benjamin Lecouteux 1 , Aurélie Nardy 7 , Gilles Sérasset 1 , Vincent Segonne 8 , Solène Evain 1 , Diandra Fabre 1 , Didier Schwab 1 1 Univ. Grenoble Alpes, CNRS, Grenoble INP, LIG, 38000 Grenoble, France 2 INA (Institut National de l’Audiovisuel), 4 Avenue de l’Europe, 94366 Bry-sur-Marne, France 3 Avignon Université, LIA, France 4 CREST (École Polytechnique, ENSAE, CNRS), 5 avenue Le Chatelier, 91120 Palaiseau, France 5 LLF (Université Paris Cité and CNRS), UFRL Olympe de Gouges, 13 place Paul Ricoeur, 75013 Paris, France 6 Univ. Grenoble Alpes, EFELIA-MIAI, IUT2 Grenoble, LIG, 38000 Grenoble, France 7 Univ. Grenoble Alpes, Lidilem, 38000 Grenoble, France 8 Université Bretagne Sud, CNRS, IRISA, France Abstract content 1. Introduction Mirroring trends in other languages, self-supervised encoders have become the standard backbone for French speech and language processing. Text models such as FlauBERT ( 69 ) and CamemBERT 79 , together with LeBenchmark 42 ; 95 for speech, have established strong baselines across a variety of downstream tasks. Yet most prior work relies on token-level reconstruction 29 ; 10 ; 115 , which can under-utilize the structural regularities of continuous signals and hinders a unified treatment of text and speech. Recent predictive approaches 71 ; 4 ; 8 that learn contextualized targets in feature space offer a promising alternative, enabling modality-specific encoders to capture richer linguistic and acoustic structure beyond the surface form. In this paper, we introduce Pantagruel , a family of French self-supervised encoders for text and speech, trained separately using the data2vec 2.0 architecture ( 9 ) , an instance of the joint embedding predictive architecture (JEPA) framework 71 . Pantagruel follows a teacher–student training paradigm in which the student predicts masked latent representations produced by a teacher that observes the full, unmasked input. Such representation-based objectives have proven highly effective for vision 4 ; 84 and audio 44 ; 110 ; 116 , yet they remain underexplored for text. Recent work suggests that textual tokens are compact, semantically dense units with minimal low-level variability, leaving limited room for embedding-based methods to improve over input-level approaches 111 . Our experiments on text-based models also confirm this hypothesis. Therefore, for text, we propose to augment the feature-space objective with masked language modeling ( 29 , MLM,) to better capture fine-grained syntactic and semantic information, yielding stronger textual representations while retaining the benefits of contextualized target prediction. Our self-supervised speech and text models are released on the HuggingFace hub 1 1 1 https://huggingface.co/PantagrueLLM . To support large-scale pre-training in French, we curate substantial corpora for each modality. For text, we use Wikipedia, OSCAR 79 and CroissantLLM 43 datasets. For speech, we assemble a diverse collection spanning read, spontaneous, professional, and broadcast speech audio data, including 14 000 audio hours from LeBenchmark 42 and 100 000 audio hours from INA-100k , a new corpus derived from the archives of France’s National Audiovisual Institute (INA) that we introduce in this paper. We evaluate Pantagruel on a broad suite of downstream tasks in both modalities and compare it to three main French baselines: FlauBERT ( 69 ) and CamemBERT ( 79 ) for text, and LeBenchmark 41 ; 42 ; 95 for speech. Contributions First, we release Pantagruel, a family of French self-supervised encoders for speech and text based on the data2vec 2.0 and JEPA frameworks, which allows training models on different modalities using the same framework. Second, we investigate embedding-based prediction objectives for text, an underexplored regime, and show that combining feature-space prediction with MLM yields competitive results for French text encoders. Third, we study the impact of the large-scale INA-100k broadcast corpus on the models’ performances. Finally, we provide a unified evaluation across speech and text tasks, where Pantagruel consistently matches or improves over established French baselines. Figure 1: Overview of the Pantagruel model architecture. The network starts with a modality-specific pre-net to extract feature vectors from the input text/speech sequence. These features are input to a teacher encoder, while randomly chosen visible tokens (in blue) are input to a student encoder. A lightweight decoder predicts the teacher’s latent representations from the student’s outputs. For text input, an additional masked language modeling (MLM) loss is used. The teacher’s parameters are updated as an exponential moving average (EMA) of the student’s. After training, only the embedding layer and the student encoder are used for fine-tuning on downstream tasks. 2. Related Work Self-supervised learning (SSL) has driven rapid progress in several domains, in particular in text and speech processing. In text, encoder-only models such as BERT 29 learn bidirectional representations through the MLM objective, with subsequent refinements improving attention mechanisms, efficiency, and context length ( 25 ; 57 ; 67 ; 115 ) , while the GPT family ( 102 ; 103 ; 17 ) popularized autoregressive pre-training for generative tasks. In speech, wav2vec2.0 ( 10 ) introduced contrastive learning over quantized representations, while HuBERT ( 59 ) adopted masked prediction of discrete clusters, and WavLM ( 23 ) improved robustness through denoising objectives. Initially developed for English, these self-supervised frameworks were rapidly extended to multilingual settings ( 29 ; 26 ; 105 ; 108 ; 7 ) . For French, many studies have adapted these self-supervised architectures to both modalities. In text, FlauBERT ( 69 ) and CamemBERT ( 79 )