Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora

Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro Rafael Mosquera Bhargavi Paranjape, Adina Williams Tal Linzen Ryan Cotterell · 2025

Read Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora by Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro Rafael Mosquera Bhargavi Paranjape, Adina Williams Tal Linzen Ryan Cotterell (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-08165.txt.

Introduction

Alex Warstadt and a large team of co-authors report the findings of the BabyLM Challenge. Their 2025 paper asks how far language models can get on little data. Children learn language from under one hundred million words. Big models need thousands of times more and still trail humans on many tests.

The challenge fixed a data budget and let teams compete on grammar, task skill, and generalization. Over thirty entries came in across three tracks with loosening data rules. Winners using one compact design beat models trained on trillions of words. Short inputs and student-teacher training also did well. Teaching order tricks, though popular, disappointed. The report tells researchers where data-efficient training pays off.

Catalog record

CREATORAlex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro Rafael Mosquera Bhargavi Paranjape, Adina Williams Tal Linzen Ryan Cotterell
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE99 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-08165

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro Rafael Mosquera Bhargavi Paranjape, Adina Williams Tal Linzen Ryan Cotterell Abstract Children can acquire language from less than 100 million words of input. Large language models are far less data-efficient: they typically require 3 or 4 orders of magnitude more data and still do not perform as well as humans on many evaluations. These intensive resource demands limit the ability of researchers to train new models and use existing models as developmentally plausible cognitive models. The BabyLM Challenge is a communal effort in which participants compete to optimize language model training on a fixed data budget. Submissions are compared on various evaluation tasks targeting grammatical ability, downstream task performance, and generalization. Participants can submit to up to three tracks with progressively looser data restrictions. From over 30 submissions, we extract concrete recommendations on how best to train data-efficient language models, and on where future efforts should (and perhaps should not) focus. The winning submissions using the LTG-BERT architecture ( Samuel et al., 2023 ) outperformed models trained on trillions of words. Other submissions achieved strong results through training on shorter input sequences or training a student model on a pretrained teacher. Curriculum learning attempts, which accounted for a large number of submissions, were largely unsuccessful, though some showed modest improvements. * * footnotetext: Equal contribution. 1 Introduction Although there have massive improvements in the effectiveness of neural language models in the last decade, humans are still the state of the art in language learning. To achieve impressive results, language models need to be trained on hundreds of times more language input than a typical human will be exposed to in an entire lifetime. The BabyLM Challenge is a shared task that invites members of the natural language processing, linguistics, and cognitive science communities to train language models in low-resource data settings, where the amount of linguistic input resembles the amount received by human language learners. In doing so, our motivations ( Section 2 ) are to improve the relevance of language models as cognitive models of human language acquisition, find more effective and data-efficient training algorithms for language models, and democratize research on language model training by emphasizing research questions that can be addressed on a smaller training budget. Figure 1: Data Scale: Modern Language Models are trained multiple orders of magnitude more word tokens than the amount available to a typical child. This image is based on Fig. 1 from Warstadt and Bowman (2022) . Participants in the shared task could submit to the Strict , Strict-Small , or Loose track, which, respectively, required models to be trained on corpora that constituted either 10 million words, 100 million words, or 100 million words plus an unlimited amount of additional non-linguistic data ( Section 3 ). These corpora were constructed from a mixture of sources including developmentally plausible domains such as child-directed speech, transcribed dialogue, and children’s literature ( Section 4 ). To enable standardized evaluation and easy comparison of the resulting models, we create a leaderboard and release an evaluation pipeline ( Section 5 ) targeting zero-shot grammatical performance, finetunability on language understanding tasks, and model inductive bias. We also contribute a novel set of zero-shot evaluation tasks targeting semantic and discourse-level phenomena. We received 31 papers making a variety of contributions, ranging from designing novel architectures and tuning hyperparameters to employing curriculum learning and training teacher–student model pairs ( Section 6 ). We conduct a meta-analysis of the results, yielding several concrete recommendations and scientific conclusions ( Section 7 ). The winners of the challenge’s various tracks made contributions that led to impressive improvements in our evaluation over not just the BabyLM baselines, but also the massively pretrained Llama 2 model ( Touvron et al., 2023 ) . The best-performing models overall ( Charpentier and Samuel, 2023 ) use the LTG-BERT architecture ( Samuel et al., 2023 ) , which synthesizes a number of recent optimizations of the Transformer architecture. The winner of the Loose track ( Xiao et al., 2023 ) trains the models continuously on the training samples belonging to the same source dataset while randomizing the dataset orders in each training epoch. Other submissions did not achieve strong downstream results, but still provided valuable scientific contributions. We received many curriculum learning submissions, including one that systematically tested a variety of strategies ( Martinez et al., 2023 ) and reported few improvements over non-curriculum baselines. Steuer et al. (2023) found that benchmark performance is not correlated with a greater ability to predict human psycholinguistic data. We plan to organize future BabyLM Challenges that will build on the success of this first iteration ( Section 8 ). The winning submission from this year sets a high baseline for next year. Future iterations will need harder and more varied evaluations, including those that emphasize human-like processing and learning; they should emphasize new approaches that were not thoroughly explored this year, such as multimodality; and, they should incentivize compute-efficiency. Altogether, the first BabyLM Challenge has been a successful initiative, and we hope that this will continue to advance research on small-scale language models. 2 Motivation The observation at the center of the BabyLM Challenge is this: Children are incredibly data-efficient language learners, and language models are not. Children are exposed to less than 100 million word tokens by age 13 ( Gilkerson et al., 2017 ) , while modern language models are typically trained on 3 or 4 orders-of-magnitude more data (Figure 1 ). This discrepancy raises two important questions: First, how is it that humans are able to learn language so efficiently? Second, what insights from human language learning can be used to improve language models? A great deal of recent work in language model training seeks improvements by scaling up pretraining data and parameters ( Raffel et al., 2020 ; Brown et al., 2020 ; Hoffmann et al., 2022 ; Touvron et al., 2023 ) . Scaling is undoubtedly central to building deployable models (though see McKenzie et al. 2023 for counterexamples) and raises its own set of scientific questions, such as quantitative scaling laws ( Kaplan et al., 2020 ) and the emergence of new abilities ( Wei et al., 2022 ) . However, increased emphasis on scaling is unlikely to lead to answers to the two questions we raised, and it excludes researchers without access to massive computational resources. Thus, there are three principal benefits to data-limited language model training which the BabyLM Challenge aims to highlight: 1. Building more cognitively and developmentally plausible models of human language acquisition and processing, 2. Optimizing training pipelines prior to scaling by allowing for faster iteration on architectures and hyperparameters, and 3. Enabling research on language model training beyond highly funded industry groups. Cognitive Modeling. Language models have been used to model aspects of human language learning and processing for decades ( Elman, 1990 ; Hale, 2001 ; Reali and Christiansen, 2005 , o.a.) . While many researchers continue to advocate for language models as cognitive models ( Keller, 2010 ; Dupoux, 2018 ; Linzen, 2019 ; Baroni, 2022 ; Warstadt and Bowman, 2022 ;

Read the complete text