Findings of the Second BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora
Michael Y. Hu Aaron Mueller Candace Ross, Adina Williams Tal Linzen Chengxu Zhuang Ryan Cotterell, Leshem Choshen Alex Warstadt Ethan Gotlieb Wilcox · 2024
Read Findings of the Second BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora by Michael Y. Hu Aaron Mueller Candace Ross, Adina Williams Tal Linzen Chengxu Zhuang Ryan Cotterell, Leshem Choshen Alex Warstadt Ethan Gotlieb Wilcox (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-05149.txt.
Introduction
Michael Hu and co-authors published a report in 2024 on the second BabyLM Challenge, a contest for training language models on little data. Human learners need far less text than models do, and the contest tries to close that gap. Entrants train on a fixed budget of at most one hundred million words. This year added better text sets and an image-and-text set.
Thirty-one entries were scored on grammar, question answering, and other skills across three tracks. A hybrid model design beat the other approaches overall. No entry beat the baselines on the image track. Follow-up checks tied more compute to better scores and found the best entries changed data, goals, and design together.
Catalog record
| CREATOR | Michael Y. Hu Aaron Mueller Candace Ross, Adina Williams Tal Linzen Chengxu Zhuang Ryan Cotterell, Leshem Choshen Alex Warstadt Ethan Gotlieb Wilcox |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-05149
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Findings of the Second BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora Michael Y. Hu Aaron Mueller Candace Ross, Adina Williams Tal Linzen Chengxu Zhuang Ryan Cotterell, Leshem Choshen Alex Warstadt Ethan Gotlieb Wilcox Abstract The BabyLM Challenge is a community effort to close the data-efficiency gap between human and computational language learners. Participants compete to optimize language model training on a fixed language data budget of 100 million words or less. This year, we released improved text corpora, as well as a vision-and-language corpus to facilitate research into cognitively plausible vision language models. Submissions were compared on evaluation tasks targeting grammatical ability, (visual) question answering, pragmatic abilities, and grounding, among other abilities. Participants could submit to a 10M-word text-only track, a 100M-word text-only track, and/or a 100M-word and image multimodal track. From 31 submissions employing diverse methods, a hybrid causal-masked language model architecture outperformed other approaches. No submissions outperformed the baselines in the multimodal track. In follow-up analyses, we found a strong relationship between training FLOPs and average performance across tasks, and that the best-performing submissions proposed changes to the training data, training objective, and model architecture. This year’s BabyLM Challenge shows that there is still significant room for innovation in this setting, in particular for image-text modeling, but community-driven research can yield actionable insights about effective strategies for small-scale language modeling. 1 Introduction This paper describes the second BabyLM Challenge and its findings. The broader goals and motivation of the challenge have remained constant since the first iteration last year. At the heart of both this year’s and last year’s challenge is the observation that children are incredibly data-efficient language learners, whereas artificial neural-network-based language models are not. On the one hand, children are exposed to less than 100 million word tokens by the age of 13 ( Gilkerson et al., 2017 ) , at which point they have mastered their native language(s). On the other hand, today’s ANN-based language models are trained on trillions of words—five to six orders of magnitude more than the typical human language learner. For a more in-depth discussion on the issue of data efficiency, see the findings of last year’s challenge ( Warstadt et al., 2023 ) as well as Wilcox et al. (2024) , a position piece written by many of the challenge organizers. The learning discrepancy between humans and models raises two important questions: First, how is it that humans are able to learn language so efficiently? And second, what insights from human language learning can be used to improve language models? It is our hope that by creating a platform for interested parties to experiment with data-limited and cognitively inspired language modeling, we can continue to make progress on these interrelated questions. In particular, our goal with BabyLM is to contribute to: 1. Building more cognitively and developmentally plausible models of human language acquisition and processing, which can be used for the scientific study of language. 2. Optimizing training pipelines prior to scaling, allowing for faster iteration on architectures and hyperparameters. 3. Enabling research on language model training to a wider group of interested researchers, beyond highly-funded industry labs. The main difference between this year’s and last year’s challenge is twofold: First, this year we allowed participants to bring their own datasets, as long as they stayed within the 100 million word limit for our Strict track, or the 10 million word limit for our Strict-Small track. The motivation behind this decision is that pretraining data quality has been linked to large improvement gains in at-scale language models ( Gunasekar et al., 2023 ) , so this year we allowed participants to improve the training data beyond the provided dataset, which was effectively a dataset baseline. Second, this year included a Multimodal track, in which participants trained on aligned text-image data, and tested their models in a novel text-image evaluation pipeline. Non-linguistic information, such as visual input, potentially plays a large role in child language acquisition. While visual input is not inherently necessary for successful language acquisition (for example, blind children learn language largely without issue), visual grounding has been linked to faster language learning Pérez-Pereira and Castro (1992) ; Campbell et al. (2024) . Furthermore, visual grounding has long been hypothesized to aid word learning: children learn nouns more easily than verbs Gentner (1982) ; McDonough et al. (2011) , arguably because the former are more easily linked to visual stimuli than the latter. Additionally, children learn concrete nouns easier than abstract nouns Bergelson and Swingley (2013) . However, visual grounding also presents several challenges: Words may be time-delayed with respect to their referents, or one word may be uttered in a context with multiple competing possible referents. With this in mind, our hope was that the Multimodal track would help to explore the space of possible computational models for visual grounding during language acquisition. Findings and takeaways. This year, we received 31 submissions from 17 different countries making diverse contributions. Examples included submissions proposing novel architectures, new training objectives, innovating on knowledge distillation methods, and proposing curriculum learning methods, among others. We conduct a meta-analysis of the results, which yields several concrete recommendations. The best-performing submissions constructed their own training datasets, proposed new model architecture, or new training objectives. Performance on the BabyLM evaluations also correlated strongly with training FLOPs, suggesting that high-compute training regimes still tend to reliably perform better, even in low-data settings. The BabyLM research community also showed growing attention to tokenization and multilingual language modeling, while maintaining interest in curriculum learning and applying linguistic biases to language models. Our data (pretraining corpora and evaluation data; [link] ), preprocessing code [link] , baselines [link] and evaluation pipeline [link] are all publicly available. We also release the submitted models of those who agreed to release them, along with their hyperparameters and results [link] . The leaderboard may be found here [link] . 2 Competition Details Tracks. The second BabyLM Challenge included three competition tracks: Strict , Strict-Small , and Multimodal . Additionally, we opened a standalone Paper track, accepting research related to cognitive modeling with language models or small-scale pretraining, similar to a workshop. The Strict and Strict-Small tracks required that submissions be trained on 100M words or less and 10M words or less, respectively. These tracks no longer required that participants use the fixed dataset from last year’s challenge, although we still provided an updated version of this dataset, described in Section 3 . Models in these tracks were evaluated on language-only evaluation tasks. In the Multimodal track, participants trained multimodal image-text models. Participants were allowed to use any model and training procedure they desired, as long as the model could assign (pseudo) log-likelihoods to strings of text, conditioned on an image. Again, participants were free to construct their own datasets, including unlimited visual inputs, as long as the text data was within a 100M word budget. To facilitate easier participation in this track, we released a suggested multimodal dataset that consisted of 50% text-only and 50% paired image-text data. Submissions to this track were evaluated on both language-only and additional multimodal tasks. 3 Pretraining Corpus Dataset Description # Words (multimodal) # Words (strict) # Images Localized Narratives a Image Caption 27M – 0.6M