Dialogue Is Not Enough to Make a Communicative BabyLM (But Neither Is Developmentally Inspired Reinforcement Learning)
Francesca Padovani Bastian Bunzeck Manar Ali Omar Momen, Arianna Bisazza Hendrik Buschmeier Sina Zarrieß · 2025
Read Dialogue Is Not Enough to Make a Communicative BabyLM (But Neither Is Developmentally Inspired Reinforcement Learning) by Francesca Padovani Bastian Bunzeck Manar Ali Omar Momen, Arianna Bisazza Hendrik Buschmeier Sina Zarrieß (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-20358.txt.
Introduction
Francesca Padovani and six co-authors ask if a small language model trained just on dialogue can talk well, in the BabyLM style. Their model, llamalogue, learns from talk text alone. Fine-tune steps then push it to make more apt replies in chat. The aim is a small model with both form skill and talk skill.
The talk-trained models lag on most stock BabyLM tests. But they shine at picking the right next line in paired chat tests. PPO fine-tune gives mixed to bad shifts. DPO fine-tune lifts scores on their chat test by more. The team takes this to mean talk data alone, and growth-style training alone, are each not quite enough.
Catalog record
| CREATOR | Francesca Padovani Bastian Bunzeck Manar Ali Omar Momen, Arianna Bisazza Hendrik Buschmeier Sina Zarrieß |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-20358
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Dialogue Is Not Enough to Make a Communicative BabyLM (But Neither Is Developmentally Inspired Reinforcement Learning) Francesca Padovani Bastian Bunzeck Manar Ali Omar Momen, Arianna Bisazza Hendrik Buschmeier Sina Zarrieß Abstract We investigate whether pre-training exclusively on dialogue data results in formally and functionally apt small language models. Based on this pre-trained llamalogue model, we employ a variety of fine-tuning strategies to enforce “more communicative” text generations by our models. Although our models underperform on most standard BabyLM benchmarks, they excel at dialogue continuation prediction in a minimal pair setting. While PPO fine-tuning has mixed to adversarial effects on our models, DPO fine-tuning further improves their performance on our custom dialogue benchmark. 1 Introduction Large language models are capable of generating language with almost human-like fluency. To do so, however, they need unfathomable amounts of textual input as training data. In comparison, humans are highly sample-efficient learners and develop a full-fledged linguistic system from input that is orders of magnitude smaller. In the past, this sample efficiency has mostly been attributed to genetically pre-endowed priors ( Chomsky 1986 ; Berwick et al. 2011 ) . More recently, the quantitative, usage-based turn in linguistics has focused on the importance of language use, interaction and grounding in the real world and more domain-general cognitive mechanisms for language learning ( Tomasello 2003 ; Tomasello 2005 ; Behrens 2021 ) . Crucially, language is primarily a tool for communication ( Fedorenko et al. 2024 ; Levinson 2025 ) , and therefore all acquisition processes must be conceptualized accordingly. Lately, the BabyLM paradigm has emerged as a novel way of testing claims of learnability with little data, small language models and linguistically inspired evaluation tasks ( Warstadt et al. 2023 ; Hu et al. 2024 ; Charpentier et al. 2025 ) . Although highly optimized models are indeed able to capture linguistic structure very accurately (e.g., Charpentier and Samuel 2023 ; Tastet and Timiryasov 2024 ), they are still trained on a wider variety of input registers than the main input modality of children, namely child-directed speech in dialogue. Observed in isolation, child-directed speech does differ tremendously from other input modalities, featuring many fragments, more questions and less canonical SV(X) sentences ( Cameron-Faulkner et al. 2003 ; Bunzeck and Diessel 2025 ) . Despite Huebner et al. 2021 finding it to be conducive pretraining data for simplified benchmarks, more recent work has shown that its effects can be described as mixed at best ( Padovani et al. 2025 ; Bunzeck et al. 2025 ) . One possible explanation for this discrepancy is that autoregressive language models, trained on a next-token prediction task, do not model the communicative aspects that are seen as crucial for language acquisition and underlie the fragmented nature of child-caregiver dialogue. Common data pre-processing protocols for BabyLMs split child-caregiver dialogues into isolated sentences, which effectively removes communicative context that is available and central for human learners. Therefore, we conceptualize the task of training a BabyLM differently: We train a small, autoregressive model 1 1 1 Given its training on dialogue data only, throughout this paper we refer to the base model as llamalogue . All models and datasets can be found in this Huggingface collection . on dialogue triplets extracted from CHILDES ( MacWhinney 2000 ) . As such, our model is not a model of the learner per se , but of the interaction and communication underlying the language learning process. Additionally, we apply different reinforcement learning paradigms to our model to make the ‘child’ component of the dialogue system more fluent and contextually appropriate when interacting with a ‘caregiver’ dialogue partner. In sum, we test the following ideas through this process: (i) How does a BabyLM trained only on child–caregiver dialogue perform? And (ii) Are there ways of teaching BabyLMs to be more communicative speakers via interaction and communication? We find that (i) our base model pre-trained exclusively on child–caregiver dialogues maintains above-chance accuracy on formal linguistic competence, while achieving higher accuracy in predicting realistic communicative turns than a baseline autoregressive model. Moreover, (ii) directly aligning preferred child responses to caregiver utterances through DPO proves more effective than interactively fine-tuning the policy via PPO with a reward function, especially when evaluated on dialogue minimal pairs. However, none of these fine-tuning techniques improves performance on more formal benchmarks. 2 Related work Learning exclusively from CDS While the standard English BabyLM corpus consists of approximately 30% child-directed speech, ample work exists on pretraining LMs from scratch on 100% child-directed speech (CDS). In a seminal paper, Huebner et al. 2021 showed that a small 5M-parameter BabyBERTa model, trained on 5M lexical tokens of child-directed speech, shows the same accuracy on Zorro ( Huebner et al. 2021 , vocabulary-limited minimal pair tasks;) as the RoBERTa-base model with 125M parameters and trained on 30B words. Similar results are presented by Feng et al. 2024 , who show that autoregressive models trained on CDS alone perform only slightly worse on Zorro than comparable architectures trained on Wikipedia data, synthetic data, or the BabyLM corpus. However, their CDS models underperform other models tremendously on semantic similarity benchmarks. Negative results are also reported by Yedetore et al. 2023 , who show that autoregressive models trained on CHILDES data fail to reliably acquire hierarchical generalizations in question formation from declaratives, and rather prefer incorrect linear generalizations. Expanding the CDS-only training paradigm to more languages than English, Salhan et al. 2024 find that developmentally-inspired curriculum learning strategies during pretraining improve scores on syntactic minimal pairs for models trained on English, French, German, Chinese, or Japanese CDS, outperforming models trained on Wikipedia data by over 10%. Conversely, Padovani et al. 2025 report less positive results. For many syntactic minimal pair benchmarks, their CDS models underperform in comparison to Wikipedia-trained models across different languages (English, German, French). Finally, Bunzeck et al. 2025 approximate German CDS on the level of utterance-level construction distributions. They also find that models trained on it are generally inferior to models trained on comparable Project Gutenberg data when evaluated on syntactic benchmarks, although the CDS models show moderate improvements on some word-level benchmarks. In sum, it can therefore be said that pre-training on CDS is only conducive to language model performance for highly specific benchmarks like Zorro (although results are inconsistent across studies) or in more specific training regimens like curriculum learning. Cognitively/developmentally plausible RL Despite reinforcement learning, especially in the form of RLHF ( Ouyang et al. 2022 ) , being an integral part of modern language modeling practices, it has only very recently begun to get adopted in cognitively inspired modeling. Zhao et al. 2023 improve their small models trained on BabyLM data by constructing a RLHF dataset from human-annotated story continuations generated with regular GPT-2 and then reinforcing these storytelling capabilities of their models. While it does not improve performance on zeroshot benchmarks, it makes their models better base models for fine-tuning tasks. In a more developmentally inspired fashion, Ma et al. 2025 generate text continuations from a student GPT-2 model and compare these to an already further trained teacher model. A reward signal is then generated from the model’s estimated ‘age’ ( viz. training steps), based on its continuations and the teacher continuation. This