Repetition Without Exclusivity: Scale Sensitivity of Referential Mechanisms in Child-Scale Language Models

JP Cacioli · 2026

Read Repetition Without Exclusivity: Scale Sensitivity of Referential Mechanisms in Child-Scale Language Models by JP Cacioli (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-13696.txt.

Introduction

Jon-Paul Cacioli studies repetition and mutual exclusivity in child-scale language models trained on child-directed speech. Mutual exclusivity is the bias to map novel words to novel referents, measured here as referential suppression. Pilot tests showed a masked model was insensitive to referential context, while autoregressive models showed repetition priming instead. In a pre-registered test, 45 GPT-2 models were trained across sizes, epochs, and seeds on AO-CHILDES. Anti-ME repetition priming was significant in all cells and never crossed zero.

Catalog record

CREATORJP Cacioli
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30582
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.13696

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Repetition Without Exclusivity: Scale Sensitivity of Referential Mechanisms in Child-Scale Language Models JP Cacioli Abstract We present the first systematic evaluation of mutual exclusivity (ME)—the bias to map novel words to novel referents—in text-only language models trained on child-directed speech. We operationalise ME as referential suppression: when a familiar object is relabelled in a two-referent discourse context, ME predicts decreased probability of the labelled noun at a subsequent completion position. Three pilot findings motivate a pre-registered scale-sensitivity experiment: (1) a masked language model (BabyBERTa) is entirely insensitive to multi-sentence referential context, precluding ME evaluation; (2) autoregressive models show robust repetition priming—the opposite of ME—when familiar nouns are re-labelled; and (3) a novel context-dependence diagnostic reveals that apparent ME-like patterns with nonce tokens are fully explained by embedding similarity, not referential disambiguation. In the confirmatory experiment, we train 45 GPT-2-architecture models (2.9M, 8.9M, and 33.5M parameters; 5, 10, and 20 epochs on AO-CHILDES; 5 seeds each) and evaluate on a pre-registered ME battery. Anti-ME repetition priming is significant in all 9 cells of the model grid (85–100% of items; all p < 2.4 × 10 − 13 p<2.4\times 10^{-13} ). Priming attenuates with improved language modelling (Spearman ρ = − 0.533 \rho=-0.533 , p = 0.0002 p=0.0002 ) but never crosses zero across a 3.8 × \times perplexity range. The context-dependence diagnostic replicates in all 9 cells, and dose–response priming increases with repetitions in 8/9 cells (all trend p < 0.002 p<0.002 ). These findings indicate that, under child-scale data and architectures, distributional learning on child-directed speech produces repetition-based reference tracking rather than lexical exclusivity: priming attenuates toward a non-zero asymptote but does not invert. We connect this dissociation to the grounded cognition literature, where ME does emerge in visually grounded models, and argue that referential grounding may be a necessary ingredient for ME—an empirical claim about required input structure, not a nativist one. 1 Introduction The BabyLM paradigm trains language models on child-scale data and evaluates linguistic competence ( 17 ; 2 ) . Evaluations focus on syntax ( 18 , BLiMP;) and natural language understanding ( 16 , GLUE;) . But referential pragmatics—how children use words to pick out objects in the world—remains untested. We address this gap using mutual exclusivity (ME) as the test case. ME is the bias to map novel words to novel referents rather than objects that already have a name ( 9 ) . Intuitively, if a child sees a familiar dog and an unfamiliar object and hears “look at the dax!”, ME leads the child to assume “dax” refers to the unfamiliar object—because the dog already has a name. The bias is central to early word learning, developmentally graded and modulated by vocabulary size ( 7 ) , weaker in bilinguals ( 1 ) , and has been studied computationally in grounded models ( 4 ; 12 ; 11 ; 13 ; 15 ) . A key feature of all computational work showing positive ME results is that the model has referents—images, object representations, or symbolic referent sets—to which labels can be mapped. BabyLM-class models are text-only. Can ME emerge from distributional learning on child-directed speech alone, without referential grounding? This is not a question about whether text-only models should exhibit ME—it is a question about whether distributional statistics are sufficient to induce it. We test this across 45 autoregressive models spanning a 12 \times parameter range and 3.8 \times perplexity range, plus a masked language model ( 6 , BabyBERTa;) . We find no evidence of ME at any scale tested; instead, models develop repetition priming that attenuates with improved language modelling but never reverses sign. We provide a mechanistic account grounded in the statistics of child-directed speech and introduce a context-dependence diagnostic that rules out artefactual ME from embedding similarity—a reusable methodological contribution for prompt-based pragmatic evaluation. 2 Background 2.1 ME in Developmental Psychology ME was first characterised by 9 as a constraint on word learning: children assume that each object has only one label, so a novel word is mapped to the unnamed object rather than one that already has a name. The bias emerges around 17 months ( 5 ) , strengthens with vocabulary growth ( 7 ) , and is modulated by language experience—bilingual children show weaker ME, consistent with their routine experience of multiple labels for the same referent ( 1 ) . An important distinction drawn by 10 separates referent selection (an in-the-moment disambiguation) from word learning (retention over time); our evaluation targets the former. Bayesian accounts formalise ME as a consequence of probabilistic inference over word–referent mappings ( 19 ) . Under this framework, the critical computational ingredient is the referent set: the learner must represent that there are distinct objects in the world, each requiring a label. The size principle—that a hypothesis consistent with fewer examples is more likely under strong sampling—naturally produces one-to-one label–referent mappings. 2.2 ME in Computational Models Cross-situational neural models trained on symbolic and image data exhibit ME when lexical competition is built into the learning or selection process ( 4 ) . Pragmatic agents that combine Rational Speech Act reasoning with gradient-based learning also develop ME ( 12 ) , as do meta-learning systems designed for few-shot word learning ( 3 ) . The most relevant recent work comes from visually grounded speech (VGS) models. 11 showed that ME emerges from contrastive learning on speech–image pairs, with stronger ME when models have more visual prior knowledge. Extending this to the bilingual case, 13 found that bilingual VGS models exhibit weaker ME—paralleling human developmental data. 15 introduced MEBench and reported that even state-of-the-art vision-language models show only weak ME bias. The key insight from this literature is that every positive ME result involves models with referents. ME has never been evaluated in text-only models trained on naturalistic child-directed speech. 2.3 Child-Scale Language Models The BabyLM Challenge ( 17 ; 2 ) trains models on corpora of 10M–100M words—approximating the linguistic input available to children—and evaluates them on syntactic benchmarks (BLiMP, Zorro) and NLU tasks (GLUE). BabyBERTa ( 6 ) trains a RoBERTa-base model on AO-CHILDES, a corpus of American-English child-directed speech ( {\sim} 4M tokens). These evaluations focus on syntax and NLU; referential pragmatics is untested. We fill this gap. 3 Method 3.1 Pre-Registration and Transparency We conducted a pilot study on a single autoregressive model and BabyBERTa (described in § 3.5 ) that yielded three mechanistic findings. Based on these exploratory results, we pre-registered a confirmatory scale-sensitivity experiment on the Open Science Framework (OSF; https://osf.io/zu7af ) before training any additional models. The pre-registration specifies four hypotheses (§ 3.6 ), statistical tests, significance thresholds, and the exact training grid. Train and evaluation scripts were archived in the registration snapshot and are publicly available at https://github.com/synthiumjp/study2-me-eval . Pilot findings are reported as exploratory with full transparency about the analysis path. 3.2 Corpus AO-CHILDES ( 6 ) contains 893,989 sentences ( {\sim} 4.07M word tokens, 6.27M BPE tokens) of American-English child-directed speech from the CHILDES database ( 8 ) . The corpus is overwhelmingly composed of single-referent naming contexts; multi-referent disambiguation constructions (e.g., “the X or the Y”) are rare. A property of CDS that is directly relevant to our findings is

Read the complete text