Language Acquisition Device in Large Language Models
Masato Mita, Taiga Someya, Ryo Yoshida, Yohei Oseki · 2026
Read Language Acquisition Device in Large Language Models by Masato Mita, Taiga Someya, Ryo Yoshida, Yohei Oseki (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-16758.txt.
Introduction
Mita and colleagues test whether pre-pretraining on a language inspired by the Language Acquisition Device hypothesis can close the data-efficiency gap between language models and humans. Their MP-STRUCT language encodes layered structure, feature-based links, and long-distance movement through MERGE, AGREE, and MOVE. A brief 500-step run matches strong formal-language baselines in token use and adds human-like pushback against languages with impossible structures. Useful landmarks that reduce link doubt emerge as a key driver of the effect.
Catalog record
| CREATOR | Masato Mita, Taiga Someya, Ryo Yoshida, Yohei Oseki |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47490 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.16758
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Language Acquisition Device in Large Language Models Masato Mita, Taiga Someya, Ryo Yoshida, Yohei Oseki Abstract Large Language Models (LLMs) remain substantially less data-efficient than humans. Pre-pretraining (PPT) on synthetic languages has been proposed to close this gap, with prior work emphasizing highly expressive formal languages such as k k -Shuffle Dyck. Inspired by the Language Acquisition Device (LAD) hypothesis, which posits that innate constraints preemptively restrict the learner’s hypothesis space to natural-language-like structure, we propose LAD-inspired PPT : pre-pretraining on MP-Struct , a formal language whose strings encode hierarchical composition, feature-based dependencies, and long-distance displacement via Merge , Agree , and Move . A brief 500-step PPT with MP-Struct matches strong formal-language baselines in token efficiency while additionally imparting a human-like resistance to structurally implausible languages (e.g., Reverse ). Analyzing simplified variants, we find that MP-Struct Core outperforms k k -Shuffle Dyck despite not being definable in C-RASP (a formal bound on transformer expressivity), challenging the prior hypothesis that effective PPT languages must be both hierarchically expressive and circuit-theoretically learnable. We show that functional landmarks , which reduce dependency resolution ambiguity, are a key driver, suggesting that effective PPT design depends not only on expressivity but also on the accessibility of dependency resolution. https://github.com/osekilab/LAD-PPT 1 Introduction Large language models (LLMs) exhibit general linguistic abilities comparable to those of humans; however, their efficiency in language acquisition remains far inferior. While humans acquire language from limited text, LLMs typically require orders of magnitude more data to achieve strong performance 19 . This gap suggests that current LLMs rely on learning from an overly permissive hypothesis space 22 , leaving substantial room for improving learning efficiency through better inductive biases. Recent work by 12 explores pre-pretraining (PPT) , where models are first trained on synthetic sequences to acquire useful structural biases before standard pretraining. They show that highly expressive formal languages, such as k -Shuffle Dyck, can improve token efficiency by exercising the model’s ability to process hierarchical dependencies. They further interpret these results through the expressivity hypothesis , which posits that a formal language conferring a helpful inductive bias should be hierarchically structured in the sense of the Chomsky hierarchy 5 (either context-free or context-sensitive) and definable in C-RASP 21 , a formal measure of circuit complexity for transformers. However, such formal languages prioritize abstract structural complexity and often lack key properties characteristic of natural language. This raises the question: beyond Chomsky-hierarchy complexity and circuit complexity, do natural-language-like properties, such as dependencies conditioned on fixed hierarchical structure, also contribute to effective PPT? In this work, we approach this question from a complementary perspective. Taking inspiration from the Language Acquisition Device (LAD) hypothesis 6 , which suggests that innate structural constraints can restrict the hypothesis space and favor natural-language-like structure, we ask whether incorporating such constraints into synthetic sequences can further improve learning efficiency. Guided by this idea, we design MP-Struct , a synthetic generator taking cues from the Minimalist Program (MP) , which produces sequences where dependencies are embedded within a fixed hierarchical structure rather than freely interleaved. We evaluate LAD-inspired PPT on Pythia-1B 2 . A brief 500-step PPT phase with MP-Struct consistently improves token efficiency over training from scratch and achieves performance comparable to strong baselines based on formal languages such as k -Shuffle Dyck. Moreover, the resulting models exhibit a directionally asymmetric inductive bias: compared to k -Shuffle Dyck, MP-Struct shows greater resistance to directionally reversed sequences, consistent with the LAD-inspired design goal of favoring natural-language-like directional structure. To better understand these effects, we analyze simplified variants of the generator. Notably, MP-Struct Core , an idealized abstraction of our generator, achieves higher efficiency than k -Shuffle Dyck despite not being definable in C-RASP, a formal lower bound on the expressivity of future-masked soft attention transformers. This observation is not fully explained by the expressivity hypothesis, which predicts that effective PPT languages should be hierarchically structured and definable in C-RASP. Our analysis instead points to a complementary factor: the accessibility of dependency retrieval. While structures that encode dependencies purely through bracketed hierarchy (e.g., k -Shuffle Dyck) define valid dependencies, they may leave multiple plausible antecedents, potentially leading to higher retrieval ambiguity. In contrast, MP-Struct and MP-Struct Core introduce explicit structural cues— functional landmarks —that can make dependencies easier to locate, thereby reducing the effective search cost for attention-based models. We hypothesize that these differences in accessibility contribute to the observed efficiency gains. More broadly, this suggests that effective PPT design may depend not only on formal expressivity, but also on how structural information is organized to support efficient dependency retrieval. 2 Related Work 2.1 LAD/UG and Minimalism A longstanding challenge in language acquisition is explaining how children converge on rich grammatical competence from comparatively limited and noisy input (often termed the poverty of the stimulus ) 6 ; 9 . The LAD hypothesis addresses this gap by proposing that learners are endowed with Universal Grammar (UG), a species-specific set of constraints that sharply restricts the hypothesis space of possible grammars. From this perspective, UG serves as a strong innate inductive bias, filtering out “non-human-like” hypotheses a priori . In contemporary generative grammar, the Minimalist Program (MP) refines this LAD/UG model by seeking to minimize the computational machinery of language to a small set of operations 4 ; 7 ; 8 . A central component of this framework is Merge , a combinatory operation that builds hierarchical structure. In addition, operations such as Move (displacement) and Agree (feature valuation) are commonly assumed to play roles in dependency formation and morphosyntactic licensing 7 ; 8 . 2.2 Pre-pretraining on Synthetic Structures Algorithm 1 Data Generation Procedure ( MP-Struct ) Notation: \mathcal{L} : lexicon, V/N/D : lexical categories, vP : verb phrase, T/C : functional heads, t : trace, u/iNum : number features, wh\in\{+,-\} . 1: Input: lexicon \mathcal{L} , parameters \theta 2: Output: token sequence S 3: Step 1: Base Structure via Merge 4: Sample lexical items V,D_{1},D_{2},N_{1},N_{2}\sim\mathcal{L} 5: DP_{subj}=\textsc{Merge}(D_{1},N_{1}) , DP_{obj}=\textsc{Merge}(D_{2},N_{2}) 6: V^{\prime}=\textsc{Merge}(V,DP_{obj}) ; vP=\textsc{Merge}(DP_{subj},V^{\prime}) 7: // Yields a hierarchical phrase structure with subject and object 8: Step 2: Functional Structure and Agree 9: Assign number feature iNum\in\{sg,pl\} to DP_{subj} 10: Create T[uNum] and Merge with vP 11: Set uNum\leftarrow iNum via \textsc{Agree}(T,DP_{subj}) 12: Form TP=[TP\ DP_{subj}\ T[uNum]\ vP] 13: // Yields a subject–verb agreement dependency encoded in the structure 14: Step 3: Move (Dependency Encoding) 15: Copy DP_{subj} to clause-initial position 16: Replace its original position with a trace: t_{subj} 17: Form TP=[TP\ DP_{subj}\ T\ [vP\ t_{subj}\ [V^{\prime}\ V\ DP_{obj}]]] 18: Sample wh\in\{+,-\} 19: Merge C[wh] with TP 20: if wh=+ then 21: Select a DP in TP and mark it as DP_{wh} 22: Copy DP_{wh} to clause-initial position 23: Replace its original position with a trace t_{wh} 24: Form CP=[CP\ DP_{wh}\ C\ [TP\ ...\ t_{wh}\ ...]] 25: end if 26: // Yields a long-distance dependency between a moved element and its trace 27: Step 4: Linearization 28: Traverse the tree in pre-order 29: Output brackets, nonterminal labels, features, and traces 30: Remove lexical terminals ( V,N,D ) \rightarrow S 31: // Yields a token sequence encoding structural relations without lexical content Pre-pretraining (PPT) on synthetic structures has emerged as a new learning paradigm for language models. Existing studies