Are Formal and Functional Linguistic Mechanisms Dissociated in Language Models?
Michael Hanna 1, Yonatan Belinkov 2, Sandro Pezzelle 1 · 2025
Read Are Formal and Functional Linguistic Mechanisms Dissociated in Language Models? by Michael Hanna 1, Yonatan Belinkov 2, Sandro Pezzelle 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-11302.txt.
Introduction
Michael Hanna and co-authors ask in 2025 whether language models keep grammar skill and world skill in separate inner machinery. Models write fluent, well formed text with ease yet stumble more on reasoning and steady fact recall. Brain science hints that people split formal language work from functional work, and some claim models should do the same, by design or by learning.
The team extracted the minimal circuits, small subgraphs of each model, that carry out set formal and functional tasks, across five models and ten tasks. Circuits for the two task kinds barely overlapped. Yet formal task circuits also barely overlapped with each other, unlike the unified language zone in human brains. A softer test told a kinder story: circuits could still solve each other's tasks within their own kind. The authors conclude a single formal network stays elusive, even if a broad split is real.
Catalog record
| CREATOR | Michael Hanna 1, Yonatan Belinkov 2, Sandro Pezzelle 1 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 87 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-11302
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Are Formal and Functional Linguistic Mechanisms Dissociated in Language Models? Michael Hanna 1, Yonatan Belinkov 2, Sandro Pezzelle 1 Abstract Although large language models (LLMs) are increasingly capable, these capabilities are unevenly distributed: they excel at formal linguistic tasks, such as producing fluent, grammatical text, but struggle more with functional linguistic tasks like reasoning and consistent fact retrieval. Inspired by neuroscience, recent work suggests that to succeed on both formal and functional linguistic tasks, LLMs should use different mechanisms for each; such localization could either be built-in or emerge spontaneously through training. In this paper, we ask: do current models, with fast-improving functional linguistic abilities, exhibit distinct localization of formal and functional linguistic mechanisms? We answer this by finding and comparing the “circuits”, or minimal computational subgraphs, responsible for various formal and functional tasks. Comparing 5 LLMs across 10 distinct tasks, we find that while there is indeed little overlap between circuits for formal and functional tasks, there is also little overlap between formal linguistic tasks, as exists in the human brain. Thus, a single formal linguistic network, unified and distinct from functional task circuits, remains elusive. However, in terms of cross-task faithfulness—the ability of one circuit to solve another’s task—we observe a separation between formal and functional mechanisms, with formal task circuits achieving higher performance on other formal tasks. This suggests the existence of a set of formal linguistic mechanisms that is shared across formal tasks, even if not all mechanisms are strictly necessary for all formal tasks. \jvol vv \jnum nn 2025 \dochead Short Paper \affilblock 1 Introduction A wide body of research has argued that language and thought are dissociated in the human brain ( Fedorenko, Ivanova, and Regev, 2024 ) . That is, such research argues that the regions of the brain that respond differentially to well-formed linguistic input are distinct from those that respond to other structured inputs, such as mathematics, music, and code ( Amalric and Dehaene, 2019 ; Chen et al., 2023 ; Ivanova et al., 2020 ) . They are also distinct from regions that respond to language-adjacent capabilities such as theory of mind and reasoning ( Shain et al., 2022 ; Monti, Parsons, and Osherson, 2009 ) . These regions of the brain, termed the language network , are thus selective for language (perhaps narrowly defined), and language alone. In recent work, Mahowald et al. (2024) argue that these two types of stimuli, to which the language network does and does not respond, correspond to two distinct types of linguistic competence: formal and functional . Formal linguistic competence is necessary for “getting the form of language right”. It involves the correct structuring of language at the sub-word, lexical, and sentence level; in other words, phonology, morphology, syntax, and lexical semantics are all domains of formal linguistic competence. Past neuroscientific work has shown all of these to be supported by the language network ( Regev et al., 2024 ; Shain et al., 2024 ; Hu et al., 2022 ) . In contrast, functional linguistic competence involves abilities that allow speakers to achieve goals in the world, but may involve non-linguistic cognition. For example, speakers may engage in formal or pragmatic reasoning, or employ world knowledge in conversation, without such abilities being intrinsic to language. Similarly, speakers may use situation modeling skills to make sense of a narrative, or engage in theory of mind reasoning to understand their interlocutors’ point of view, although language use need not entail the use of these abilities. Moreover, exercising these abilities does not engage the brain’s language network. Mahowald et al. furthermore claim that this distinction between formal and functional linguistic competence is reflected in the performance of today’s large language models (LLMs). In particular, LLMs have strong formal linguistic competence as evidenced by their strong performance on syntax benchmarks and their general ability to output fluent and natural text ( Hu et al., 2020 ; Gauthier et al., 2020 ; Warstadt et al., 2020 ) . However, their functional linguistic competence is markedly worse: LLMs frequently output false reasoning, hallucinate untrue facts, and fail at complex social reasoning ( Dziri et al., 2023 ; Xu, Jain, and Kankanhalli, 2024 ; Strachan et al., 2024 ) . Much work has attempted to ameliorate these problems via retrieval-augmented generation or the use of chain-of-thought reasoning ( Gao et al., 2024 ; Wei et al., 2022 ) . However, Mahowald et al. offer another solution: perhaps LLMs would have stronger functional linguistic abilities if formal and functional linguistic abilities were as distinct in LLMs as they are in the human brain. Such a dissociation in LLMs could come about in two ways: either it could be explicitly built into them, or it could arise naturally via training, or the model’s inductive biases. Today’s LLMs have no explicit formal-functional modularity; however, it remains unknown whether formal-functional modularity has nonetheless arisen. In this paper, we seek to answer that question: to what extent are formal and functional linguistic competence dissociated in the internals of today’s LLMs? If the two are not dissociated, new architectures or training procedures that bias models towards a formal-functional dissociation may be necessary to achieve this. If the two are already dissociated (despite the fact that LLMs struggle more with functional linguistic competence), this might indicate that dissociation does not suffice to improve LLMs’ abilities. This question of dissociation is also relevant due to the increase in work that uses LLMs to explicitly model language in the human brain, oftentimes by predicting activations within the brain using those from LLMs ( Tuckute, Kanwisher, and Fedorenko, 2024 ; Sucholutsky et al., 2024 ) . The presence or absence of a dissociated language network in LLMs could help us judge whether such comparisons are licensed. Rather than comparing activations within models and brains, though, we propose to characterize mechanisms within models, and verify whether they are organized in the same way as the human brain. To investigate whether formal and functional competence are distinctly localized within the LLMs, we draw on techniques from LLM interpretability, and in particular, mechanistic interpretability , which studies the mechanisms that underlie LLMs’ behavior using low-level causal methods ( Ferrando et al., 2024 ) . Concretely, we study the formal-functional distinction using circuits : a circuit is a small path through the LLM that contains all of the mechanisms underlying its behavior on a task of interest ( Olah et al., 2020 ; Wang et al., 2023 ) . Circuits provide causal guarantees that the localization found is correct, as all parts of the model outside of the circuit can be ablated without changing model behavior; see Section 3 for more details. This targeted, causal evidence for the correctness of our localization is a notable advantage of our framework, as such evidence is harder to come by in human brains. 1 1 1 In humans, one must rely on either natural experiments (e.g., individuals with brain damage) or less precise causal methodologies. These are numerous, ranging from more targeted methods such as transcranial magentic stimulation or focused ultrasound, to the more general interventions used in psychedelics research, but they are overall less precise than our causal interventions; optogenetics allows for finer-grained causal manipulation of