Causal Interventions Reveal Shared Structure Across English Filler–Gap Constructions
Sasha Boguraev, Christopher Potts, Kyle Mahowald · 2025
Read Causal Interventions Reveal Shared Structure Across English Filler–Gap Constructions by Sasha Boguraev, Christopher Potts, Kyle Mahowald (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-16002.txt.
Introduction
Sasha Boguraev and co-authors prod chat models from inside to test if one shared gear serves a flock of tongue forms. The forms, where a front part waits for its mate past a gap, span asks and kin clauses. Tongue lore holds the flock shares deep traits. Their 2025 study swaps inner states between runs of kin forms and watches the calls bend. Like bends mark like gears.
The models do meet on near-same deep reads of the flock. Three hid wells bend those reads: how oft a form runs, what fills the gap, and what wraps the clause. In-tool proof can thus push tongue lore, not just ape it.
Catalog record
| CREATOR | Sasha Boguraev, Christopher Potts, Kyle Mahowald |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-16002
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Causal Interventions Reveal Shared Structure Across English Filler–Gap Constructions Sasha Boguraev, Christopher Potts, Kyle Mahowald Abstract Large Language Models (LLMs) have emerged as powerful sources of evidence for linguists seeking to develop theories of syntax. In this paper, we argue that causal interpretability methods, applied to LLMs, can greatly enhance the value of such evidence by helping us characterize the abstract mechanisms that LLMs learn to use. Our empirical focus is a set of English filler–gap dependency constructions (e.g., questions, relative clauses). Linguistic theories largely agree that these constructions share many properties. Using experiments based in Distributed Interchange Interventions, we show that LLMs converge on similar abstract analyses of these constructions. These analyses also reveal previously overlooked factors – relating to frequency, filler type, and surrounding context – that could motivate changes to standard linguistic theory. Overall, these results suggest that mechanistic, internal analyses of LLMs can push linguistic theory forward. 1 Introduction Language models can generate and process utterances typically thought to require rich linguistic grammatical structure ( Futrell et al. 2019 ; Wilcox et al. 2018 ; Manning et al. 2020 ; Hu et al. 2020 ) , including much-studied syntactic constructions like long-distance filler–gap constructions ( Wilcox et al. 2024 ) . These results have been taken to challenge claims that these phenomena can be learned only with strong innate priors ( Piantadosi 2023 ; Futrell and Mahowald 2025 ) . Figure 1: Causal intervention overview. Here, we illustrate our methodology when we intervene within a class, transferring an embedded wh- filler–gap structure into a corresponding minimal pair that didn’t previously have one. We then show intervening across classes, inserting a wh- filler–gap into a gap-less cleft sentence Despite the strong performance, questions remain as to whether models acquire syntax in ways that are posited by linguists to be human-like (e.g., acquiring rich grammatical abstraction and syntactic structure). Causal interpretability methods now make it possible to characterize the abstract mechanisms underlying neural networks ( Vig et al. 2020 ; Finlayson et al. 2021 ; Geiger et al. 2021 ; Meng et al. 2022 ; Geiger et al. 2023 ; Wang et al. 2023 ) . These methods have revealed non-trivial linguistic syntactic structure is learned by models ( Arora et al. 2024 ; Lasri et al. 2022 ; Finlayson et al. 2021 ; Mueller et al. 2022 ; Lakretz et al. 2019 ) . But a key hypothesis in the history of linguistics is that seemingly different linguistic constructions can share underlying structure. For instance, compare “I wonder what the lion ate.” to “It was the gazelle that the lion ate.” The former is an embedded wh- clause and the latter is a cleft construction. These are distinct constructions but share something in common: both have a long-distance dependency with an extracted element, often specified with a linguistic trace: “I wonder what t the lion ate ___ t .” and “It was the gazelle t that the lion ate ___ t .” Thus, many linguistic theories predict common processing characteristics between these sentences ( Fodor 1989 ) . On the other hand, there is also reason to expect wh- sentences to be quite different from clefts since both wh- elements and clefts have idiosyncratic properties ( Ross 1967 ; Culicover 1999 ; Ginzburg and Sag 2001 ) . To tackle these questions, we take advantage of advances in large open source models as well as advances in mechanistic interpretability, specifically the Causal Abstraction framework ( Geiger et al. 2023 ) and Distributed Alignment Search (DAS; Geiger et al. 2024 ). Our resulting methodology gives us direct access to the abstract causal mechanisms learned by these models. By accessing these causal mechanisms, we can take a filler-gap mechanism learned on Construction A (e.g., wh- sentences), transfer it to Construction B (e.g., clefts), and see if we get predictable filler–gap behavior (see Figure 1 ). If we do, this would be strong evidence of underlying shared structure learned by the model. Importantly, this method gives us a gradient measure of transfer. As such, we explore whether more similar constructions transfer more readily to each other; whether some constructions in general tend to serve as sources of transfer; whether mechanisms transfer across clauses; and whether transfer is greater when lexical items are shared across constructions ( Pickering and Branigan 1998 , an effect predicted by the “lexical boost” in syntactic priming, whereby syntactic structures are primed more strongly when there is lexical overlap) . Ultimately, we find strong generalization in LMs across a range of filler–gap constructions, with effects observed at all positions within constructions. We observe lexical boost: effects are stronger when lexical items match (e.g., the same animacy). Moreover, we identify source constructions whose underlying mechanisms generalize broadly, as well as sink constructions that consistently benefit from such transferred mechanisms. Finally, we provide evidence that such generalization does not seem to extend across clausal boundaries. We claim these experiments make good on the promise that studying LMs can help us better understand linguistic structure and language learning in general by not just serving as proxies for data-driven learners, but by helping us develop linguistically interesting hypotheses ( Futrell and Mahowald 2025 ; Portelance and Jasbi 2024 ; Potts 2023 ) . 1 1 1 Code will be released with final version. Construction Prefix Filler NC Article NP Verb Label Emb. Wh-Question ( Know -Class) I know who/that the man liked ./him Emb. Wh-Question ( Wonder -Class) I wonder who/if the man liked ./him Matrix Wh-Question Who/"" did the man like ?/him Restrictive Relative Clause The boy who/and the man liked was/him Cleft It was the boy/clear that the man liked ./the boy Pseudo-Cleft Who/That the man liked was/it Topicalization Actually, the boy/"" the man liked ./the boy Subject-Verb Agreement The boy/boys that the man liked is/are Transitive/Intransitive Verbs Last night/Yesterday some/that man/boy ran/liked ./him Table 1: Exemplar minimal pairs for each evaluated construction’s single-clause, animate extraction variant. The filler/label combinations are used to evaluate whether the model is processing the construction correctly and whether our causal interventions are successful. The final two constructions are used as control conditions. For a full set of examples, including the multi-clause and inanimate extraction variants, see Appendix A . 2 Filler–Gaps and Neural Models Consider the following sentence: [ The bagel ]_{t} , I liked __ t . The embedded clause, I liked , seems incomplete, lacking an object. However, the sentence is grammatical, as the fronted entity the bagel is understood to be the object of the anteceding clause. Grammatical constructions of this nature are termed filler–gaps , due to constituents appearing as ‘fillers’ in non-canonical positions, colloquially being said to leave a ‘gap’ at its canonical position. This grammatical family encompasses a wide range of common constructions including wh -questions, relative clauses, clefts, and more. Filler–gap dependencies have long been a target of linguistic inquiry. They are believed to require sophisticated syntactic machinery, beyond simple surface statistics, since a word might appear quite linearly far from a word that it depends on for its meaning ( Chomsky 1957 ; Ross 1967 ) . They have been of interest in computational linguistics for the same reason: earlier models like n-gram models