Learning the Wrong Lessons: Syntactic-Domain Spurious Correlations in Language Models
Chantal Shaib, Vinith M. Suriyakumar, Levent Sagun, Byron C. Wallace, Marzyeh Ghassemi · 2025
Read Learning the Wrong Lessons: Syntactic-Domain Spurious Correlations in Language Models by Chantal Shaib, Vinith M. Suriyakumar, Levent Sagun, Byron C. Wallace, Marzyeh Ghassemi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-21155.txt.
Introduction
A model's answer can be swayed by the grammatical shape of a question rather than its meaning. Chantal Shaib and four co-authors document that failure in a 2025 paper selected as a NeurIPS 2025 spotlight. They study syntactic templates, recurring part-of-speech patterns in text.
Such templates are common in training data and leak into model outputs. The authors characterize how syntax, domain, and semantics interact in task-instruction pairs. In some training data, a syntactic template correlates spuriously with a subject domain. Models then learn the association and let syntax override the prompt's actual semantics. Using a synthetic training set, they show the correlation lowers performance. The lesson is that dataset builders must watch grammatical patterns, not only content, when curating instruction data.
Catalog record
| CREATOR | Chantal Shaib, Vinith M. Suriyakumar, Levent Sagun, Byron C. Wallace, Marzyeh Ghassemi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-21155
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Learning the Wrong Lessons: Syntactic-Domain Spurious Correlations in Language Models Chantal Shaib, Vinith M. Suriyakumar, Levent Sagun, Byron C. Wallace, Marzyeh Ghassemi Abstract For an LLM to correctly respond to an instruction it must understand both the semantics and the domain (i.e., subject area) of a given task-instruction pair. However, syntax can also convey implicit information Recent work shows that syntactic templates —frequent sequences of Part-of-Speech (PoS) tags—are prevalent in training data and often appear in model outputs. In this work we characterize syntactic templates, domain, and semantics in task-instruction pairs. We identify cases of spurious correlations between syntax and domain, where models learn to associate a domain with syntax during training; this can sometimes override prompt semantics. Using a synthetic training dataset, we find that the syntactic-domain correlation can lower performance (mean 0.51 ± 0.06 \pm 0.06 ) on entity knowledge tasks in OLMo-2 models (1B-13B). We introduce an evaluation framework to detect this phenomenon in trained models, and show that it occurs on a subset of the FlanV2 dataset in open (OLMo-2-7B; Llama-4-Maverick), and closed (GPT-4o) models. Finally, we present a case study on the implications for safety finetuning, showing that unintended syntactic-domain correlations can be used to bypass refusals in OLMo-2-7B Instruct and GPT-4o. Our findings highlight two needs: (1) to explicitly test for syntactic-domain correlations, and (2) to ensure syntactic diversity in training data, specifically within domains, to prevent such spurious correlations. Content Warning: This paper contains examples of harmful language. 1 Introduction LLMs are increasingly being deployed in a variety of important domains zhang2025healthcare ; smith2024finance ; garcia2024law ; chen2023science . To be reliable in such areas they should understand the semantics of instructions and the domains of the tasks gururangan2020dont , yet recent work shows that the correctness of their responses can change when encountering subtle linguistic variations in instructions sun2023evaluating ; cao2024worst . This raises a fundamental question: Do LLMs leverage semantic and domain knowledge, or are they only exploiting shallow syntactic patterns from their training data at scale? Syntax, semantics, and domain interact together to provide structure, meaning, and content for accurate language interpretation and task execution chomsky2002syntactic ; chomsky2014aspects ; goldberg2024usage . While the latter two are well-studied in LLMs moskvoretskii2024are ; jha2024rosetta ; chen2023domain , the former has only recently gained traction as an important component for understanding writing idiosyncrasies in model responses shaib2024detection ; sun2025idiosyncrasies ; qi2024safety . Recent work has shown that model training data contains syntactic templates , i.e., frequently appearing n -grams of part-of-speech in training data, and that these templates are learned and subsequently repeated by the model ( shaib2024detection ; wadhwa2025taughtthattracingteachers ; fang2025knowledge ) . Ideally, LLMs should not rely on syntax cues alone to perform a task: Semantic and domain (e.g., subject matter) ‘‘understanding’’ should also inform generated responses. 1 1 1 In practice, it is difficult to completely separate syntax from semantics. In this work, we delineate these concepts by focusing on a specific subset of syntactic structures based on part-of-speech tags ( syntactic templates ), which can plausibly map to many meanings. In this work we systematically study the interactions between syntactic templates and domain, and find evidence that LLMs learn to rely on a spurious correlation between these features to improve performance. Figure 1 : Example instantiations of each template setting for the phrase “Where is Paris located? France", where (Paris, France) is the entity pair denoting the domain country . Each template setting modifies either syntax, domain, or semantics. If a model answers “France” in the antonym or disfluent settings, this may be due to over reliance on syntax. Figure 1 introduces our canonical example. We ask a trained model “Where is Paris located?” and it correctly answers with “ France .” This instruction-task pair has the syntactic template of adverb, verb, proper noun, verb(past participle), proper noun , a country domain, and a meaning indicating that we are asking for fact-completion . In the most extreme case, if we modify the domain and semantics (substituting antonyms, random words) while keeping the template fixed, we would not expect the model to respond with the correct answer. Surprisingly, however, when prompting the model with an incoherent sentence like “Quickly sit Paris clouded?”, the model still responds with “ France .’’ 2 2 2 One may question whether this just indicates entity memorization ; we discuss this in § 4 . When a model responds with the correct response in the antonym or disfluent settings, we posit that this is indicative of a spurious correlation between between syntax and domain. This may be problematic when the correlation degrades performance in a new domain. For instance, responding with “ France ” when asking about the food in “ Paris ”. Building on this observation, we first formally characterize the relationship between syntactic templates, semantics, and domain knowledge in LLM instruction-following settings (§ 2 ). We then empirically show that OLMo-2 ( olmo20242 ; 1B, 7B, and 13B) learns spurious syntactic-domain correlations that degrade semantic understanding using synthetic data sourced from Wikipedia (§ 3 ). Our quasi-synthetic experiments inform a benchmarking procedure to detect this failure mode in trained models, which we release for community use (§ 4 ), and use to demonstrate that spurious syntactic-domain correlations occur in off-the-shelf OLMo-2, Llama-4-Maverick 3 3 3 Accessed at https://www.together.ai/models/llama-4-maverick and GPT4o models openai2024gpt4ocard (§ 5 ). Finally, we demonstrate domain-syntax correlations impact LLM security by giving rise to a new exploit for bypassing model refusals to harmful requests in OLMo-2-7B-Instruct and GPT4o-mini (§ 6 ). 2 Syntactic-Domain Spurious Correlations Problem Statement In Figure 1 , we first ask “Where is Paris located?”, where the correct answer is “ France ” and falls into the in-domain category of geography . Then we ask a second question: “What food do they eat in Paris ?”and the correct answer is “ croissant ”. This prompt is a new, cross-domain category of cuisine . Both prompts here follow unique syntactic templates , t_{\texttt{geography}} and t_{\texttt{cuisine}} respectively, defined by part-of-speech (PoS) tags. To answer correctly, there are two possible behaviours: the model can answer the first question by only recognizing the syntactic template associated with t_{\texttt{geography}} , or, can leverage information about the combination of semantics, domain, and content. The first case, however, can be brittle: if the model only relies on syntax, it may perform poorly when encountering an input with template t_{\texttt{geography}} but in the cuisine domain. To formalize this intuition, consider a dataset X where we represent each example as a triple x=(d,t,e) , where: • d is a domain of knowledge (e.g., geography , cuisine ), • t is a syntactic template determined by the sentence structure of the prompt-response pair, • e contains the entities that comprise the semantic content (e.g., Paris, France). Our dataset X spans domains \mathcal{D} (i.e. distributions over groups of sequences), with each domain d associated with frequent syntactic templates \mathcal{T}_{d} , often the most common n -grams or PoS patterns. For simplicity in § 3 , we assume minimal overlap between domain template sets, confirmed via bigram similarity analysis (Appendix Fig. 7 ). In practice, however, this assumption is not required—our results hold regardless. Strong domain-template associations risk models