Implicit Representations of Grammaticality in Language Models

Yingshan Susan Wang, Linlu Qiu, Zhaofeng Wu, Roger P. Levy, Yoon Kim · 2026

Read Implicit Representations of Grammaticality in Language Models by Yingshan Susan Wang, Linlu Qiu, Zhaofeng Wu, Roger P. Levy, Yoon Kim (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-05197.txt.

Introduction

Yingshan Susan Wang and four co-authors ask whether language models inside separate grammatical sentences from ungrammatical ones, beyond merely judging which string is more likely. Models generate well-formed text and win tightly controlled comparisons, yet raw probabilities do not sharply split grammatical from ungrammatical strings overall. The authors probe their inner states with a simple linear probe instead.

Trained on natural text and made-up ungrammatical variants, the probe also worked on human-curated judgment benchmarks and beat score-based judgements. On plausibility tests, where both sentences are well-formed, the probe did worse than raw probability, as expected for a grammar-only signal. An English-trained probe also transferred well to grammar tests in other languages.

Catalog record

CREATORYingshan Susan Wang, Linlu Qiu, Zhaofeng Wu, Roger P. Levy, Yoon Kim
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.05197

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Implicit Representations of Grammaticality in Language Models Yingshan Susan Wang, Linlu Qiu, Zhaofeng Wu, Roger P. Levy, Yoon Kim Abstract Grammaticality and likelihood are distinct notions in human language. Pretrained language models (LMs), which are probabilistic models of language fitted to maximize corpus likelihood, generate grammatically well-formed text and discriminate well between grammatical and ungrammatical sentences in tightly controlled minimal pairs. However, their string probabilities do not sharply discriminate between grammatical and ungrammatical sentences overall. But do LMs implicitly acquire a grammaticality distinction distinct from string probability? We explore this question through studying internal representations of LMs, by training a linear probe on a dataset of grammatical and (synthetic) ungrammatical sentences obtained by applying perturbations to a naturalistic text corpus. We find that this simple grammaticality probe generalizes to human-curated grammaticality judgment benchmarks and outperforms LM probability-based grammaticality judgments. When applied to semantic plausibility benchmarks, in which both members of a minimal pair are grammatical and differ in only plausibility, the probe however performs worse than string probability. The English-trained probe also exhibits nontrivial cross-lingual generalization, outperforming string probabilities on grammaticality benchmarks in numerous other languages. Additionally, probe scores correlate only weakly with string probabilities. These results collectively suggest that LMs acquire to some extent an implicit grammaticality distinction within their hidden layers. 1 1 1 The code for this project is available at https://github.com/SusanWYS/grammaticality_probe . 1 Introduction Figure 1: The grammaticality probe is a linear classifier over LM hidden states , trained on a synthetic dataset created from perturbing sentences from a generic text corpus. For evaluation, we measure how well probe scores and LM logprob discriminate between grammatical and ungrammatical sentences on acceptability-judgment corpora. As a baseline, we also evaluate the grammaticality probe on semantic plausibility datasets. The probe is more discriminative for syntactic acceptability than LM logprobs, whereas LM logprobs are more sensitive to semantic plausibility. In human language, a sentence’s grammaticality —whether it conforms to the rules of language—is distinct from its likelihood —whether it is likely to occur in naturalistic text. Consider the following pair of examples from Levy et al. 2025 (itself adapted from Bock and Miller 1991 ): \ex . *The key to the cabinets are on the table. \ex . The key to the cabinets was so thoroughly rusted that it was impossible to fit into the keyhole. The first sentence violates the rule of English that the number of the verb are should match that of the singular head noun of the subject key , and is therefore ungrammatical. However, this type of error is common when a plural noun ( cabinets ) intervenes before the verb, so the sentence is almost certainly more likely to be used than the second sentence, which is grammatical but expresses a complex and unusual meaning. Insofar as this grammaticality–likelihood distinction —a consequence of the broader competence–performance distinction —is present in humans, the extent to which LMs are able to capture this distinction bears directly on their viability as candidate computational models of human language processing and acquisition—a matter of considerable ongoing debate in linguistics and cognitive science ( Piantadosi 2023 ; Katzir 2023 ; Millière 2024 ; Fox and Katzir 2024 ; Futrell and Mahowald 2025 , i.a. ) . LMs are at their core statistical systems that assign probabilities to strings, and thus naturally capture a notion of likelihood—they are literally trained to maximize the likelihood of the training data. However, their training data inevitably contain noise arising from (amongst others) performance factors, and sentences such as (1) occur frequently in naturalistic text. Thus, LMs do not (nor are they designed to) categorically assign lower probabilities to ungrammatical sentences than grammatical ones; indeed, Levy et al. 2025 note that GPT-2 assigns 146 trillion times more probability to (2) than to (1), and prior studies find that LM probabilities are generally poor at distinguishing grammatical from ungrammatical sentences ( Leivada et al. 2024a ; Leivada et al. 2024b ; Leivada et al. 2025 ) . Hu et al. 2025 further show that string probabilities alone are theoretically inappropriate quantities for extracting grammaticality judgments under a simple generative model of language. Nevertheless, LM probabilities have been shown to be effective at adjudicating grammaticality across pairs of sentences that differ from one another minimally ( Marvin and Linzen 2018 ; Futrell et al. 2019 ; Warstadt et al. 2020 ; Hu et al. 2020 , i.a. ) , indicating that while the grammaticality–likelihood distinction is not explicitly baked into LMs, they acquire behavioral generalizations that indicate some knowledge of a grammar. But as Leivada et al. 2024a note, this type of grammaticality assessment based on minimal pairs is different from the standard by which we assess grammaticality judgments in humans. However, while string probabilities from LMs are not designed to (only) encode grammaticality, in neural LMs these string probabilities are the result of internal computations performed by the model. In the present work we thus explore the hypothesis that LMs, through large-scale training, learn to implicitly encode grammaticality judgments within their hidden representations. 2 2 2 Metalinguistic judgments, wherein an LM is explicitly asked to answer whether a sentence is grammatical or not, provides another perspective on the grammaticality–likelihood distinction in LMs. While this approach ostensibly mirrors how we might extract grammaticality judgments from adult native speakers, Hu and Levy 2023 observe that metalinguistic judgments can underestimate a model’s linguistic capability. Moreover, this type of assessment assumes that an LM is able to perform basic question-answering out of the box, which may not be the case for many LMs of interest (e.g., smaller LMs that have not been instruction-tuned). Hence, this work does not use it as a major baseline. A standard approach to testing such hypotheses is through supervised probes, wherein a simple model (e.g., a linear classifier) is trained to predict some phenomena of interest ( Alain and Bengio 2017 ; Belinkov 2022 ) . In the case of grammaticality judgments, this might involve training a probe on acceptability judgment datasets on human-curated datasets such as CoLA ( Warstadt et al. 2019 ) or BLiMP ( Warstadt et al. 2020 ) . However, such datasets are often small and costly to collect, and thus probes trained on them could overfit; ideally, such datasets should be used as held-out sets for evaluation. Perturbation Original Perturbed Insertion Analysts agreed. Democratization Analysts constructed agreed. “They had to do it.” “They had to Krenz do it.” She bursts into tears and walks away. She bursts into tears fringe and walks away. Deletion What is his usual food , Nelly? What his , Nelly? Be a good lad; and I’ll do for you. Good lad; I’ll you. They allowed this country to be credible. They allowed to be credible. Local shuffle It was jolly of you to make up your mind to come. It was of you to make jolly up your mind to come. Now we shall have some discussion , we’ll see to that. Now have discussion shall we some , we’ll see to that. Have you been reading Spencer ? you reading Spencer Have been ? Table 1: Original sentences and their perturbed counterparts for three perturbation types. We propose an approach for learning a grammaticality

Read the complete text