The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Models

Nora Graichen, Iria de-Dios-Flores, Gemma Boleda, Universitat Pompeu Fabra, ICREA*Equal senior authorship. [email protected] · 2026

Read The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Models by Nora Graichen, Iria de-Dios-Flores, Gemma Boleda, Universitat Pompeu Fabra, ICREA*Equal senior authorship. [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-19926.txt.

Description

Nora Graichen, Iria de-Dios-Flores, and Gemma Boleda review what is known about grammar inside language models. Transformer models handle syntax well in practice, but how they represent it is debated. Interpretability research has attacked the question from many angles. A systematic map of that literature was missing.

Their review covers several hundred articles evaluating the syntactic abilities of transformer models. It organises over a thousand model results across syntactic phenomena and interpretability methods. Patterns emerge about which methods agree and which phenomena are robustly captured. Such stock-taking helps researchers avoid repeating settled experiments. It also highlights where claims about machine grammar rest on thin evidence. The stock take is broad and fair.

Catalog record

CREATORNora Graichen, Iria de-Dios-Flores, Gemma Boleda, Universitat Pompeu Fabra, ICREA*Equal senior authorship. [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-19926

Browse the full library · About the Tower of Babel