Understanding Syntactic Generalization in Structure-inducing Language Models

David Arps, Hassan Sajjad, Laura Kallmeyer · 2025

Read Understanding Syntactic Generalization in Structure-inducing Language Models by David Arps, Hassan Sajjad, Laura Kallmeyer (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-07969.txt.

Introduction

David Arps, Hassan Sajjad, and Laura Kallmeyer study how structure inducing language models generalize syntax in this 2025 paper. These models learn language while building tree shaped sentence structures as a side product. The team trained three designs from scratch: Structformer, UDGN, and GPST. Training data covered English, German, and Chinese, plus synthetic bracket strings. The models were then judged on the trees they induced, on grammar acceptability tasks, and on how training unfolded.

No design won everywhere. The induced trees differed sharply between designs, even when task scores were close. GPST was the most consistent overall and handled long distance links in bracket strings best. Small models fed lots of synthetic data proved a handy testbed for basic questions. The authors argue that judging such models only by task scores hides what matters: the structures they actually learn.

Catalog record

CREATORDavid Arps, Hassan Sajjad, Laura Kallmeyer
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-07969

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Understanding Syntactic Generalization in Structure-inducing Language Models David Arps, Hassan Sajjad, Laura Kallmeyer Abstract Structure-inducing Language Models (SiLM) are trained on a self-supervised language modeling task, and induce a hierarchical sentence representation as a byproduct when processing an input. A wide variety of SiLMs have been proposed. However, these have typically been evaluated on a relatively small scale, and evaluation of these models has systematic gaps and lacks comparability. In this work, we study three different SiLM architectures using both natural language (English) corpora and synthetic bracketing expressions: Structformer Shen et al. 2021 , UDGN Shen et al. 2022 , and GPST Hu et al. 2024b . We compare them with respect to (i) properties of the induced syntactic representations (ii) performance on grammaticality judgment tasks, and (iii) training dynamics. We find that none of the three architectures dominates across all evaluation metrics. However, there are significant differences, in particular with respect to the induced syntactic representations. The Generative Pretrained Structured Transformer (GPST; Hu et al. 2024) performs most consistently across evaluation settings, and outperforms the other models on long-distance dependencies in bracketing expressions. Furthermore, our study shows that small models trained on large amounts of synthetic data provide a useful testbed for evaluating basic model properties. 1 Introduction Linguistic research has shown that sentence structure has many hierarchical properties ( Radford 2004 ; Van Valin 2005 , see, e.g.,) . While the majority of mainstream large language models process sentences in a strictly sequential fashion, over the past years, a number of language modeling architectures have been proposed that integrate self-supervised language modeling tasks with hierarchical sentence structure induction. These models have demonstrated competitive performance on NLP tasks, unsupervised parsing, and grammatical generalization benchmarks Hu et al. 2024b ; Shen et al. 2022 ; Momen et al. 2023 but many of their basic properties, demonstrating their strengths from learning a hierarchical structure, are yet underexplored. For instance, Williams et al. 2018 find that Yogatama et al. 2017 ’s RL-SPINN model falls back on trivial baseline syntactic structures, and Choi et al. 2018 ’s ST-GUMBEL induces structures that vary strongly when the model is trained repeatedly. In addition, many of the models are trained on relatively small datasets such as the PennTreebank ( Marcus et al. 1993 , 1M words,) or BLLIP ( Charniak et al. 2000 , 30M words,) , which leaves the question of their scalability to large datasets unanswered. This calls for a systematic comparison of self-supervised hierarchically biased language models to understand their strengths and limitations and their potential to becoming a competitive architecture for NLP applications and linguistic research. In this work, we aim to fill this gap by conducting a comprehensive study covering aspects of architecture, scalability, self-consistency over several training runs, and syntactic generalizations learned by the models. To this end, besides using English data, we provide new benchmark data for testing such models, based on formal languages that exhibit non-trivial syntactic structures . Furthermore, we choose three substantially different model architectures and compare them with respect to the syntactic structures they induce when being trained either on English data or on synthetic formal language data. As a result, this paper contributes a systematic evaluation, benchmark data and evaluation pipeline that inform the design and evaluation of structure-inducing language models . Concretely, we make several contributions. In Section 2 , we take stock of the different approaches that have been proposed to combine language modeling with constituency or dependency parsing. Out of these, we select three representative model architectures : The Unsupervised Dependency Graph Network ( Shen et al. 2022 , UDGN) relies on a bidirectional LSTM parser and constrains self-attention using non-projective dependency graphs. The StructFormer Shen et al. 2021 , on the other hand, extends a transformer encoder with an unsupervised CNN-based parser that constrains self-attention using projective dependency graphs. Both UDGN and StructFormer are trained on masked language modeling, and both induce probabilistic adjacency matrices of a syntactic dependency graph. The Generative Pretrained Structured Transformer ( Hu et al. 2024b , GPST) induces a binary constituency tree from an inside-outside autoencoder. Span representations from this autoencoder are fed into a generative transformer that generates a sequence of shift-reduce operations together with the text sequence. In Section 3 , we describe the English language datasets that we use for training these models. We also introduce several formal bracketing languages that we evaluate these models on. While bracketing languages Chomsky and Schützenberger 1959 are inherently less complex than natural language, they have several advantages that allow to evaluate SiLMs in closely controlled settings: They lack structural ambiguities, their syntactic structures are known, and they have relatively small vocabularies. To make up for the lack of ambiguities in Dyck- k languages of k bracket types, we introduce the Dyck- u language that mimics subject-verb agreement in natural languages , and includes an unspecified bracket type. In Section 5 , we evaluate the models with respect to several properties of the induced syntactic structures, and their syntactic generalizations . We find that, while the SiLMs are close on some aspects of performance, the formal languages provide a useful testbed for estimating the capabilities of SiLMs, and that GPST is most consistent across different evaluation aspects, and outperforms both transformers and other SiLMs n minimal pair evaluations . All SiLM architectures show significant variation in induced syntactic representations when retraining identical architectures on the same data . 1 1 1 Code and models available at https://github.com/davidarps/silm 2 Structure-Inducing Language Models We define Structure-inducing Language Models (SiLMs) as neural architectures that are trained on a self-supervised language modeling task, and produce some syntactic representation t_{x} as a byproduct when processing an input sequence x . The exact nature of t_{x} may differ between SiLM architectures, but all SiLMs share the property that t_{x} must be learned in an unsupervised way: No annotated syntactic trees are available during training. The nature of the self-supervised language modeling task may vary. We focus on autoregressive, masked language modeling tasks, and autoregressive language modeling preceded by bidirectional span encodings. The most crucial difference for building t_{x} is that in masked language modeling, bidirectional context is available for constructing token predictions and t_{x} ; while in autoregressive language modeling, both the token predictions and syntactic representations are built incrementally. The nature of the syntactic inductive bias shaping t_{x} , as well as the neural architecture that builds t_{x} and the neural LM architecture may also vary. In the following sections, we provide a brief overview over existing approaches, and describe in more detail the models we explore. 2.1 Related Work A wide range of work since the 1990s connects unsupervised parsing and language modeling tasks using neural architectures Sun et al. 1993 ; Chen 1995 ; Hihi and Bengio 1995 ; Schmidhuber 1991 . Existing work on the connection of unsupervised parsing and language modeling can be partitioned into several broad categories depending on the central backbone of the neural architectures used. These include approaches based on Transformers and self-attention, which utilize the fact that attention heads naturally learn to track syntactic relations Shen et al. 2021 ; He et al. 2024 ; Li et al. 2020a ; Li and Lu

Read the complete text