GiLT: Augmenting Transformer Language Models with Dependency Graphs
Tianyu Huang, Yida Zhao, Chuyan Zhou, Kewei Tu · 2026
Read GiLT: Augmenting Transformer Language Models with Dependency Graphs by Tianyu Huang, Yida Zhao, Chuyan Zhou, Kewei Tu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-15562.txt.
Introduction
Tianyu Huang, Yida Zhao, Chuyan Zhou, and Kewei Tu add dependency graphs to transformer language models. Graphs link words by grammar ties. Past work used tree shapes, chiefly constituency trees. Their GiLT feeds graph traits into attention weights as the graph grows with token guesses. No extra structure tokens are added. With semantic graphs it lifts syntactic generalization and keeps perplexity in step with rivals. Tuning from a pretrained model also lifts downstream task scores.
Catalog record
| CREATOR | Tianyu Huang, Yida Zhao, Chuyan Zhou, Kewei Tu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38948 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.15562
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
GiLT: Augmenting Transformer Language Models with Dependency Graphs Tianyu Huang, Yida Zhao, Chuyan Zhou, Kewei Tu Abstract Augmenting Transformers with linguistic structures effectively enhances the syntactic generalization performance of language models. Previous work in this direction focuses on syntactic tree structures of languages, in particular constituency tree structures. We propose Graph-Infused Layers Transformer Language Model (GiLT) which leverages dependency graphs for augmenting Transformer language models. Unlike most previous work, GiLT does not insert extra structural tokens in language modeling; instead, it injects structural information into language modeling by modulating attention weights in the Transformer with features extracted from the dependency graph that is incrementally constructed along with token prediction. In our experiments, GiLT with semantic dependency graphs achieves better syntactic generalization while maintaining competitive perplexity in comparison with Transformer language model baselines. In addition, GiLT can be finetuned from a pretrained language model to achieve improved downstream task performance. Our code is released at https://github.com/cookie-pie-oops/GiLT-LM . 1 Introduction Transformer language models (LMs) have shown excellent performance in language modeling and downstream tasks ( 30 ) . Notably, linguistic structures such as syntactic and semantic parses that have been deemed essential in traditional natural language processing are absent from the model design and training process of Transformer LMs. Over the past decade, a number of researchers have been trying to integrate linguistic structures into neural language models. Among them are syntactic LMs which jointly model syntactic structures and surface words ( 4 ) . These include earlier work such as RNNG, which combines constituency parsing with recurrent neural networks ( 9 ; 16 ; 21 ) , and recent studies that incorporate constituency and dependency syntax into Transformers ( 35 ; 26 ; 28 ; 20 ; 36 ) . Empirically, they achieve stronger syntactic generalization compared with standard Transformer LMs while retaining competitive language modeling performance. However, existing researches in this direction have two major limitations. First, most of them are based on constituency syntactic tree structures. Dependency tree structures, another important form of syntax, receive much less attention ( 36 ) . In addition, little work has been done to jointly model linguistic structures other than syntactic trees in LMs. Second, most of the existing methods require inserting additional tree-building operations into the input and output sequence, leading to longer sequence lengths and higher computational cost, and making it harder to finetune a pretrained LM into a syntactic LM. An exception is Pushdown Layers ( 20 ) , which leverages syntactic trees to guide attention computation without changing the LM’s symbol space. In this paper, we propose the Graph-Infused Layer Transformer LM (GiLT) that addresses the above-mentioned limitations in integrating linguistic structures into Transformer LMs. GiLT is based on dependency graphs that subsume both syntactic dependency trees and semantic dependency graphs, thus extending syntactic LM research beyond syntax. Inspired by Pushdown Layers ( 20 ) , GiLT incrementally constructs dependency graphs without changing the symbol space of the underlying LMs, and modulates attention scores with features computed from graph attributes such as node degrees, depths and distances. Experimental results show that GiLT achieves gains in syntactic generalization over baselines with almost no degradation in perplexity on language modeling. Furthermore, GiLT finetuned from a pretrained GPT2 achieves better performance on downstream tasks compared with the original pretrained GPT2, suggesting that the Graph-Infused layer is a competitive alternative to standard self-attention. In summary, our contributions are as follows: • We propose Graph-Infused Layers , leveraging dependency graphs to enhance LMs by our novel graph-based feature tapes without modifying the input or output space. • Comprehensive experiments on language modeling, syntactic evaluation and finetuning on text classification demonstrate competitive perplexity, improved syntactic generalization and language understanding. Ablation study on feature tapes shows the importance of each part and test on generation speed illustrates the advantage of not requiring extra tokens. 2 Background 2.1 Pushdown Layers Transformer LM with Pushdown Layers ( 20 , Pushdown-LM,) is a type of syntactic LMs that incrementally builds constituency syntactic trees and modulates attention scores based on the constituency trees. Unlike other syntactic LMs, it does not change the symbol space of the underlying LM. At each decoding step i , Pushdown-LM predicts shift/reduce operations to simulate the status of a pushdown automaton that corresponds to the partially built constituency tree, and records on a stack tape \mathbf{t}_{i} the depths of all the tokens that are already generated in the partially built constituency tree. Pushdown-LM then augments self-attention with stack tape \mathbf{t}_{i} : \tilde{\alpha}^{l}_{ij}=[\mathbf{h}^{l}_{j}+\mathbf{d}_{ij}^{l}]^{\top}\mathbf{W}_{k}^{\top}\mathbf{W}_{q}\mathbf{h}_{i}^{l} (1) where \tilde{\alpha}_{ij}^{l} is the attention score before softmax assigned to the j -th token from the i -th token at layer l , \mathbf{h}^{l}_{j} is the hidden state of j -th token at the l -th attention block, \mathbf{d}_{ij}^{l} is the embedding of the depth of the j -th token recorded in \mathbf{t}_{i} , and \mathbf{W}_{k} and \mathbf{W}_{q} are learnable query and key matrices in self-attention. In this way, structural information from the constituency tree is implicitly introduced into self-attention computation and thus influences the decoding of the underlying LM. 2.2 Semantic Dependency Graphs A semantic dependency graph forms as a directed acyclic graph instead of a tree. The dependencies in the graph, where nodes correspond to words, illustrate semantic relations ( e.g. , agent and patient in 23 ). The graph often includes a virtual root node. In this paper, we consider three types of semantic dependency graphs from 22 as discussed below. DELPH-IN MRS-Derived Bi-Lexical Dependencies ( 11 , DM,) are derived from Deep Bank 10 , in which roots designate the highest-scoping predicate in the graph. Enju Predicate–Argument Structures (PAS) originate from Enju Treebank 19 , which is obtained by automatically annotating the PTB. The root of PAS denotes the semantic head in the sentence. Prague Semantic Dependencies (PSD) are based on Prague Czech-English Dependency Treebank 12 , where the roots mostly correspond to main verbs. 3 Graph-Infused Layers We introduce a dependency-graph-based language model, Graph-Infused Layers Transformer LM (GiLT), which simultaneously generates tokens that form sentences, and dependencies that incrementally construct dependency graphs over the sentences. We first score possible dependencies that link the current word to previous words (Section 3.1 ), then update the dependency graph based on the scoring (Section 3.2 ), and utilize the graph-based feature tapes (Section 3.3 ), which characterize generated tokens in the graph, to modulate attention computation (Section 3.4 ). 3.1 Dependency Scoring Whenever a word w_{i} is generated by the Transformer LM, we score all possible dependencies connected from and to w_{i} with a biaffine mechanism. Since a word may correspond to multiple tokens, we first define the word-level representations that serve as input to the biaffine module. Suppose a word w_{i} is tokenized into m tokens with input embeddings \{\mathbf{x}_{k},\cdots,\mathbf{x}_{k+m-1}\} and corresponding hidden states \{\mathbf{h}_{k}^{l},\cdots,\mathbf{h}_{k+m-1}^{l}\}\subseteq\mathbb{R}^{d} from all layers l=1,\dots,L . We define its word-level representation \mathbf{o}_{i}\in\mathbb{R}^{3d} by concatenating three components: (i) the hidden state from the middle layer, \mathbf{h}_{k-1}^{L/2} ; (ii) the hidden state from the penultimate layer, \mathbf{h}_{k-1}^{L-1} ; (iii) the input embedding of the first token, \mathbf{x}_{k} , which provides direct lexical information about the word. According to the assumption in 20 , since \mathbf{h}_{k-1}^{L/2} and \mathbf{h}_{k-1}^{L-1} are hidden states from sufficiently deep layers used to predict