Exploring Graph Representations of Logical Forms for Language Modeling

Michael Sullivan · 2025

Read Exploring Graph Representations of Logical Forms for Language Modeling by Michael Sullivan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-14523.txt.

Introduction

Michael Sullivan argues that language models should read logic, not raw text, and builds one to prove it. A model fed formal logical structures starts with grammar and meaning already encoded, so it may learn from far less data. His 2025 paper introduces GFoLDS, a pretrained model over graph representations of logical forms.

Trained on the same data as a standard transformer, the graph model won downstream tasks by wide margins. The built-in structure lets it pick up complex patterns early instead of spending data on basics. Scaling checks suggest more size and more data would widen the lead. The work is a proof of concept with teeth. NLP researchers get evidence that representation, not just scale, buys efficiency.

Catalog record

CREATORMichael Sullivan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE73 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-14523

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Exploring Graph Representations of Logical Forms for Language Modeling Michael Sullivan Abstract We make the case for language models over logical forms (LFLMs), arguing that such models are more data-efficient than their textual counterparts. To that end, we introduce the G raph-based Fo rmal- L ogical D istributional S emantics (GFoLDS) prototype, a pretrained LM over graph representations of logical forms, as a proof-of-concept of LFLMs. Using GFoLDS, we present strong experimental evidence that LFLMs can leverage the built-in, basic linguistic knowledge inherent in such models to immediately begin learning more complex patterns. On downstream tasks, we show that GFoLDS vastly outperforms textual, transformer LMs pretrained on similar amounts of data, indicating that LFLMs can learn with substantially less data than models over plain text. Furthermore, we show that the performance of this model is likely to scale with additional parameters and pretraining data, suggesting the viability of LFLMs in real-world applications. 1 Introduction Although recent advances in LLMs have led to remarkable performance on a wide variety of benchmarks, the consistent improvements exhibited by SoTA LLMs are largely due to corresponding increases in model size Villalobos et al. 2022 ; Muennighoff et al. 2024 . Given the Chinchilla Scaling Laws Hoffmann et al. 2022 and the rate at which SoTA LLMs are expanding, Villalobos et al. 2022 estimate that high-quality English training data will be exhausted at some point between 2026 and 2032: language model expansion is outpacing available natural language production. This suggests that—without models that use significantly less data than current approaches—LLMs’ performance increases will begin to decelerate substantially in the near future. However, there is a considerable amount of evidence in the literature (e.g. Xu et al. 2021 ; Wu et al. 2021b ; Prange et al. 2022 ; Sachan et al. 2021 ; Zhou et al. 2020 ; Zhang et al. 2020 ; Zhang et al. 2022 , etc.) indicating that linguistically-informed LMs—models whose inputs and/or architectures are augmented by linguistic knowledge—can improve performance without consuming more (text) data. In this paper, we argue for the use of LMs over logical forms (LFLMs): LMs that take as input semantic representations, rather than text. In particular, we posit the following hypothesis: Figure 1: DMRS representation of the sentence “ every bored person saw her yesterday .” The Linguistic-Knowledge Catalysis Hypothesis (LKCH): The (aspects of) linguistic knowledge incorporated into LFLMs obviates the need to learn elementary linguistic phenomena, allowing them to immediately begin learning more complex patterns. One relevant corollary of the LKCH is that LFLMs can learn with less data: the basic linguistic knowledge built into LFLMs facilitates more rapid learning of advanced phenomena. To make the case for LFLMs, we introduce the G raph-based Fo rmal- L ogical D istributional S emantics (GFoLDS; Section 3 ) model 1 1 1 All code used in this paper available on GitHub: https://github.com/mjs227/GFoLDS as a proof-of-concept of such LMs. GFoLDS is a pretrained, encoder graph transformer Wu et al. 2021a over structures derived from Dependency Minimal Recursion Semantics ( Copestake 2009 , DMRS;) representations (see Figure 1 ): directed, labeled acyclic graphs in which nodes correspond to predicates and edge labels denote relations (i.e. semantic roles) between them, in a similar manner to Abstract Meaning Representation ( Banarescu et al. 2013 , AMR;) structures. Unlike AMR, which abstracts away from semantic features such as tense and number, DMRS includes these features, resulting in a more faithful representation of linguistic meaning. The contributions of this work are threefold: firstly, we provide experimental support towards the validity of the LKCH , demonstrating that—from the start of pretraining—GFoLDS achieves near-peak performance on tasks designed to evaluate its elementary linguistic knowledge, and that this translates to more rapid learning of complex phenomena (Section 4 ). Secondly, we demonstrate the viability of pretrained LFLMs in Section 5 , by comparing the performance of GFoLDS to that of BERT ( Devlin et al. 2019 , trained on \sim 6.5 times more data than GFoLDS;) on a range of downstream tasks. Although the actual BERT models outperform GFoLDS, our model outperforms—by a wide margin—BERT models pretrained on the same data as GFoLDS, indicating that LFLMs can learn useful representations with much less data than their textual counterparts. Thirdly, we establish that LFLMs have the potential to compete with textual LLMs at scale: in Section 6 , we present evidence indicating that GFoLDS is likely to scale with respect to parameter count and pretraining dataset size. 2 Related Work There exists a body of research indicating that injecting graph representations of linguistic structures into textual LMs can improve downstream performance. Xu et al. 2021 fuse dependency parse graphs into pretrainined transformer encoders ( Liu et al. 2019 , e.g. BERT and RoBERTa;) , and surpass the then-SoTA results on relation classification, entity typing, and question answering tasks, demonstrating the general utility of linguistically-informed LMs. Similarly, Wu et al. 2021b inject syntactic dependency parse graphs into a pretrained BERT model, yielding then-SoTA results on semantic role labeling and relation extraction tasks. These authors additionally compare the respective impacts of syntactic and semantic representations on performance, and find that semantic representations are more beneficial with respect to downstream performance than syntactic structures. Prange et al. 2022 show that linguistic structures can be incorporated into the input of GPT-2 ( Radford et al. 2018 ) to improve next word prediction accuracy and entropy. Of particular relevance is their finding that Elementary Dependency Structures ( Oepen and Lønning 2006 , EDS;) —a semantic framework that is related to DMRS—yield greater performance improvements than syntactic (or other semantic) representations. All of the models described thus far in this section are hybrid architectures that merge textual and graph representations. Furthermore, in these approaches, the textual component of the model is initialized from a pretrained LM such as BERT, RoBERTa, or GPT-2. This contrasts with the GFoLDS model, which takes only graph representations as input and is pretrained from scratch. To the best of our knowledge, Functional Distributional Semantics at Scale ( Lo et al. 2023 , FDSAS;) represents the only 2 2 2 Aside from its predecessor: Functional Distributional Semantics Emerson 2018 . extant LFLM aside from GFoLDS. FDSAS is a variational autoencoder over DMRS graphs that learns probabilistic truth-conditional functions for each predicate (node label). FDSAS is not, however, a transformer and so does not benefit from the flexibility afforded by such architectures: it is unclear how to scale the model to yield a deeper architecture, and it does not generate usable hidden states that can be passed to (for example) a classification head for downstream tasks. Additionally, its rigorous formal-semantic foundations give rise to a degree of inflexibility in this model: Lo et al. 2023 discarded prepositions, quantifiers, and modal verbs from FDSAS’ training dataset out of necessity. 3 GFoLDS In this section, we describe the GFoLDS model architecture (Section 3.1 ), preprocessing steps that we employed on the DMRS graphs (Section 3.2 ), and the model’s pretraining procedure (Section 3.3 ). 3.1 Architecture Figure 2: Top-level architecture of the GFoLDS model (top) and positional encoding network (bottom). The GFoLDS model is a variant of the graph transformer paradigm Wu et al. 2021a , which was originally introduced for molecule

Read the complete text