LLMs as a synthesis between symbolic and continuous approaches to language

Gemma Boleda · 2025

Read LLMs as a synthesis between symbolic and continuous approaches to language by Gemma Boleda (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-11856.txt.

Introduction

Gemma Boleda argues that large language models heal an old split in language science in this 2025 position paper. For seventy years, symbolic accounts of language, built on discrete rules, have battled distributed accounts built on smooth neural patterns. Each camp reads deep learning's success as its own victory or as mere engineering.

Boleda's claim is synthesis. The architectures can hold both crisp, category-like structure and fuzzy, graded patterns, and trained models use that freedom. Work on interpretability shows grammar knowledge stored in near-discrete form inside the models. Behavior emerges as the models slide between modes as needed, which may explain their wild success.

Catalog record

CREATORGemma Boleda
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-11856

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LLMs as a synthesis between symbolic and continuous approaches to language Gemma Boleda Abstract Since the middle of the 20th century, a fierce battle is being fought between symbolic and continuous approaches to language and cognition. The success of deep learning models, and LLMs in particular, has been alternatively taken as showing that the continuous camp has won, or dismissed as an irrelevant engineering development. However, in this position paper I argue that deep learning models for language actually represent a synthesis between the two traditions. This is because 1) deep learning architectures allow for both continuous/distributed and symbolic/discrete-like representations and computations; 2) models trained on language make use this flexibility. In particular, I review recent research in mechanistic interpretability that showcases how a substantial part of morphosyntactic knowledge is encoded in a near-discrete fashion in LLMs. This line of research suggests that different behaviors arise in an emergent fashion, and models flexibly alternate between the two modes (and everything in between) as needed. This is possibly one of the main reasons for their wild success; and it is also what makes them particularly interesting for the study of language and cognition. Is it time for peace? 1 Introduction Since the middle of the 20th century, a fierce battle is being fought between two antagonistic approaches to language and cognition. Although the details vary, they can be broadly characterized as follows. Symbolic approaches use discrete formalisms to represent language. Examples in computational linguistics (CL) are POS tags, parse trees, and discrete word senses. 1 1 1 In early work, these approaches were paired with top-down processing of linguistic data, through rule-based systems defined by hand. In later work, the processing part has instead been data-driven: data is manually annotated according to a given representation system, and a processing algorithm is induced from the data via machine learning. The latter includes modern neural networks trained for, e.g., dependency parsing. Continuous approaches use distributed representations, in the form of high-dimensional algebraic objects such as vectors. In CL, static word embeddings ( Mikolov et al. 2013 , à la word2vec;) are a prime example. The debate has taken different forms in different fields; in cognitive science, this opposition has been dubbed classicism vs connectionism Buckner and Garson 2019 ; in AI, different terms are used by different authors Russell and Norvig 2020 ; in linguistics, the issues underlying the divide between generative and cognitive linguists are related to this debate Harris 1993 . The crux of the debate is that, across all these fields, some researchers focus on the rule-like behavior of language and cognition and others on its slippery nature. However, the fact that this debate exists might be a testimony to the fact that language and cognition are both symbolic (or discrete) and continuous (or fuzzy) —and everything in between (see Section 2 ). Figure 1: Non-linear functions such as the sigmoid provide the potential for both continuous and near-discrete behavior. Focusing on language, in this position paper I argue that modern LLMs support both continuous and (near-)discrete representations and processing, and thus are a synthesis between the two antagonistic positions. 2 2 2 I center the discussion on LLMs as the most widely adopted type of model, but in the discussion I will also include other models, such as neural machine translation models. I will signal when I do. This may seem a strange position to adopt, since neural networks undoubtedly fall in the continuous camp. However, something that is often overlooked in the debate is the fact that neural networks have the potential for (near-)discrete behavior. This potential comes from the non-linearities in their architecture Minsky and Papert 1988 . Take the sigmoid as an example (Figure 1 ): when its input falls near 0, the value passed on will be continuous; but when its input is larger or smaller, it will be quasi-binary. This allows networks to learn to combine its inputs in a way that leverages the two behaviors. Crucially, while neural network architectures allow for flexibility in behavior, what they will do with this potential in practice is an open question. The present paper is motivated by the fact that LLMs do seem to indeed exploit the potential for quasi-symbolic behavior with respect to language: A lot of recent work within interpretability provides evidence for near-discrete representations and processes, as discussed in Section 3 . What is more, these representations arise in an emergent fashion; LLMs learn to behave in a a quasi-symbolic fashion, because that allows them to perform better at linguistic tasks. This, in turn, may be one of the reasons for their amazing success at capturing natural language. 2 How discrete is language? Linguists have found symbolic formalisms useful across all main domains of language, such as phonology Chomsky and Halle 1968 ; Prince and Smolensky 1993 , morphosyntax Chomsky 1957 ; Bresnan 1982 ; Langacker 1987 ; Pollard and Sag 1994 ; Goldberg 1995 , semantics Montague 1974 ; Partee et al. 1990 ; Pustejovsky 1995 , and pragmatics Grice 1989 ; Sperber and Wilson 1995 . In this article, I will focus on morphosyntax and semantics. Work in morphosyntax posits for instance that words belong to different parts of speech (such as determiner, noun, or verb) and can stand in different syntactic relations (such as subject, object, or indirect object). Languages mark morphosyntax formally, and restrictions in the co-occurrence of linguistic units (morpheme, words, clauses) are governed by morphosyntactic properties. For instance, in English only verbs inflect for tense; and, in most, verbs past tense is signaled ty the suffix -ed (“follow/followed”). Similarly, only some verbs allow for indirect objects, and the indirect object in English is marked by the preposition to (see example ( 1 )). In many languages different units in the sentence display agreement Wechsler and Zlatić 2003 . Example ( 1 ) showcases how, in Spanish, there is gender and number agreement within the noun phrase: the highlighted suffix -a on the determiner and adjective mark feminine gender, in agreement with the noun’s lexical gender. Similarly, in English, subjects and verbs agree in number; in example ( 3 ), the singular subject (“A student”) cannot combine with a plural verb (“are”). (1) John gave/*prepared a drink to Mary (2) L a s partes interesad a s the.FEM.PL party.PL interested.FEM.PL ‘The interested parties’ (3) A student is/*are crossing the street In compositional semantics and the syntax-semantics interface, we find phenomena such as negation, where, in a sentential context, adding negation reverses polarity ( Zeijlstra 2007 , see example ( 4 )) , and anaphora, where syntactic constraints determine the shape of anaphoric pronouns: for instance, in ( 5 ), the pronoun “him” cannot refer to Mark Chomsky 1981 . (4) I will/will not come to lunch (5) Mark i combs himself i /*him i All of these phenomena are largely symbolic and discrete, in that there is no “in between” state: the choice between “is” and “are” is determined by the number of the subject; “not” is a like a binary switch for polarity in sentences; etc. However, even in this realm one only needs to scratch the surface for discreteness to break down. The border between parts

Read the complete text