First numerical observation of the Berezinskii-Kosterlitz-Thouless transition in language models

Yuma Toji, Jun Takahashi, Vwani Roychowdhury, Hideyuki Miyahara · 2024

Read First numerical observation of the Berezinskii-Kosterlitz-Thouless transition in language models by Yuma Toji, Jun Takahashi, Vwani Roychowdhury, Hideyuki Miyahara (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-01212.txt.

Introduction

Yuma Toji and co-authors report a phase change inside a simple language model in a 2024 paper. Natural languages show power law patterns like those near phase changes in physics. Large models added fresh hints of such links. Still, no clear case of a true phase change in a language model had been shown.

The team builds a random language model shaped by context rules, echoing a classic spin model. A tuned measure of symbol bias jumps from zero to nonzero as a knob turns. That jump marks a sharp change point in the endless length limit. The change matches a famous vortex type pattern known for odd critical traits across a whole phase.

Catalog record

CREATORYuma Toji, Jun Takahashi, Vwani Roychowdhury, Hideyuki Miyahara
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-01212

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

First numerical observation of the Berezinskii-Kosterlitz-Thouless transition in language models Preprint: APS/123-QED Yuma Toji, Jun Takahashi, Vwani Roychowdhury, Hideyuki Miyahara Abstract Several power-law critical properties involving different statistics in natural languages – reminiscent of scaling properties of physical systems at or near phase transitions – have been documented for decades. The recent rise of large language models (LLMs) has added further evidence and excitement by providing intriguing similarities with notions in physics such as scaling laws and emergent abilities. However, specific instances of classes of generative language models that exhibit phase transitions, as understood by the statistical physics community, are lacking. In this work, inspired by the one-dimensional Potts model in statistical physics we construct a simple probabilistic language model that falls under the class of context sensitive grammars (CSG), and numerically demonstrate an unambiguous phase transition in the framework of a natural language model. We explicitly show that a precisely defined order parameter – that captures symbol frequency biases in the sentences generated by the language model – changes from strictly 0 to a strictly nonzero value (in the infinite-length limit of sentences), implying a mathematical singularity arising when tuning the parameter of the stochastic language model we consider. Furthermore, we identify the phase transition as a variant of the Berezinskii-Kosterlitz-Thouless (BKT) transition, which is known to exhibit critical properties not only at the transition point but also in the entire phase. This finding leads to the possibility that critical properties in natural languages may not require careful fine-tuning nor self-organized criticality, but is generically explained by the underlying connection between language structures and the BKT phases. From a statistical physics perspective, our paper captures a rare phenomenon of the BKT phase in a one-dimensional system; the BKT phase is widely studied and believed to exist only in two-dimensional systems. Introduction. – Natural languages exhibit striking statistical regularities, such as Zipf’s law, which describes a power-law relationship between word frequencies and their rank Piantadosi 2014 . Similarly, it has been shown that the number of bits of information provided by a symbol about another symbol drops roughly as a power law as a function of their distance Lin and Tegmark 2017 . These critical properties, observed across diverse languages, have inspired explanations rooted in statistical physics, drawing parallels between linguistic phenomena and phase transitions. For example, the scaling behavior inherent in Zipf’s law suggests underlying mechanisms that resemble those driving critical phenomena in physical systems Simkin and Roychowdhury 2011 . Recent advances in generative language models, including large language models (LLMs) OpenAI 2022 ; Pichai and Hassabis 2023 ; Naveed et al. 2023 ; Hadi et al. 2024 , have further highlighted intriguing connections between language and physics. Notably, studies have uncovered scaling laws and emergent capabilities in these models, which apparently become evident as the system size increases Kaplan et al. 2020 ; Wei et al. 2022 ; Chen et al. 2023 . These findings have prompted the question of whether the statistical properties of natural languages and the behavior of their generative models can be understood as manifestations of genuine phase transitions in physical terms. These observed phenomena in LLMs, combined with the fact that they become apparent when some parameters of the system are large enough (network depth, width, training data set size, etc.), evoke parallels to phase transitions in the field of statistical physics. The fact that in critical (continuous) phase transitions many quantities such as the correlation function exhibit power-law scaling also fuels the approach to study these emergent phenomena in languages from a statistical physics perspective. The notion of phase transitions in the thermodynamic limit, i.e., in an idealized limit where some parameter governing the system size tends to infinity, enables us to study qualitative differences in the system in a rigorous way. Such a formal qualitative study is essential because once we have a parameterized language model, it will always have a “trivial regime” corresponding to trivial distribution of characters/words/tokens, and an objectively defined hallmark of nontriviality – such as phase transitions – will be a theoretically powerful tool to distinguish meaningful sentences from those generated in the trivial regimes. Just as there is a clear and mathematically well-defined boundary between water and ice, the existence of a phase transition in language models could lead us to understand the intrinsic difference separating e.g., a language-like structure that actually conveys information versus generated text that is just statistical gibberish. A potential existence of a spin-glass-like phase transition in probabilistic context-free grammars (CFGs) has recently attracted attention DeGiuli 2019 ; Giuli 2019 (See Fig. 1 (a) for a schematic diagram of CFGs). However, detailed analyses of CFGs have revealed the absence of such claimed phase transitions in some naturally assumed thermodynamic limit Nakaishi and Hukushima 2022 . While consensus is yet to be reached on this matter Lalegani and De Giuli 2023 ; Nakaishi and Hukushima 2024 , it would be fair to say that numerical and analytical support for a true phase transition, i.e., a mathematically singular behavior in a clearly defined thermodynamic limit, is lacking. Furthermore, as probabilistic CFG is one of the simplest probabilistic models of natural language, this conundrum with the existence of phase transition naturally raises the question of how complex a natural language model needs to be in order to exhibit phase transitions. Phase transition in statistical physics is a precisely defined mathematical concept that requires careful analysis. We provide a schematic figure of the types of phase transitions investigated in this paper in Fig. 1 (b). Figure 1: (a) Schematic of a CFG. This diagram shows the syntactic structure of “The boy with blue pants found a cat.” Natural languages are, however, context sensitive and in this paper we define a simple context sensitive grammar (CSG) that demonstrates phase transition. (b) We consider the relative frequencies of letters or symbols in the alphabet – as captured by the net magnetization of the corresponding physical system – as an order parameter in our CSG. The Binder parameter, which is the normalized kurtosis of the underlying order parameter (see Eq. 8 ) can capture phase transition effectively, especially for one-dimensional models. A schematic of the Binder parameter is shown for the standard second-order phase transition (upper) and the BKT transition (lower). For the the BKT transition the the Binder parameter’s dependence on temperature – which determines the probability with which a rule is applied in the underlying CSG – does not become a step function in the thermodynamic limit. (c) Correlation functions for the disordered, critical, and ordered phases. When a system is critical, the correlation function shows a polynomial decay. In particular, to demonstrate phase transition one should be able to construct an order parameter , which is strictly zero in the disordered phase and is nonzero in the ordered phase, thus allowing a rigorous characterization of different phases in a system. This switch from zero to nonzero is inevitably accompanied by a mathematical singularity (a divergence or discontinuity in the derivative, etc) of the order parameter. Note that these features are only realized in the thermodynamic limit. In practice, one can only observe finite size systems and extrapolate the data to see consistency with the existence of

Read the complete text