PLDR-LLMs REASON AT SELF-ORGANIZED CRITICALITY

Burc Gokden · 2026

Read PLDR-LLMs REASON AT SELF-ORGANIZED CRITICALITY by Burc Gokden (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-23539.txt.

Introduction

Gokden argues that some language models reason because they sit at self-organized criticality. PLDR models trained at that edge show outputs like a phase transition. At criticality, the correlation length diverges and outputs settle into a steady state. In that state, outputs act like scaling functions and universality classes. That likeness, he says, is what yields reasoning skill. He defines an order parameter from the outputs' stats. Models reason best when that parameter sits near zero at criticality. Benchmark scores for near-critical training back the claim.

Catalog record

CREATORBurc Gokden
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE74000
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.23539

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PLDR-LLMs REASON AT SELF-ORGANIZED CRITICALITY Burc Gokden Abstract We show that PLDR-LLMs pretrained at self-organized criticality exhibit reasoning at inference time. The characteristics of PLDR-LLM deductive outputs at criticality is similar to second-order phase transitions. At criticality, the correlation length diverges, and the deductive outputs attain a metastable steady state. The steady state behaviour suggests that deductive outputs learn representations equivalent to scaling functions, universality classes and renormalization groups from the training dataset, leading to generalization and reasoning capabilities in the process. We can then define an order parameter from the global statistics of the model’s deductive output parameters at inference. The reasoning capabilities of a PLDR-LLM is better when its order parameter is close to zero at criticality. This observation is supported by the benchmark scores of the models trained at near-criticality and sub-criticality. Our results provide a self-contained explanation on how reasoning manifests in large language models, and the ability to reason can be quantified solely from global model parameter values of the deductive outputs at steady state, without any need for evaluation of curated benchmark datasets through inductive output for reasoning and comprehension. 1 Introduction Large Language Model from Power Law Decoder Representations (PLDR-LLM) are language models that are composed of highly non-linear, multi-head power law graph attention (PLGA) mechanism as building blocks of their decoder layers ( Gokden 2025 ; Gokden 2024 ; Gokden 2021 ; Gokden 2019 ) . The PLGA mechanism follows a series of well-defined non-linear transformations to learn a generalization of the query states through learnable power law scaling coefficients and exponents from the data. The well-defined structure of the PLGA tensor network allows for definition of a set of deductive outputs that inform on both local and global characteristics of the attention mechanism. Linear transformations by query and key vectors can then extract the representations relevant to the input from the energy-curvature tensor of the PLGA as the attention to be applied on value vector to predict the next token. Compared to the widely adopted scaled-dot product attention (SDPA) based LLMs, the PLGA learns higher degree of symmetries from the data through its treatment of the learned energy-curvature tensor, which is one of the deductive outputs. For SDPA-LLMs, this tensor is predefined as identity and only linear transformations by query and key vectors are part of the language model. While this configuration makes SDPA-LLMs easier to train and quick to infer through linear transformations, the fundamental principles that may explain many of LLM characteristics have been source of much debate. The PLGA demonstrates unique characteristics during training and inference that were investigated in detail from the perspective of neural network optimization methodologies. However, this approach has its limits and lacks a complete understanding when all aspects of PLDR-LLMs under training and inference are considered; hampering a much deeper, and possibly a full analytical treatment of LLMs. During training, the PLDR-LLM exhibits reasoning at specific pairs of total warm-up step count and maximum learning rate and follows a loss curve that appears underfit from a typical machine learning model optimization point of view. At other pairs of values, when reasoning is not achieved, the loss curve becomes overfit and the generated text output is a random sequence of tokens at inference. Moreover, when PLDR-LLM is pretrained under conditions so that it exhibits reasoning capabilities, it was shown that the deductive outputs of PLGA behave as tensors at a steady state such that they are only negligibly perturbed for any input during inference ( Gokden 2025 ) . Thus, the query and key vectors are only needed to extract representations relevant to the input from deductive outputs as an attention tensor to be applied on the value vector. This makes the model very efficient for data transfer and computation during inference, enabling the caching of the deductive outputs and skipping the execution of non-linear section. The SDPA-LLM satisfies the condition of being at steady state implicitly under constraints, since what would be a final output of PLGA is predefined as identity tensor at all times. Recognizing the long standing inadequacies of the traditional loss optimization approach for PLDR-LLMs in general, we propose an alternative explanation. The above characteristics of PLDR-LLMs at training and inference indicate that there is a phase transition for the loss curve at a specific maximum learning rate when the input as batches of tokens are slowly driven up to that level through a linear warm-up schedule. The steady state, high dimensional symmetry behaviour of the deductive outputs at the right combinations of warm-up step counts and maximum learning rates indicates that long range, global scale interactions are established across the entire model. Respectively, the linear warm-up rate and maximum learning rate act as control parameters for extrinsic driving (forward propagation) and intrinsic dissipative (backward propagation) forces at different time scales in a PLGA mechanism that learns through power law scaling coefficients and exponents. In the light of these observations made in the previous studies, we demonstrate through experiments focused on global behaviour of small size models that PLDR-LLM architecture is a mechanism of generating reasoning and comprehension at self-organized criticality. When considered in context of the approach in ( Bak et al. 1988 ) that first introduced the concept of self-organized criticality, the batches of sequence of tokens represent the grains of sand, and PLDR-LLM is the model that governs and generates the dynamics of a sandpile. In this paper, we investigate and extend on the observations of unique PLDR-LLM characteristics from the perspectives of self-organized criticality and second-order phase transitions. Our work aims to set a path for a complete characterization and understanding of how intelligence emerges in large language models by using small size PLDR-LLMs as an experimental vehicle. We make the following fundamental contributions: • We empirically show that PLDR-LLMs achieve reasoning and ability to generalize when long range interactions overlap at criticality, leading to a global metastable steady state for all deductive outputs of the model. • We define a simple global order parameter which can be used as a metric to define how well a PLDR-LLM can reason. This metric does not depend on any curated benchmark datasets, is robust against stochastic sampling and is an intrinsic characteristic of the model. It can be used with high precision to rank even small size models in a reliable manner. In this picture, an order parameter close to zero is an indication of high reasoning and generalization capabilities for a PLDR-LLM. • We provide simple and straightforward explanations on why scaling of LLM size and token amount are dependent on each other and larger LLMs have improved generalization capabilities. We provide answers to why certain approaches such as rotary positional embedding and gated linear units (GLUs) improve performance of LLMs. • The self-organized criticality paradigm is also thought to be the basis of numerous physical phenomena including the human brain, solar flares, and earthquakes. Specifically, our work aligns the fundamental dynamics of large language models with observations made for the human brain and provides an artificial test bed for detailed experiments on complex systems. • Pytorch implementation of PLDR-LLM for multi-gpu training and inference used in this study is available at https://github.com/burcgokden/PLDR-LLM-Self-Organized-Criticality and pretrained models with

Read the complete text