Self-orthogonalizing attractor neural networks emerging from the free energy principle
Tamas Spisak, Karl Friston · 2025
Read Self-orthogonalizing attractor neural networks emerging from the free energy principle by Tamas Spisak, Karl Friston (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-22749.txt.
Introduction
Tamas Spisak and Karl Friston showed in 2025 how networks with attractor states can grow by themselves out of the free energy principle. Attractors, stable patterns a system settles into, mark brains and many complex systems. Their derivation starts from a universal split of random dynamical systems, with no learning rule imposed by hand. Inference and learning rules emerge instead from the math.
In the resulting networks, attractors store prior beliefs on a free energy landscape. Reading data updates those beliefs, and slow tuning of links trims long-term surprise. The attractors turn out nearly orthogonal, which helps the network generalize. Ordered data make the links lopsided and push the system out of equilibrium, a natural extension of classic Boltzmann machines.
Catalog record
| CREATOR | Tamas Spisak, Karl Friston |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 110 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-22749
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Self-orthogonalizing attractor neural networks emerging from the free energy principle Tamas Spisak, Karl Friston Abstract Attractor dynamics are a hallmark of many complex systems, including the brain. Understanding how such self-organizing dynamics emerge from first principles is crucial for advancing our understanding of neuronal computations and the design of artificial intelligence systems. Here we formalize how attractor networks emerge from the free energy principle applied to a universal partitioning of random dynamical systems. Our approach obviates the need for explicitly imposed learning and inference rules and identifies emergent, but efficient and biologically plausible inference and learning dynamics for such self-organizing systems. These result in a collective, multi-level Bayesian active inference process. Attractors on the free energy landscape encode prior beliefs; inference integrates sensory data into posterior beliefs; and learning fine-tunes couplings to minimize long-term surprise. Analytically and via simulations, we establish that the proposed networks favor approximately orthogonalized attractor representations, a consequence of simultaneously optimizing predictive accuracy and model complexity. These attractors efficiently span the input subspace, enhancing generalization and the mutual information between hidden causes and observable effects. Furthermore, while random data presentation leads to symmetric and sparse couplings, sequential data fosters asymmetric couplings and non-equilibrium steady-state dynamics, offering a natural generalization of conventional Boltzmann Machines. Our findings offer a unifying theory of self-organizing attractor networks, providing novel insights for AI and neuroscience. Keywords attractor networks, free energy principle, active inference, orthogonal representations, self-organization • Attractor networks are derived from the Free Energy Principle ( FEP ) applied to a universal partitioning of random dynamical systems. • This approach yields emergent, biologically plausible inference and learning dynamics, forming a multi-level Bayesian active inference process. • The networks favor approximately orthogonalized attractor representations, optimizing predictive accuracy and model complexity. • Sequential data presentation leads to asymmetric couplings and non-equilibrium steady-state dynamics, generalizing conventional Boltzmann Machines. • Simulations demonstrate orthogonal basis formation, generalization, sequence learning, scalability and resistance to catastrophic forgetting. Graphical Abstract 1 Introduction From whirlpools and bird flocks to neuronal and social networks, countless natural systems can be characterized by dynamics organized around attractor states ( Haken, 1978 ) . Such systems can be decomposed into a collection of - less or more complex - building blocks or “particles” (e.g. water molecules, birds, neurons, or people), which are coupled through local interactions. Attractors are an emergent consequence of the collective dynamics of the system, arising from these local interactions, without any individual particles exerting global control. Attractors are a key concept in dynamical systems theory, defined as a set of states in the state space of the system to which nearby trajectories converge ( Guckenheimer et al., 1984 ) . Geometrically, the simplest attractors are fixed points and limit cycles (representing periodic oscillations). However, the concept extends to more complex structures like strange attractors associated with chaotic behavior, as well as phenomena arising in stochastic or non-equilibrium settings, such as probability distributions over states (stochastic attractors), transient states reflecting past dynamics (ghost attractors or attractor ruins), and trajectories that cycle through sequences of unstable states (sequential attractors or heteroclinic cycles). Artificial attractor neural networks ( Amit, 1989 ) represent a class of recurrent neural networks specifically designed to leverage attractor dynamics. While the specific forms and behaviors of these networks are heavily influenced by the chosen inference and learning rules, self-organization is a key feature of all variants, as the stable states emerge from the interactions between network elements without explicit external coordination. This property makes them particularly relevant as models for self-organizing biological systems, including the brain. It is clear that the brain is also a complex attractor network. Attractor dynamics have long been proposed to play a significant role in information integration at the circuit level ( Freeman, 1987 ; Amit, 1989 ; Deco and Rolls, 2003 ; Nartallo-Kaluarachchi et al., 2025 ) ( Tsodyks (1999) ) and have become established models for canonical brain circuits involved in motor control, sensory amplification, motion integration, evidence integration, memory, decision-making, and spatial navigation (see Khona and Fiete (2022) for a review). For instance, the activity of head direction cells - neurons that fire in a direction-dependent manner - are known to arise from a circular attractor state, produced by a so-called ring attractor network ( Zhang, 1996 ) . Multi- and metastable attractor dynamics have also been proposed to extend to the meso- and macro-scales ( Rolls, 2009 ) , “accommodating the coordination of heterogeneous elements” ( Kelso, 2012 ) , rendering attractor dynamics an overarching computational mechanism across different scales of brain function. The brain, as an instance of complex attractor networks, showcases not only the computational capabilities of this network architecture but also its ability to emerge and evolve through self-organization. When discussing self-organization in attractor networks, we will differentiate two distinct levels. First, we can talk about operational self-organization : the capacity of a pre-formed network to settle into attractor states during its operation. This however does not encompass the network’s ability to “build itself” – to emerge from simple, local interactions without explicit programming or global control, and to adaptively evolve its structure and function through learning. This latter level of self-organization is what we will refer to as adaptive self-organization . Architectures capable of adaptive self-organization would mirror the nervous system’s capacity to not just function as an attractor network, but to become - and remain - one through a self-directed process of development and learning. Further, adaptive self-organization would also be a highly desirable property for robotics and artificial intelligence systems, not only boosting their robustness and adaptability by means of continuous learning, but potentially leading to systems that can increase their complexity and capabilities organically over time (e.g. developmental robotics). Therefore, characterizing adaptive self-organization in attractor networks is vital for advancing our understanding of the brain and for creating more autonomous, adaptive, brain-inspired AI systems. The Free Energy Principle ( FEP ) offers a general framework to study self-organization to non-equilibrium steady states as Bayesian inference (a.k.a., active inference). The FEP has been pivotal in connecting the dynamics of complex self-organizing systems with computational and inferential processes, especially within the realms of brain function ( Friston et al., 2023 ; Friston and Ao, 2012 ; Palacios et al., 2020 ) . The FEP posits that any ‘thing’ - in order to exist for an extended period of time - must maintain conditional independence from its environment. This entails a specific sparsely coupled structure of the system, referred to as a particular partition , that divides the system into internal, external, and boundary (sensory and active) states (see Figure 2 A). It can be shown that maintaining this sparse coupling is equivalent to executing an inference process, where internal states deduce the causes of sensory inputs by minimizing variational free energy (see Friston et al. (2023) or ( Friston et al., 2023 ) for a formal treatment). Here, we describe the specific class of adaptive self-organizing attractor networks that emerge directly from the FEP , without the need for explicitly imposed learning or inference rules (Figure 1 ). First, we show that a hierarchical formulation of particular partitions - a concept that is applicable to any complex dynamical system - can give rise