Recursive Dynamics in Fast-Weights Homeostatic Reentry Networks: Toward Reflective Intelligence
B. G. Chae · 2025
Read Recursive Dynamics in Fast-Weights Homeostatic Reentry Networks: Toward Reflective Intelligence by B. G. Chae (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-06798.txt.
Introduction
B G Chae introduces a neural network layer built to let a model refer back to its own earlier internal states. The Fast-Weights Homeostatic Reentry Layer combines fast-weight associative memory, a stabilizing homeostatic rule, and learned feedback that re-enters past activity into current processing. Standard transformers, by contrast, run purely feedforward at inference time.
In controlled tests the author swept the strength of the feedback and tracked it with three new measures of internal dynamics. A stable band appeared at moderate feedback strength, where reentry was expressive yet orderly, with smooth growth in one measure and steady low-frequency cycles in another. The results are presented as evidence that balanced feedback and self-regulation can produce reflective, thought-like internal processing.
Catalog record
| CREATOR | B. G. Chae |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-06798
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Recursive Dynamics in Fast-Weights Homeostatic Reentry Networks: Toward Reflective Intelligence B. G. Chae Abstract This study introduces the Fast-Weights Homeostatic Reentry Layer (FH-RL), a neural mechanism that integrates fast-weight associative memory, homeostatic regularization, and learned reentrant feedback to approximate self-referential computation in neural networks. Unlike standard transformer architectures that operate in a purely feedforward manner during inference, FH-RL enables internal recurrence without external looping, allowing prior latent states to be dynamically re-entered into the ongoing computation stream. We conduct controlled experiments sweeping the reentry gain γ \gamma and evaluate emergent internal dynamics using three novel metrics: the Information Reentry Ratio (IRR), Eigen-Spectrum Recursion Index (ESRI), and Representational Drift Periodicity (RDP). Results show that reentry quantity increases proportionally with γ \gamma , while the learned feedback matrix W r W_{r} remains bounded and becomes more structured at moderate gains. Critically, a stable reflective band emerges around γ ≈ 0.10 \gamma\approx 0.10 – 0.20 0.20 , where internal feedback is maximally expressive yet spectrally stable: IRR rises smoothly, ESRI remains near zero, and RDP exhibits consistent low-frequency cycles. These findings provide quantitative evidence that reflective, thought-like internal processing can arise from a principled balance between feedback amplification and homeostatic regulation, linking modern fast-weight architectures to theories of cortical reentry and recursive cognition. 1 Introduction Human thought is inherently recursive—we can think about what we are thinking. Neuroscientific theories such as Edelman’s Reentrant Loop Hypothesis and Tononi’s Integrated Information Theory ( Edelman 1989 ; Tononi & Edelman 1998 ) propose that consciousness emerges from self-referential feedback among cortical regions. Transformer-based language models have achieved remarkable performance across a wide range of natural language processing tasks ( Vaswani et. al. 2017 ) . However, their computation remains strictly feed-forward and autoregressive, lacking the recursive, self-referential dynamics that characterize human cognition. Human thought continuously revisits and re-interprets its own internal representations—a process often described as reentrant processing in cognitive neuroscience ( Tononi & Edelman 1998 ; Lamme 2006 ) . This recursive self-reference underlies meta-cognition, imagination, and reflective reasoning, yet current Transformers represent context only through one-way attention. Recent advances in prompting methods such as chain-of-thought reasoning ( Wei et. al. 2022 ) , ReAct ( Yao et. al. 2022 ) , and self-reflection frameworks ( Shinn et. al. 2023 ) enable models to simulate recursive reasoning through token-level iterative prompts. However, these mechanisms operate externally at the sequence level, rather than internally within the neural state dynamics. In contrast, biological cognition exhibits intrinsic recursive dynamics, where cortical areas engage in bidirectional feedback loops to refine internal states ( Friston 2010 ; Lamme 2006 ) . This motivates the search for neural architectures with built-in reentrant computation rather than prompt-level emulation. Early fast-weight models ( Hinton & Plaut 1987 ) introduced the idea of rapidly adapting synaptic weights to capture short-term correlations between sequential inputs. In these early formulations, a single fast-weight matrix was updated by an outer product of the current hidden state, enabling transient memory but without explicit control of stability. Subsequent approaches, such as Fast-Weights RNNs ( Ba et. al. 2016 ) and Fast-Weight Programmers ( Schlag et. al. 2021 ) , further improved associative memory mechanisms and computational efficiency, often leveraging the implicit low-rank structure arising from outer-product updates. However, these architectures typically lacked mechanisms for recurrent self-feedback: information flowed forward in time but was not recursively reinjected into the representational stream. More recent transformer-derived designs, including Linear Transformers ( Katharopoulos et. al. 2020 ) , Retentive Networks (RetNet; Sun et. al. 2023 ), and Self-Referential Weight Matrix (SRWM; Irie et. al. 2022 ), introduced efficient state-retention kernels that implicitly approximate low-rank memory. While these works mark progress toward biologically-inspired temporal continuity, they still maintain a feed-forward computational graph and omit explicit reentrant feedback between internal states. In this paper, we propose the Fast-Weights Homeostatic Reentry Layer (FH-RL) , a minimal extension to the Transformer architecture designed to emulate this recursive property. The key intuition is to allow the model’s internal state at time step not only to condition future tokens indirectly through hidden states, but also to explicitly reenter the input stream as a controlled feedback signal. Table 1 describes the comparison for existing fast-weights approaches and reentrant models. By combining fast associative memory, homeostatic normalization, and reentrant feedback, FH-RL provides a differentiable approximation of the cortical reentry loop in the human brain, as illustrated in Fig. 1. Contributions: • We introduce a biologically motivated reentrant fast-weight mechanism that adds self-referential feedback to the Transformer sequence model. • We propose quantitative metrics—Information Reentry Ratio (IRR), Eigen-Spectrum Recursion Index (ESRI), and Representational Drift Periodicity (RDP)—to measure recursive reasoning. • Through controlled experiments, we show that FH-RL improves long-range stability and exhibits measurable self-referential internal dynamics. Table 1: Comparison of fast-weight and reentrant models. Model / Year Low-rank (U,V) decomposition Reentrant feedback Main purpose Hinton & Plaut (1987) ✗ ✗ Short-term associative memory Ba et al. (2016) ✗ ✗ RNN temporal extension Schlag et al. (2018, 2021) ✓(static) ✗ Efficient linearized attention RetNet (Sun et al., 2023) ✓(implicit) ( \approx ) Temporal retention only FH-RL (ours) ✓(dynamic) ✓explicit loop Self-referential recursive computation Figure 1: Conceptual reentrant loop in the FH-RL model using fast-weights and learned feedback projection. 2 Theoretical Framework 2.1 Low-Rank Fast Weights as Dynamic Associative Memory Fast-weight mechanisms allow a neural system to encode transient associations between input patterns without modifying long-term parameters. In the proposed FH-RL, these associations are maintained in a low-rank factorized form: W_{t}\approx U_{t}^{T}V_{t},\quad U_{t},V_{t}\in\mathbb{R}^{r\times d}. (1) Rather than storing a full d\times d weight matrix, the model tracks two rank- r subspaces U_{t} and V_{t} , representing pre- and post-synaptic traces. These factors evolve according to an exponential moving-average rule: \displaystyle U_{t} \displaystyle=(1-\alpha)U_{t-1}+\alpha\,\text{normalize}(Q_{t}+\epsilon_{t}^{U}), (2) \displaystyle V_{t} \displaystyle=(1-\alpha)V_{t-1}+\alpha\,\text{normalize}(K_{t}+\epsilon_{t}^{V}), (3) where Q_{t}=W_{q}x_{t} , K_{t}=W_{k}x_{t} are query and key projections of the input representation, \alpha controls adaptation speed, and \epsilon_{t}^{U,V}\sim\mathcal{N}(0,\sigma^{2}I) are small perturbations. (typically \sigma\in[10^{-4},10^{-3}] ). These stochastic perturbations induce gradual differentiation among rank slots, enabling the fast-weight subspace to evolve dynamically rather than collapse into a single direction. This low-rank formulation offers two advantages: (i) computational efficiency \mathcal{O}(rd) instead of \mathcal{O}(d^{2}) , and (ii) a biologically interpretable update rule analogous to short-term synaptic plasticity. 2.2 Homeostatic Regulation of Dynamic Memory At each step t , the instantaneous associative output y_{t} is computed by projecting the value embedding through the dynamic fast-weight memory: \displaystyle y_{t} \displaystyle=(U_{t}^{\top}V_{t})\,V_{t}^{(v)},\quad V_{t}^{(v)}=W_{v}x_{t}, (4) where V_{t}^{(v)} is the value representation of the current token, and W_{t}^{(\mathrm{eff})}=U_{t}^{\top}V_{t}\in\mathbb{R}^{d\times d} (5) acts as a transient, input-dependent operator constructed from short-term query–key correlations. Thus, y_{t}=W_{t}^{(\mathrm{eff})}V_{t}^{(v)} (6) represents a context-conditioned transformation derived entirely from fast associative binding. Fast-weight systems can exhibit runaway amplification if unconstrained. To ensure stable recursive dynamics, FH-RL introduces a homeostatic normalization that scales activations toward unit norm: y_{t}\leftarrow\frac{y_{t}}{1+\beta(\|y_{t}\|_{2}-1)},\qquad\beta>0, (7) which implements continuous negative feedback: activity above the target level is damped, while sub-threshold signals are slightly amplified. Biologically, this mechanism parallels activity-dependent homeostasis and synaptic scaling in cortical circuits, which normalize neural response magnitudes while preserving relative representational structure ( Turrigiano 2008 ; Mayzel & Schneidma 2024 ) . 2.3 Reentrant Feedback Integration The distinctive feature of FH-RL is that its fast-weight output is recursively reinjected into the next input, forming a closed feedback loop: