Latent Object Permanence: Topological Phase Transitions, Free-Energy Principles, and Renormalization Group Flows in Deep Transformer Manifolds

Faruk Alpay, Bugra Kilictas · 2026

Read Latent Object Permanence: Topological Phase Transitions, Free-Energy Principles, and Renormalization Group Flows in Deep Transformer Manifolds by Faruk Alpay, Bugra Kilictas (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-19942.txt.

Introduction

Faruk Alpay and Bugra Kilictas offer a theory of how language models reason in steps. Multi-step reasoning suggests an inner process that turns smooth internal states into stable, reusable units. The authors model this as a phase transition in the geometry of the latent activation manifold across the network's depth. They call the phenomenon latent object permanence.

Their account draws on the free-energy principle and renormalization group flows, tools from physics for describing systems at many scales. A phase transition view predicts sharp changes in behaviour as inputs vary. It also suggests why reasoning can fail suddenly. The theory is speculative but testable in principle. It aims to give interpretability research a physical vocabulary.

Catalog record

CREATORFaruk Alpay, Bugra Kilictas
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-19942

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Latent Object Permanence: Topological Phase Transitions, Free-Energy Principles, and Renormalization Group Flows in Deep Transformer Manifolds Faruk Alpay, Bugra Kilictas Abstract The emergent capability of Large Language Models (LLMs) to perform multi-step reasoning suggests an internal mechanism that behaves as if it discretizes continuous representations into stable, reusable units. We model this phenomenon as a phase transition in the geometry of a latent activation manifold ℳ \mathcal{M} across depth. Let h ( l ) ∈ ℝ d h^{(l)}\in\mathbb{R}^{d} be the residual stream at layer l l and C ( l ) = Cov ⁡ ( h ( l ) ) C^{(l)}=\mathrm{Cov}(h^{(l)}) its covariance. We study (i) the spectral density ρ l ​ ( λ ) \rho_{l}(\lambda) of C ( l ) C^{(l)} , (ii) intrinsic-dimension proxies such as effective rank, and (iii) a sparsity-like order parameter based on Object Integrity Ω \Omega . We propose that sufficiently large models exhibit a critical depth fraction γ c ≈ 0.42 \gamma_{c}\approx 0.42 where symmetry breaking occurs: the latent dynamics shift from a high-entropy “liquid” regime (diffuse superposition, Marchenko–Pastur-like bulk) to a low-entropy “solid” regime (spectral gaps, stable basins). We formalize these basins as Transient Class Objects (TCOs), and connect their emergence to (a) a free-energy variational principle underlying attention softmax and (b) a discrete Renormalization Group (RG) flow that contracts irrelevant directions. We provide sufficient conditions guaranteeing spectral collapse (transverse contraction) and give rigorous mixture-model results relating logical separability to low-rank spiked covariance structure. 1 Introduction Interpretability work on Transformer models [ 1 ] often treats the latent space \mathbb{R}^{d} as a continuous semantic field where meaning is encoded in approximately linear directions [ 2 , 3 ] . Yet multi-step reasoning requires operations that are effectively discrete: negation, quantification, variable binding, and compositional control flow. Bottleneck hypotheses—e.g. the Consciousness Prior [ 4 ] and capsule-like factorization [ 5 ] —suggest that high-level cognition requires sparse, manipulable factors that behave like latent “objects.” We investigate whether deep Transformers spontaneously implement such discretization via a mechanism analogous to the Renormalization Group (RG) [ 7 ] : a coarse-graining flow that integrates out short-range correlations (local syntax) and stabilizes long-range operators (logical/semantic relations). Unlike shallow-layer accounts emphasizing feature superposition [ 8 , 6 ] , we focus on deep-layer regimes where reasoning emerges, and ask whether the latent geometry exhibits signatures of a phase transition. Core thesis. At sufficient scale, depth acts like an implicit cooling schedule : attention becomes sharper, free energy decreases, covariance spectra develop spikes and gaps, and effective dimensionality collapses. We interpret the post-critical regime as a “solid” phase in which latent trajectories concentrate near stable basins (TCOs) supporting object permanence across steps. 2 Preliminaries and Observables 2.1 Depth as Discrete Time and Pushforward Dynamics Consider a Transformer with L residual blocks. Let h^{(l)}\in\mathbb{R}^{d} denote the (tokenwise) residual stream at layer l . Define normalized depth \gamma\coloneqq\frac{l}{L}\in[0,1]. (1) Treat each layer as a measurable map F_{l}:\mathbb{R}^{d}\to\mathbb{R}^{d} (including attention, MLP, residual addition). Then the distribution \mathsf{P}_{l} of h^{(l)} evolves by pushforward \mathsf{P}_{l+1}=(F_{l})_{\#}\mathsf{P}_{l}. (2) 2.2 Covariance Spectrum, Effective Rank, and Participation Ratio Let \mu^{(l)}=\mathbb{E}[h^{(l)}] and C^{(l)}\coloneqq\mathbb{E}\big[(h^{(l)}-\mu^{(l)})(h^{(l)}-\mu^{(l)})^{\top}\big]\succeq 0. (3) Let eigenvalues be \lambda^{(l)}_{1}\geq\cdots\geq\lambda^{(l)}_{d}\geq 0 , and define the spectral density (empirical measure) \rho_{l}(\lambda)\coloneqq\frac{1}{d}\sum_{i=1}^{d}\delta(\lambda-\lambda^{(l)}_{i}). (4) Define normalized eigenvalues \hat{\lambda}^{(l)}_{i}=\lambda^{(l)}_{i}/\mathrm{Tr}(C^{(l)}) and spectral entropy S(C^{(l)})\coloneqq-\sum_{i=1}^{d}\hat{\lambda}^{(l)}_{i}\log\hat{\lambda}^{(l)}_{i},\qquad R_{\mathrm{eff}}(C^{(l)})\coloneqq\exp(S(C^{(l)})). (5) Also define the participation ratio (PR) dimension d_{\mathrm{PR}}(C^{(l)})\coloneqq\frac{\mathrm{Tr}(C^{(l)})^{2}}{\mathrm{Tr}\!\big((C^{(l)})^{2}\big)}. (6) Proposition 2.1 (Basic bounds) . For any C\succeq 0 with C\neq 0 , 1\leq d_{\mathrm{PR}}(C)\leq\mathrm{rank}(C)\leq d,\qquad 1\leq R_{\mathrm{eff}}(C)\leq d. 2.3 Object Integrity Order Parameter For h\in\mathbb{R}^{d}\setminus\{0\} define \Omega(h)\coloneqq 1-\frac{\|h\|_{1}}{\sqrt{d}\,\|h\|_{2}}. (7) Proposition 2.2 (Sharp bounds and extremizers) . For any h\neq 0 , 0\leq\Omega(h)\leq 1-\frac{1}{\sqrt{d}}. Moreover, \Omega(h)=0 iff |h_{1}|=\cdots=|h_{d}| , and \Omega(h)=1-\frac{1}{\sqrt{d}} iff h is 1-sparse. Proof. By Cauchy–Schwarz, \|h\|_{1}\leq\sqrt{d}\|h\|_{2} gives \Omega(h)\geq 0 with equality iff all |h_{i}| equal. Also \|h\|_{1}\geq\|h\|_{2} gives \Omega(h)\leq 1-1/\sqrt{d} with equality iff h is 1-sparse. ∎ Define the depth profile mean and susceptibility m(\gamma)\coloneqq\mathbb{E}[\Omega(h^{(l)})],\qquad\chi(\gamma)\coloneqq\mathrm{Var}(\Omega(h^{(l)})),\quad\gamma=l/L. (8) 3 Information Geometry of the Latent Manifold 3.1 Fisher Metric Induced by the Output Distribution Let P_{\theta}(y\mid h) denote the next-token distribution defined by the output head (e.g. linear map + softmax). The Fisher information metric on latent coordinates is g_{ij}(h)\coloneqq\mathbb{E}_{y\sim P_{\theta}(\cdot\mid h)}\left[\frac{\partial\log P_{\theta}(y\mid h)}{\partial h_{i}}\frac{\partial\log P_{\theta}(y\mid h)}{\partial h_{j}}\right]. (9) This metric quantifies local sensitivity of predictions to perturbations in h , and thus is a natural candidate for a task-relevant Riemannian structure on \mathcal{M} . Remark 1 . If the output head is linear logits \ell=W_{\text{out}}h+b , then \partial_{h}\log P(y\mid h)=W_{\text{out}}^{\top}(e_{y}-p) , so g(h) is controlled by W_{\text{out}} and the output covariance \mathrm{Cov}(e_{y}) under p . This makes g empirically estimable (up to sampling) from logits and W_{\text{out}} . 3.2 Curvature as a Proxy for Hierarchy Curvature quantities (e.g. Ricci curvature) measure how geodesics converge/diverge and can encode representational hierarchy. While we do not assume constant curvature, we propose the following as a diagnostic hypothesis : Definition 3.1 (Hyperbolic embedding hypothesis (diagnostic)) . Deep semantic hierarchies are facilitated when the effective latent geometry exhibits negative curvature in relevant subspaces. Early layers are expected to behave closer to locally Euclidean geometry (syntax), while deeper layers may induce more negatively curved effective geometry (hierarchical semantics). A convenient conceptual model is a forced geodesic equation on (\mathcal{M},g) driven by an attention-induced potential U_{\mathrm{attn}} : \frac{D^{2}h^{i}}{d\tau^{2}}+\Gamma^{i}_{jk}(h)\frac{dh^{j}}{d\tau}\frac{dh^{k}}{d\tau}=-g^{ij}(h)\frac{\partial U_{\mathrm{attn}}}{\partial h^{j}}, (10) where D/d\tau is the covariant derivative and \Gamma^{i}_{jk} are Christoffel symbols. The forced term is a schematic way to express that attention reshapes trajectories towards preferred basins. 4 Thermodynamics of Attention: A Free-Energy Principle 4.1 Softmax as a Gibbs Distribution Consider one attention head with query q\in\mathbb{R}^{d_{k}} , keys \{k_{j}\}_{j=1}^{T} , values \{v_{j}\}_{j=1}^{T} . The attention weights are a_{j}=\frac{\exp(\beta\langle q,k_{j}\rangle)}{\sum_{r=1}^{T}\exp(\beta\langle q,k_{r}\rangle)},\qquad\beta\coloneqq\frac{1}{\sqrt{d_{k}}}. (11) Define energies E_{j}\coloneqq-\langle q,k_{j}\rangle . Then a_{j}\propto\exp(-\beta E_{j}) is a Gibbs distribution. Proposition 4.1 (Variational characterization of attention via free energy) . Let \Delta_{T}=\{p\in\mathbb{R}^{T}_{\geq 0}:\sum_{j}p_{j}=1\} . Define the free-energy functional \mathcal{F}(p)\coloneqq\sum_{j=1}^{T}p_{j}E_{j}+\frac{1}{\beta}\sum_{j=1}^{T}p_{j}\log p_{j}. (12) Then \mathcal{F}(p) is minimized over \Delta_{T} uniquely by the Gibbs distribution p_{j}^{\star}=\frac{\exp(-\beta E_{j})}{\sum_{r=1}^{T}\exp(-\beta E_{r})}=\frac{\exp(\beta\langle q,k_{j}\rangle)}{\sum_{r=1}^{T}\exp(\beta\langle q,k_{r}\rangle)}. Proof. Let Z=\sum_{r=1}^{T}\exp(-\beta E_{r}) and define p^{\star}_{j}=\exp(-\beta E_{j})/Z . Compute D_{\mathrm{KL}}(p\|p^{\star})=\sum_{j}p_{j}\log\frac{p_{j}}{p^{\star}_{j}}=\sum_{j}p_{j}\log p_{j}+\beta\sum_{j}p_{j}E_{j}+\log Z. Thus \mathcal{F}(p)=\sum_{j}p_{j}E_{j}+\frac{1}{\beta}\sum_{j}p_{j}\log p_{j}=\frac{1}{\beta}D_{\mathrm{KL}}(p\|p^{\star})-\frac{1}{\beta}\log Z, which is minimized iff D_{\mathrm{KL}}(p\|p^{\star})=0 , i.e. p=p^{\star} . ∎ 4.2 Cooling with Depth and Crystallization Intuition Although \beta=1/\sqrt{d_{k}} is fixed per architecture, the effective sharpness of softmax depends on energy scale. If typical magnitudes \|q\| and \|k\| grow with depth, then \langle q,k\rangle scales up, making \exp(\beta\langle q,k\rangle) more peaked. This induces a depth-wise “cooling” effect: the entropy term becomes relatively less important, pushing the system toward low-entropy selections, consistent with basin formation. 5 Random Matrix Theory Baseline and Spiked Covariance 5.1 Marchenko–Pastur as a Null Model Let X\in\mathbb{R}^{T\times d} have i.i.d. entries with mean 0 and variance \sigma^{2} , and define the sample covariance C=\frac{1}{T}X^{\top}X\in\mathbb{R}^{d\times d}. Let the aspect ratio be c\coloneqq\frac{d}{T}\in(0,\infty). (13) Definition 5.1 (Marchenko–Pastur distribution) . As d,T\to\infty with d/T\to c , the empirical spectral distribution of C converges (under standard conditions) to the Marchenko–Pastur law [ 16 ] with density \rho_{\mathrm{MP}}(\lambda)=\frac{1}{2\pi\sigma^{2}c}\,\frac{\sqrt{(\lambda_{+}-\lambda)(\lambda-\lambda_{-})}}{\lambda}\,\mathbf{1}_{[\lambda_{-},\lambda_{+}]}(\lambda),\qquad\lambda_{\pm}=\sigma^{2}(1\pm\sqrt{c})^{2}. (14) We use \rho_{\mathrm{MP}} as a baseline for “unstructured” (noise-like) covariance. Deviations via outlier eigenvalues (spikes) suggest low-rank signal components. 5.2 Low-Rank Signal and Spikes A canonical structured model is the spiked covariance form C_{\mathrm{pop}}=\sigma^{2}\mathrm{Id}+\sum_{r=1}^{k}\theta_{r}u_{r}u_{r}^{\top},\qquad u_{r}\in\mathbb{R}^{d},\ \|u_{r}\|_{2}=1,\ \theta_{r}>0, (15) with k\ll d . Empirically, a “solid” phase corresponds to: (i) a bulk roughly MP-like, plus (ii) k outlier eigenvalues beyond \lambda_{+}

Read the complete text