From Density Matrices to Phase Transitions in Deep Learning: Spectral Early Warnings and Interpretability

Max Hennick, Guillaume Corlouer · 2026

Read From Density Matrices to Phase Transitions in Deep Learning: Spectral Early Warnings and Interpretability by Max Hennick, Guillaume Corlouer (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-29805.txt.

Introduction

Hennick and Corlouer bring a quantum chemistry tool into deep learning: the two-datapoint reduced density matrix. Sudden jumps in model skill during training are hard to predict. Tracking this matrix over a sliding window gives two warning signals. One is a spectral heat capacity tied to critical slowing down. The other, a participation ratio, shows how broad the change is. Top eigenvectors stay easy to read. Tests cover grokking, induction heads, and emergent misalignment.

Catalog record

CREATORMax Hennick, Guillaume Corlouer
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE110258
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.29805

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

From Density Matrices to Phase Transitions in Deep Learning: Spectral Early Warnings and Interpretability Max Hennick, Guillaume Corlouer Abstract A key problem in the modern study of AI is predicting and understanding emergent capabilities in models during training. Inspired by methods for studying reactions in quantum chemistry, we present the “2-datapoint reduced density matrix”. We show that this object provides a computationally efficient, unified observable of phase transitions during training. By tracking the eigenvalue statistics of the 2RDM over a sliding window, we derive two complementary signals: the spectral heat capacity , which we prove provides early warning of second-order phase transitions via critical slowing down, and the participation ratio, which reveals the dimensionality of the underlying reorganization. Remarkably, the top eigenvectors of the 2RDM are directly interpretable making it straightforward to study the nature of the transitions. We validate across four distinct settings: deep linear networks, induction head formation, grokking, and emergent misalignment. We then discuss directions for future work using the 2RDM. 1 Introduction Phase transitions are a ubiquitous concept in modern physics, and can be loosely defined as a qualitative change in the properties of a system as one acts upon it (increasing the temperature, applying an external magnetic field, etc.) As such, it is common to define a phase transition during training as a qualitative change in the internal structure of a model’s parameters that produces a corresponding change in behaviour or capabilities. Such transitions can be continuous (the structure changes gradually, with detectable precursors in fluctuation statistics, generally referred to as second order transitions ) or discontinuous (the structure reorganizes abruptly, with precursors that are brief or absent, referred to as first order transitions ). Understanding, detecting, and predicting phase transitions allows us to better understand and control model training. While the study of phase transitions in deep learning has seen significant progress in recent years, most work is either looking at the properties of a specific transition, and most general methods require large amounts of computational overhead, either from sampling model parameters or from the storage of gradient vectors, limiting their applicability to large models (see section 1.1 for discussion). Furthermore, no current method is generally capable of acting as a leading indicator of phase transitions in an online setting. To handle these issues, we introduce the 2-datapoint reduced density matrix (2RDM) as a computationally efficient and theoretically grounded backbone for studying phase transitions. The 2RDM is effectively the per-sample loss covariance (under perturbations of the model parameters) over a set of selected probe samples. The central claim of this work is that the covariance of per-sample losses serves as a general, low-cost observable of phase transitions during training as it effectively captures the underlying geometry of the loss landscape that is relevant for optimization. When evaluated on an appropriate probe set, this object captures when and how models reorganize across a wide range of settings. We show that the 2RDM can be used to derive early warning signals in the continuous (second order) case and concurrent detection signals in the discontinuous (first order) case, as well as providing other general metrics for the behaviour of the phase transition. Contributions. The main contributions of this work are as follows: 1. We introduce the 2-datapoint reduced density matrix (2RDM), a computationally efficient observable for studying phase transitions during neural network training that requires only forward passes on a small probe set. 2. We derive two complementary spectral diagnostics from the 2RDM: the spectral heat capacity (SHC), which we prove provides early warning of second-order phase transitions via critical slowing down (Propositions C.3 and C.4 ), and the participation ratio, which characterizes the dimensionality of the underlying reorganization. 3. We show that the top eigenvectors of the 2RDM are directly interpretable, providing mechanistic information about what is changing during a transition, not just when. 4. We validate the framework across four distinct settings:deep linear networks, induction head formation, grokking, and emergent misalignment. 5. We establish formal connections between the 2RDM and several existing frameworks, including the cumulant expansion of the free energy, the POLCA method [ 16 ] , the local learning coefficient [ 23 ] [ 46 ] , and the loss kernel [ 1 ] , situating the 2RDM within the broader theoretical landscape. 1.1 Related Work Phase transitions during learning have become a rich research area in recent years. Most research focuses on studying specific phase transitions, such as grokking [ 53 ] [ 30 ] [ 21 ] , lazy-to-rich transitions [ 14 ] [ 9 ] [ 22 ] , neural collapse [ 32 ] [ 52 ] [ 19 ] , in-context learning [ 31 ] [ 10 ] [ 39 ] [ 29 ] , and emergent misalignment [ 44 ] [ 40 ] [ 5 ] , among others. However, these works are not aimed at general methods for discovering and understanding new phase transitions and emergent structures during learning. There are lines of research dedicated to such methods [ 2 ] [ 16 ] , with a large number of these building on ideas from singular learning theory [ 47 ] . These broadly can be split into entropy based methods [ 23 ] [ 46 ] and influence function like methods [ 20 ] [ 24 ] [ 1 ] . We note as well that there is a natural relationship between the methods introduced here and the method introduced in [ 16 ] , which is explained in-depth in appendix C.6 . Furthermore, we note that the 2RDM is closely related to the loss kernel of [ 1 ] , with the relationship explained in detail in appendix C.8 . 2 The 2-Datapoint Reduced Density Matrix Borrowing a term from quantum chemistry (the 2-electron reduced density matrix) we introduce the 2-datapoint reduced density matrix below 1 1 1 Note that this is, on a technical level, not exactly the 2RDM, however it can be made a density matrix by normalization. : Definition 2.1 (2-datapoint reduced density matrix (2RDM)) . Let (x_{1:n},y_{1:n}) be a collection of n datapoints called the probe set , let \theta^{0} be a choice of parameter for a model with the loss \ell_{i}=\ell((x_{i},y_{i}),\theta^{0}) . Let \rho be a probability distribution of parameters near \theta^{0} . The 2-datapoint reduced density matrix for \rho is then given by the matrix with entries C_{ij}=\text{Cov}_{\theta\sim\rho}[\ell_{i},\ell_{j}] (1) We note that this object is theoretically motivated by the cumulant expansion, with details in appendix B . There are two main types of the 2RDM we consider. First is the Gaussian 2RDM where \rho(\theta)=\mathcal{N}(0,\sigma^{2}I)+\theta^{0} . Now consider a time step window of some fixed length m during training, and take the weights \theta_{m}=[\theta_{t_{0}},...,\theta_{t_{m}}] . For each \theta_{t} we compute a vector \vec{l}(\theta_{t}) of per sample losses l(\theta_{t})_{i}=l(x_{i},\theta_{t}) for some set of datapoints x_{1:n} . One way of interpreting this is if we consider a distribution over the path itself \rho_{\tau} , we are in some sense sampling from this distribution. As such, we define the Dynamical 2RDM as \text{Cov}_{\theta\sim{\rho_{\tau}}}[l_{i},l_{j}]=C_{ij}^{t_{0},t_{m}} (2) where l_{i} is the loss on the i th sample, and t_{0},t_{m} are the start and end times of the window. A discussion on the advantages and disadvantages of

Read the complete text