Liquid and solid layers in a thermal deep learning machine
Gang Huang, Lai Shun Chan, Hajime Yoshino, Ge Zhang, Yuliang Jin · 2025
Read Liquid and solid layers in a thermal deep learning machine by Gang Huang, Lai Shun Chan, Hajime Yoshino, Ge Zhang, Yuliang Jin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-06789.txt.
Introduction
Gang Huang and co-authors test a physical picture of how deep neural networks store what they learn, in a research paper from 2025. Theory had predicted that in wide networks, the weights near the input and output ends are pinned down by training data. The middle weights, by contrast, should stay mostly free, giving a solid-liquid-solid layer pattern.
The team built a thermal version of a deep learning machine and trained it with a physics simulation run at very low temperature. The trained machine still scored well on real benchmark data. Measures of how weights moved over time showed the end layers had ordered, layered design spaces, while the middle layer stayed formless. The authors sum up the results in a phase diagram that maps storage load against network depth.
Catalog record
| CREATOR | Gang Huang, Lai Shun Chan, Hajime Yoshino, Ge Zhang, Yuliang Jin |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-06789
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Liquid and solid layers in a thermal deep learning machine Gang Huang, Lai Shun Chan, Hajime Yoshino, Ge Zhang, Yuliang Jin Abstract Based on deep neural networks (DNNs), deep learning has been successfully applied to many problems, but its mechanism is still not well understood – especially the reason why over-parametrized DNNs can generalize. A recent statistical mechanics theory on supervised learning by a prototypical multi-layer perceptron (MLP) on some artificial learning scenarios predicts that adjustable parameters of over-parametrized MLPs become strongly constrained by the training data close to the input/output boundaries, while the parameters in the center remain largely free, giving rise to a solid-liquid-solid structure. Here we establish this picture, through numerical experiments on benchmark real-world data using a thermal deep learning machine that explores the phase space of the synaptic weights and neurons. The supervised training is implemented by a GPU-accelerated molecular dynamics algorithm, which operates at very low temperatures, and the trained machine exhibits good generalization ability in the test. Global and layer-specific dynamics, with complex non-equilibrium aging behavior, are characterized by time-dependent auto-correlation and replica-correlation functions. Our analyses reveal that the design space of the parameters in the liquid and solid layers are respectively structureless and hierarchical. Our main results are summarized by a data storage ratio - network depth phase diagram with liquid and solid phases. The proposed thermal machine, which is a physical model with a well-defined Hamiltonian, that reduces to MLP in the zero-temperature limit, can serve as a starting point for physically interpretable deep learning. Introduction. In deep learning, a fundamental problem is to understand how the mapping between the input data {\bf S}_{0} and the output data {\bf S}_{L} is established through intermediate hidden layers of neurons, denoted by internal representations {\bf S}_{l} Monasson and Zecchina 1995 , where l=1\ldots L-1 and L is the number of layers in the DNN. Noticeable progress has been made within the framework of statistical mechanics, based on the intrinsic connection between neural networks and spin glass problems Hopfield 1982 ; Engel and Van den Broeck 2001 ; Carleo et al. 2019 ; Nishimori 2001 ; Gardner 1987 ; Gardner 1988 ; Baldassi et al. 2016 ; Choromanska et al. 2015 ; Ballard et al. 2017 ; Baity-Jesi et al. 2018 ; Franz et al. 2019 ; Geiger et al. 2019 ; Yoshino 2020 ; Yoshino 2023 ; Huang et al. 2024 ; Winter and Janssen 2024 . The statistical mechanics theory of supervised learning of single perceptrons by Gardner Gardner 1987 ; Gardner 1988 has been extended to MLPs by Yoshino Yoshino 2020 ; Yoshino 2023 . Such a theory considers ensembles of machines specified by realizations of synaptic weights (connection weights between neurons) {\bf J}_{l} as well as the neurons (internal representations) Monasson and Zecchina 1995 in the hidden layers {\bf S}_{l} . The phase space volume of the machines, called Gardner’s volume , is the number of machines that meet all the constraints imposed by the training data. Such constraints are specified by the boundary conditions: {\bf S}_{0} and {\bf S}_{L} are quenched boundaries that may be (i) generated simply randomly (random scenario: a random constraint satisfaction problem) or (ii) provided through a randomly generated teacher-machine (teacher-student scenario: a statistical inference problem). The replica theory for the MLP finds that typical machines in Gardner’s volume exhibit a solid-liquid-solid structure with the hidden layers near boundaries in the solid states and the middle layers in the liquid states (see Fig. 1 a). The solid state corresponds to a glass in the (i) random scenario, and a crystal associated with the teacher machine in the (ii) teacher-student scenario. Such a structure could also balance representation and generalization, via respectively the vast microscopic states in liquid layers and the strong correlations to the input/output boundaries in solid layers. An interesting implication of the theory, which is based on the internal representation, is that one can replace the conventional machine learning algorithms based on back-propagation by physically more sensible ones, called thermal deep learning machines (TDLM) . The conventional algorithms try to minimize a loss function that depends on the actual labels {\bf S}_{L} , and the predicted outputs {\bf S}^{\prime}_{L} generated by a sequence of highly convoluted linear and non-linear operations on {\bf S}_{0} . In particular, such a conventional loss function does not explicitly depend on the internal representation. In contrast, the TDLM has a well-defined Hamiltonian consisting of local interactions among the internal representation \{{\bf S}_{l}\} and \{{\bf J}_{l}\} , with the training data imposed as fixed boundary conditions {\bf S}_{0} and {\bf S}_{L} . With the given Hamiltonian, the TDLM can be simulated at any temperature. Under the framework of TDLMs, the learning problem can be conveniently tackled using theoretical and numerical tools developed in statistical physics. In physics, there are numerous examples of boundary-induced phenomena, including liquid-gas wetting Nelson et al. 2004 , melting dynamics Krzakala and Zdeborová 2011a ; Krzakala and Zdeborová 2011b , the point-to-set correlation length in supercooled liquids Biroli et al. 2008 ; Kob et al. 2012 ; Hocky et al. 2012 ; Ikeda and Ikeda 2015 , and vapor-deposited ultrastable glasses Singh et al. 2013 ; Berthier et al. 2017 . Deep learning provides another interesting playground where breaking of the translational symmetry imposed by the boundaries can bring non-trivial consequences. Figure 1: Liquid-solid phase diagram of deep learning. (a) Schematic phase diagram for the state of network parameters in the l -th layer with a given capacity ratio \alpha=M/N , drawn according to the theoretical prediction Eq. ( 1 ) in N\to\infty limit Yoshino 2020 ; Yoshino 2023 . (b) Numerical phase diagram obtained by MD learning simulations, color coded according to the spin overlap between replicas, q_{l}^{*}\equiv q_{l}(t=t^{*}=5\times 10^{5},t_{\rm w}=5\times 10^{4}) . Here L=10,M=2000 are fixed, and N is varied. The boundary between liquid (green) and glass (blue) phases is estimated approximately at q_{l}^{*}\approx 1/e . Although the picture provided by the replica theory for MLP is interesting, the current theory considers completely random, structureless input data Yoshino 2020 ; Yoshino 2023 , which is very far from reality. Also, as a static theory, the replica theory does not address training dynamics. In order to examine the theoretical picture in deep learning of real-world data, one has to perform numerical experiments. This is the motivation of the current study. Our simulations establish the coexistence of liquid and solid layers in learning dynamics, summarized by a numerical phase diagram (see Fig. 1 b). For MNIST benchmark data LeCun et al. ; Qiao 2007 , the TDLM trained by a GPU-accelerated molecular dynamics (MD) algorithm, has good generalization with a test accuracy up to \sim 98\% . The training procedure contains three dynamical regimes, with strong aging effects appearing in the intermediate time regime. Our analyses based on time-correlation functions and replica overlap order parameters show distinct dynamic and thermodynamic features of the network layers in the center and near the boundaries. The states in the central layers are liquid-like, characterized by rapid relaxation dynamics and nearly zero overlaps between differently trained machines. In contrast, the states in the near-boundary layers are solid-like, characterized by slow dynamics and a hierarchical organization of the overlap order