When low-loss paths make a binary neuron trainable: detecting algorithmic transitions with the connected ensemble
Damien Barbier · 2026
Read When low-loss paths make a binary neuron trainable: detecting algorithmic transitions with the connected ensemble by Damien Barbier (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-23241.txt.
Introduction
Damien Barbier studies when a single binary neuron becomes trainable, using a statistical-mechanics lens. Binary neurons take only two states, making them cheap but hard to train. The connected ensemble framework tracks how low-loss paths form in rugged loss landscapes.
Those paths reveal when a network can learn a simple task at all. They also show which minima training should aim for. Knowing the transition point between impossible and trainable guides the design of efficient, discrete networks. Low-loss paths are the diagnostic object, revealing which minima training should target once trainability switches on.
Catalog record
| CREATOR | Damien Barbier |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 71 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-23241
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
When low-loss paths make a binary neuron trainable: detecting algorithmic transitions with the connected ensemble Preprint: APS/123-QED Damien Barbier Abstract We study the connected ensemble, a statistical-mechanics framework that characterizes the formation of low-loss paths in rugged landscapes. First introduced in 12 , this ensemble allows one to identify when a network can be trained on a simple task and which minima should be targeted during training. We apply this new framework to the symmetric binary perceptron model (SBP), and study how its typical connected minima behave. We show that connected minima exist only above a critical threshold κ connected \kappa_{\rm connected} , or equivalently below a critical constraint density α connected \alpha_{\rm connected} . This defines a parameter range in which training the network is easy, as local algorithms can efficiently access this connected manifold. We also highlight that these minima become increasingly robust and closer to one another as the task on which the network is trained becomes more difficult. I Introduction Once confined to a handful of theoretical models, the characterization of systems with a rugged loss landscape has become a ubiquitous challenge in recent years. It now spans a wide range of disciplines, encompassing diverse phenomena: from our understanding of neural networks and their learning capabilities to the phylogenetics of biological systems. With different perspectives, each field has unveiled important breakthroughs: the glassy transition and aging dynamics in disordered physics 32 ; 33 ; 17 , worst-case complexity and locally stable algorithms in computer science 20 ; 1 ; 19 or even the inference of phylogenetic trees in biology 35 ; 26 . The main drawback of these advances is usually their reliance on strong statistical-mechanics postulates. They typically describe systems at equilibrium, assuming ergodicity and the principle of indifference (or entropy maximization). However, over the years, dynamical systems that actually fall within this statistical-mechanics framework have become the exception rather than the norm. Neural networks (NN), virus dynamics -to name a few- are the new setups of interest, and they do not reach the standard Gibbs-Boltzmann equilibrium implied with these postulates. Understanding in which regions such setups can navigate remains in general an open problem. Interestingly, although often unnoticed, a common picture seems to emerge from three very different on-going research directions: machine learning, biophysics, and computer science. Whether it is the training of a neural network, the evolution process of a virus, or stable algorithms for a constraint satisfaction problem, dynamics/algorithms always appear to converge towards unexpectedly flat regions in these problems’ loss landscapes 13 ; 31 ; 8 ; 23 . In other words, these landscapes contain strong dynamical attractors formed by aggregations of minima. To understand the formation of such attractors, toy models with rugged landscapes have been designed and thoroughly studied; lattice protein model in biophysics 29 ; 27 ; 31 , committee machine in machine learning 36 , or quantum spin glass in physics 14 ; 9 to name a few. Regardless of the model, the overall setup remains the same: a system with a large number of degrees of freedom has to navigate in a complex landscape, its dynamics only allowing for incremental reconfigurations. In the plethora of available toy models, a case of particular interest is the perceptron model. Often presented as a one-layer NN, it consists of an optimization problem in which a N -dimensional vector {\bf x} has to correctly classify an extensive number of random data points \{\xi^{\mu}\}_{\mu\in[\![1,M]\!]} (for which extensive means M\propto N ) given an activation function. Since its first appearance in statistical physics in the 1980s 21 , many variants of this model have been scrutinized with different phase spaces for {\bf x} , data point distributions or activation functions 2 ; 28 ; 5 ; 3 . Its main interest lies in its relative simplicity compared to the models mentioned above; while still exhibiting the same phenomenology of flat dynamical attractors. For this model, the discrepancy between standard landscape characterization and dynamics can be summarized as follows. On the one hand, using statistical mechanics tools to analyze the landscape, it has been shown that typical solutions to the perceptron task (a.k.a. equilibrium solutions) cannot be reached in polynomial time by local algorithms. This is because these typical solutions are in fact surrounded by a sea of poorly performing configurations. In the literature, these are often referred to as isolated solutions 28 or as exhibiting the overlap-gap property 20 . To reach such solutions, an algorithm would have to jump over wide regions of poorly performing configurations. Realistic learning routines, which can only perform local updates, are incapable of making such long jumps efficiently. On the other hand, modern algorithms are now capable of returning viable solutions to the perceptron task. These new local algorithms appear to be insensitive to the abundance of isolated solutions in the loss landscape. In fact, numerous works 7 ; 6 ; 5 have pointed out that such algorithms actually target atypically flat regions. In other words, they target regions where many solutions can be found close to each other. This leaves one major problem: the statistical mechanics tools at our disposal are incapable of describing such atypical manifolds. In this direction, a recent breakthrough has been the introduction of the connected ensemble 13 ; 12 . It is an approach based on a simple idea: studying how paths of minima are distributed in the perceptron loss landscape. If typical solutions are isolated (and therefore inaccessible), a solution path \{{\bf x}^{k}\}_{k\in[\![0,k_{f}]\!]} should, with high probability, lie in flat (and accessible) regions. This is because each configuration {\bf x}^{k} lies close to its direct neighbors {\bf x}^{k\pm 1} . Therefore, this path structure contains at least one flat direction, namely along the path itself. Local algorithms can then access these manifolds because the solutions they contain are not separated from each other by wide regions composed of poorly performing configurations. First studied in 12 , this new statistical ensemble unveils the structure of the flat manifolds present in the perceptron loss landscape. More practically, it also led to the design of modified Monte Carlo algorithms that access these flat regions. The main limitation of the description proposed in 12 is its reliance on the no-memory Ansatz . In a few words, this hypothesis allows one to carry out analytical computations by fixing the geometry of the solution paths. Although these no-memory paths exist and are accessible, they do not correspond to the most numerous paths in the perceptron loss landscape. In this paper, we will focus on characterizing flat manifolds beyond the no-memory Ansatz . To maintain continuity with 12 , we study the same model, namely the symmetric binary perceptron (SBP). This will allow us to address the differences between the more general flat manifolds and the no-memory ones. The remainder of this paper is organized as follows. In Sec. II , we introduce the perceptron model and define -in a generic form- the statistical ensemble for connected minima. In Sec. III , we analyze the saddle point of the connected free energy. In particular, we will address the geometry of the paths, their robustness, and the algorithmic thresholds for training the SBP. II Definitions and connected