Sharp feature-learning transitions and Bayes-optimal neural scaling laws in extensive-width networks
Minh-Toan Nguyen, Jean Barbier · 2026
Read Sharp feature-learning transitions and Bayes-optimal neural scaling laws in extensive-width networks by Minh-Toan Nguyen, Jean Barbier (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-10395.txt.
Introduction
Minh-Toan Nguyen and Jean Barbier study the limits of learning a one-hidden-layer teacher net with hierarchical features from noisy queries. The goal is transfer to a smaller student model. Their 2026 paper works in the high-dimensional regime where teacher width scales linearly with input dimension, a setting only recently made tractable.
Using a leave-one-out argument, validated numerically, the authors derive sharp accounts of the Bayes-optimal generalization error and individual feature overlaps via fixed-point equations. Those equations reveal sharp phase transitions: with more data, teacher features turn recoverable one by one, each through a sudden jump in overlap. That defines an effective width: learnable features at a given data budget. That unifies feature-learning and refinement regimes under one scaling relation. A student trained with Adam near the effective width achieves those optimal scaling laws up to a small algorithmic gap.
Catalog record
| CREATOR | Minh-Toan Nguyen, Jean Barbier |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 82 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.10395
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Sharp feature-learning transitions and Bayes-optimal neural scaling laws in extensive-width networks Minh-Toan Nguyen, Jean Barbier Abstract We study the information-theoretic limits of learning a one-hidden-layer teacher network with hierarchical features from noisy queries, in the context of knowledge transfer to a smaller student model. We work in the high-dimensional regime where the teacher width k k scales linearly with the input dimension d d – a setting that captures large-but-finite-width networks and has only recently become analytically tractable. Using a heuristic leave-one-out decoupling argument, validated numerically throughout, we derive asymptotically sharp characterizations of the Bayes-optimal generalization error and individual feature overlaps via a system of closed fixed-point equations. These equations reveal that feature learnability is governed by a sequence of sharp phase transitions: as data grows, teacher features become recoverable sequentially, each through a discontinuous jump in overlap. This sequential acquisition underlies a precise notion of effective width k c k_{c} – the number of learnable features at a given data budget n n – which unifies two distinct scaling regimes: a feature-learning regime in which the Bayes-optimal generalization error ε BO \varepsilon^{\rm BO} scales as n 1 / ( 2 β ) − 1 n^{1/(2\beta)-1} , and a refinement regime in which it scales as n − 1 n^{-1} , where β > 1 / 2 \beta>1/2 is the exponent of the power-law feature hierarchy. Both laws collapse to the single relation ε BO = Θ ( k c d / n ) \varepsilon^{\rm BO}=\Theta(k_{c}d/n) . We further show empirically that a student trained with Adam near the effective width k c k_{c} achieves these optimal scaling laws (up to a small algorithmic gap), and provide an information-theoretic account of the associated scaling in model size. 1 Introduction Figure 1: Adam -trained students (all curves except the purple one, right panel) on a teacher with \sigma(x)=\tanh(2x) (with the 2x matching the convention in 6 ), k=100 , d=200 , \Delta=0.01 , power-law readout with \beta=1 (3 batches, 10^{4} epochs, learning rate 0.003 ; averages over 9 runs, shaded regions show \pm 1 standard deviation). Left: At fixed sample rate n/(kd)=2 , a reduced-width student (solid = test error; faint = training error) outperforms the full-width one. Middle: Generalization error vs. student width k_{s} at several sample rates n/(kd) . Performance degrades beyond a critical width; the dashed line indicates the theoretically predicted optimal model-size exponent 1-2\beta . Right: Generalization error vs. data size n for three width choices – full-width ( k_{s}=k ), effective width ( k_{s}=k_{c} from theory), and optimized width (best-performing k_{s}\in\{10,20,\ldots,100\} found by grid search) – together with the Bayes-optimal prediction (purple). Dashed lines indicate the predicted scaling exponents: 1/(2\beta)-1 in the feature-learning regime and -1 in the refinement regime , see Section 3 and Appendix G for the derivation of exponents. The performance of large neural networks often follows simple power laws in data, compute, and model size 26 ; 31 ; 28 . A growing line of theory attributes this regularity to a hierarchical organization of the target function: features carry geometrically decaying importance and a learner acquires them roughly in order 29 ; 34 ; 2 ; 36 ; 22 ; 20 . In a teacher-student setting – the basic primitive behind data-driven model distillation 11 ; 27 ; 25 – this hierarchy raises an immediate design question: at a fixed data budget, how should one size the student? Figure 1 exhibits two empirical regularities for an Adam -trained student. (i) There is a critical width beyond which the test error increases (middle panel); below this threshold, the error follows a power law in model size with exponent 1-2\beta for a power-law feature hierarchy of exponent \beta>1/2 . (ii) At or near the optimal width, the error obeys two distinct power laws in n (right panel): a slow regime with exponent 1/(2\beta)-1 and a faster regime with exponent -1 . These are striking but mechanism-agnostic: they could reflect properties of Adam , or fundamental information-theoretic constraints. To disentangle the two, we lift all computational constraints and adopt a Bayes-optimal viewpoint, asking: Q1. Given a fixed dataset, which features of the teacher are statistically recoverable? Q2. How does the recoverable subset of features constrain the scaling law of the Bayes-optimal generalization error \varepsilon^{\rm BO} in the data size n ? We address Q1 and Q2 in a one-hidden-layer teacher–student model in the extensive-width regime k=\Theta(d) , with k the teacher width and d the input dimension. This regime captures the large-but-finite width of practical networks, lies strictly between the well-studied multi-index ( k=\Theta(1) ) and NTK ( k\gg d ) extremes, and is the minimal setting for non-trivial feature learning beyond the linear data scale ( n\gg d ). Our analytical handle is a heuristic leave-one-out (cavity) decoupling argument, validated numerically throughout. Paper organization. Section 1.1 introduces the model and hypotheses. The main results – fixed-point equations for the feature overlaps and Bayes-optimal error, and the ensuing scaling laws – are stated in Section 2 and analyzed in Section 3 , which covers the phase transitions, the error decomposition, the two scaling regimes and their unification via k_{c} , and the Adam -trained student. The leave-one-out derivation is in Section 4 . Section 5 concludes. Derivations, numerical methods, and additional asymptotic results are collected in the Appendices. The code for the simulations in this paper is publicly available at https://github.com/Minh-Toan/scaling-laws . 1.1 Problem setup Notations. In addition to the standard O -notation, we write A\simeq B if A-B=o(1) ; A\sim B if A/B\simeq 1 , and with a slight abuse of notation, we also use \sim to indicate a random variable following a probability law; A\gtrsim B means A\geq cB for some absolute constant c>0 ; A=\Theta(B) if cB\leq A\leq CB for absolute c,C>0 ; A=\Theta_{p}(B) if c,C may depend on parameter p . Teacher and dataset. Given inputs \mathbf{x}_{\mu} drawn i.i.d. from \mathcal{N}(0,\mathbf{I}_{d}) and noise variables z_{\mu} that are i.i.d. standard Gaussian, the dataset \mathcal{D}=(\mathbf{x}_{\mu},y_{\mu}^{\rm out})_{\mu\in[n]} consists of input-output pairs with noisy responses y_{\mu}^{\rm out}=y_{\mu}+\sqrt{\Delta}\,z_{\mu} and clean responses generated by the teacher neural network: \displaystyle y_{\mu}=\mathbf{v}^{\top}\sigma(\mathbf{W}^{0}\mathbf{x}_{\mu}). (1) \sigma is an element-wise activation, \mathbf{v}\in\mathbb{R}^{k} is the readout vector, and \mathbf{W}^{0}\in\mathbb{R}^{k\times d} is the teacher’s weight matrix. Each row \mathbf{w}_{i}^{0} of \mathbf{W}^{0} defines a latent direction –a feature – used to project the input. Students. We study two types of learners. The Bayesian student knows the data-generation mechanism (activation \sigma , prior P_{0} , noise variance \Delta , and readout \mathbf{v} 1 1 1 Assuming \mathbf{v} is known to the Bayesian student is a convenient simplification. However, in the scaling regime we consider, k=\Theta(d) , it does not change the Bayes-optimal (BO) error compared to the case where \mathbf{v} is unknown as proved in 6 . ) but not the specific realization \mathbf{W}^{0} . For a new input \mathbf{x} , it predicts the corresponding teacher’s output by computing the posterior mean \mathbb{E}[\mathbf{v}^{\top}\sigma(\mathbf{W}\mathbf{x})\mid\mathcal{D}] , which achieves the minimum mean square error, also known as the Bayes-optimal error \varepsilon^{\rm BO} . On the other hand, the SGD student is a one-hidden-layer network of width k_{s}\leq k and same input dimension d , with weights (\mathbf{W},\mathbf{v}) trained with Adam . The L_{2} norm of each