Dropout Universality: Scaling Laws and Optimal Scheduling at the Edge-of-Chaos
Lucas Fernandez Sarmiento · 2026
Read Dropout Universality: Scaling Laws and Optimal Scheduling at the Edge-of-Chaos by Lucas Fernandez Sarmiento (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-21648.txt.
Introduction
Lucas Fernandez Sarmiento asks whether dropout should stay a static setting or vary over depth. He develops a mean-field account of dropout near the edge of chaos, finding distinct classes for smooth and kinked units with their scaling powers. The resulting propagation theory, bound by maximizing dropout's regularization, points to putting dropout near the input. Tests on vision, speech, and finance time series show gains often in MLPs, with smaller gains in attention models.
Catalog record
| CREATOR | Lucas Fernandez Sarmiento |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 99148 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.21648
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Dropout Universality: Scaling Laws and Optimal Scheduling at the Edge-of-Chaos Lucas Fernandez Sarmiento Abstract We develop a mean-field theory of dropout as a perturbation of critical signal propagation at the edge of chaos, and show that it predicts a simple, no-cost change to standard practice: front-loaded dropout schedules cut test loss by 18 18 – 35 % 35\% over constant dropout in MLPs and Vision Transformers at fixed budget. The theoretical mechanism is that dropout shifts the perfect-alignment fixed point, making the depth scale for information propagation finite even at critical initialization. We derive critical and crossover scaling laws for correlation decay and establish that smooth activations and kinked, ReLU \ReLU -like activations constitute distinct universality classes, with different critical exponents and a universal two-parameter scaling collapse in detuning and dropout strength. The distinction traces to the analytic structure of the correlation map: smooth activations admit a Taylor expansion near perfect alignment, while kinked activations develop a branch point with universal non-analyticity. As a corollary, the framework yields saturated dropout profiles under fixed budget; a regularization-reach argument then selects front-loaded schedules, with accuracy gains as a consistent secondary effect. We also discuss how the same Gaussian-kernel structure extends the theory beyond MLPs toward CNNs and residual architectures. 1 Introduction Mean-field analyses of randomly initialized deep networks reveal an order-to-chaos phase diagram that controls both signal propagation and gradient penetration with depth ( 30 ; 33 ; 4 ; 3 ) . In this work, we study how this picture is modified by dropout, and use the resulting theory to derive a no-cost change to standard practice: we find that front-loaded dropout schedules cut test loss by 18 – 35\% over constant dropout at fixed budget in MLPs and Vision Transformers (Table 3 ). Using the representation-group (coarse-graining) language of ( 32 ; 3 ) , we show that dropout behaves as a relevant perturbation: it displaces the critical fixed point and grows under depth coarse-graining, pushing the dynamics away from criticality and determining the macroscopic phase. Concretely, dropout deforms the correlation map ( 33 ) by adding a correlation-independent shift at perfect alignment, so that perfect correlation between inputs is no longer a fixed point for any nonzero dropout. Thus, even at the edge-of-chaos ( 34 ; 26 ; 5 ) , the depth correlation length is finite. We interpret this shift using Landau theory, as an equation of state for the decorrelation order parameter m\equiv 1-c^{\ast} (with c^{\ast} the asymptotic inter-signal correlation), derive the resulting scaling laws, and show that smooth and kinked, \ReLU -like activations exhibit the same qualitative phase diagram but different critical scaling in the presence of dropout. Additionally, we show that the two control parameters (dropout strength and distance to criticality) admit a scaling collapse into a single universal form, 1 1 1 This is analogous to the homogeneous scaling of connected correlation functions near a critical point; related scaling forms hold for one-particle-irreducible vertex functions after the usual Legendre transform ( 41 ) . and we illustrate how these predictions compare with mean-field recursions. The main conceptual contributions are threefold: first, while dropout was previously shown to destroy the order-to-chaos critical point ( 33 ) , we show that the deformed mean-field map still retains a nontrivial c^{\ast}<1 fixed point. This allows us to have a well-defined correlation length recursion even in the presence of dropout, and yields a Landau equation of state for m=1-c^{\ast} together with a two-parameter scaling collapse. Second, we show that universality classes and scaling laws are set by the analytic structure of activations: smooth and kinked activations have distinct critical exponents (Table 2 ), a distinction reflected in their Hermite spectral structure (App. C ). Since the same Gaussian activation kernels arise beyond MLPs, including CNNs and residual branches (App. A.4 ), we expect the qualitative smooth/kinked split to be a well-motivated heuristic. Third, we treat dropout as a depth-dependent dynamical field; treating classically fixed hyperparameters as fields that can change through the network opens a new dimension for hyperparameter optimization across depth, and potentially over training time. This work complements prior studies of depth-dependent regularization: stochastic depth drops whole residual blocks with increasing probability ( 17 ) , curriculum dropout anneals dropout over training time ( 24 ) , and LayerDrop removes transformer layers for efficiency ( 11 ) . Our schedules are spatial depth profiles, so they are complementary to temporal curricula and layer-dropping mechanisms. Qualitatively different edge-of-chaos behavior for smooth and \ReLU -like activations was previously observed in ( 12 ) ; here we extract the scaling exponents and identify the corresponding universality classes. 2 2 2 Code, configurations, seeds, and figure-generation scripts are available in the commit-pinned dropout-universality-experiments repository. See App. D for the full reproducibility statement. Conflict of Interest Disclosure. The author declares no financial conflicts of interest related to this work. 2 Mean-Field Theory Background We first state the mean-field assumptions for MLPs, which provide the baseline for the dropout analysis ( 30 ; 33 ; 4 ; 3 ) . We consider fully-connected MLPs at random initialization, where preactivations are well-approximated as Gaussian random variables with self-consistently determined statistics (means, variances, covariances). In the strict infinite width limit this Gaussian description becomes exact ( 23 ) , while at large but finite width deviations from Gaussianity can be organized perturbatively in the depth-to-width aspect ratio (schematically L/N ), as treated comprehensively in ( 32 ; 3 ) . The intuition behind this treatment is that as we add more activations, through the CLT, the statistics will become more and more Gaussian, approaching this distribution asymptotically in the large N limit. We consider an MLP of depth L , N_{l} neurons in layer l , and an activation \phi:\mathbb{R}\to\mathbb{R} (e.g. \tanh , \ReLU , etc. ). We use the standard i.i.d. Gaussian initialization W^{l}_{ij}\sim\mathcal{N}\left(0,\frac{\sigma_{w}^{2}}{N_{l}}\right),\qquad b^{l}_{i}\sim\mathcal{N}\left(0,\sigma_{b}^{2}\right), (1) take the input as y^{0}_{i;a}=x_{i;a} , and propagate forward via z^{l}_{i;a}=W^{l}_{ij}y^{l}_{j;a}+b^{l}_{i},\qquad y^{l+1}_{i;a}=\phi\left(z^{l}_{i;a}\right), (2) where a labels the input. Throughout, expectations \mathbb{E}[\cdot] are over the random weights and biases at initialization, with inputs held fixed. We also use the standard Gaussian measure \int Dz(\cdots)\;\equiv\;\frac{1}{\sqrt{2\pi}}\int_{-\infty}^{\infty}dze^{-z^{2}/2}(\cdots), (3) and similarly for \int Dz_{1}Dz_{2} . In the infinite-width limit this becomes exact layer-by-layer, while at finite width the first layer is exactly Gaussian (being a linear map of fixed inputs) and subsequent layers are only approximately Gaussian, with controlled non-Gaussian corrections parametrized by the depth-to-width ratio ( 32 ) . A fuller treatment of the MFT background is given in App. A.2 . We track the single-input preactivation variance q^{l}_{aa} , the two-input covariance q^{l}_{ab} , and the induced correlation c^{l}_{ab} : \displaystyle q^{l}_{aa} \displaystyle=\sigma_{w}^{2}\int Dz\phi^{2}\left(\sqrt{q^{l-1}_{aa}}z\right)+\sigma_{b}^{2} (4) \displaystyle q^{l}_{ab} \displaystyle=\sigma_{w}^{2}\int Dz_{1}Dz_{2}\phi(u_{1})\phi(u_{2})+\sigma_{b}^{2} (5) \displaystyle u_{1} \displaystyle=\sqrt{q^{l-1}_{aa}}z_{1},\qquad c^{l-1}_{ab}=\frac{q^{l-1}_{ab}}{\sqrt{q^{l-1}_{aa}q^{l-1}_{bb}}}, (6) \displaystyle u_{2} \displaystyle=\sqrt{q^{l-1}_{bb}}\left(c^{l-1}_{ab}z_{1}+\sqrt{1-\left(c^{l-1}_{ab}\right)^{2}}z_{2}\right). (7) In the absence of dropout, for a wide array of activations q^{l}_{ab},c^{l}_{ab} settle to fixed points q^{*},c^{*} ; in particular, there exists a c^{*}=1 fixed point as a straightforward calculation shows. The variance fixed point sets the typical activation scale, while the correlation fixed point describes whether distinct inputs become indistinguishable under depth iteration. We denote the recurrence relation for c^{l}_{ab} by c^{l}_{ab}=F(c^{l-1}_{ab}) . To probe its stability, one linearizes the map at c=1 , defining the (angular) susceptibility \chi_{\perp}\equiv\left.\frac{\partial c^{l}_{ab}}{\partial