Singular Learning and Occam’s Razor in Deep Monomial Networks
Kathlén Kohn, Giovanni Luca Marchetti, Farhan Shabir, Vahid Shahverdi, Weisheng Wang · 2026
Read Singular Learning and Occam’s Razor in Deep Monomial Networks by Kathlén Kohn, Giovanni Luca Marchetti, Farhan Shabir, Vahid Shahverdi, Weisheng Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-28464.txt.
Introduction
Kathlén Kohn, Giovanni Luca Marchetti, Farhan Shabir, Vahid Shahverdi, and Weisheng Wang explain a quiet bias of deep learning: trained networks tend toward simpler functions than they could express. Singular learning theory locates the cause at critical points where the model's Jacobian loses rank, points baked in by the architecture itself.
Focusing on deep fully connected networks whose activation functions are monomials, pure powers, the team brings polynomial algebra to bear, including Mason's theorem from the theory of polynomial equations. They show that once the activation degree is large enough, criticality occurs precisely at subnetworks, configurations where some neurons are switched off or redundant. Gradient descent is thus funneled toward effectively smaller networks, a mathematical account of Occam's razor inside deep learning.
Catalog record
| CREATOR | Kathlén Kohn, Giovanni Luca Marchetti, Farhan Shabir, Vahid Shahverdi, Weisheng Wang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.28464
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Singular Learning and Occam’s Razor in Deep Monomial Networks Kathlén Kohn, Giovanni Luca Marchetti, Farhan Shabir, Vahid Shahverdi, Weisheng Wang Abstract In the optimization of neural networks, gradient dynamics are influenced by critical points that arise from the model’s architecture. These critical points occur where the Jacobian of the model’s parametrization is rank-deficient, and are the most pronounced singularities studied in Singular Learning Theory. We investigate such points in deep fully-connected networks with monomial activations via tools from polynomial algebra such as Mason’s Theorem. We show that, for sufficiently large activation degree, criticality occurs precisely at subnetworks, i.e., at parameter configurations where some neurons are inactive or redundant. This offers a mathematical perspective on the implicit bias in deep neural networks, explaining the tendency of these models to converge toward simpler functions. Keywords: critical points, neuromanifolds, singular learning theory 1 Introduction Deep neural networks are typically trained by following (a discretization of) the gradient flow of their objective. The equilibria of this dynamics are the critical points of the objective, i.e., parameters where the gradient vanishes. These can be local optima (maxima and minima), or possibly-degenerate saddles. Interestingly, some critical points are induced by the model and its architecture [ 34 ] , rather than depending directly on the given data or objective. They arise when the parametrization of the model exhibits a Jacobian that is rank-deficient. At these critical parameters, the Riemannian metric induced by the model – typically, the Fisher Information Metric – drops rank. Based on this, such parameters are also referred to as ‘singularities’, and are the central focus of Singular Learning Theory (SLT) [ 38 ] . According to SLT, critical points of the parametrization play a role of implicit biases in learning, from both dynamical and Bayesian perspective [ 41 ] . In this work, we study the critical points of the parametrization of a certain class of neural networks. Namely, we consider deep fully-connected networks with a monomial activation function – a setting introduced in [ 15 ] , and later developed in several works [ 9 , 35 , 18 , 26 , 1 , 7 ] . For these models, the end-to-end function is a (homogeneous) multivariate polynomial. This makes them particularly amenable to theoretical analysis, since it is possible to apply powerful tools of polynomial algebra. In fact, this work fits into a broad research program aimed at bridging algebra and algebraic geometry with deep learning – a field recently termed as Neuroalgebraic Geometry [ 25 ] . As a main result, we prove that, for sufficiently large activation degree, the critical points of the parametrization are precisely subnetworks , i.e., parameters of the network where some neurons are redundant or inactive. Equivalently, rank deficiency occurs exactly at parameter configurations where some hidden neuron can be removed without changing the realized function. In light of the relation between critical points and implicit biases, this can be interpreted as providing a mathematical perspective on the tendency of deep neural networks to converge, during training, towards ‘simple’ functions – more specifically, towards configurations where a number of neurons can be pruned. This tendency is a crucial behavior, suggesting that deep neural networks implement, implicitly, a form of the Occam’s Razor. Our result complements the main findings of [ 9 ] , where it is shown that subnetworks are the only non-identifiable parameters 1 1 1 This statement is stronger than the original phrasing in [ 9 ] , but follows from the same proof – see Section 3.2 . , meaning that they admit larger symmetries than expected. Our proof strategy, however, differs from [ 9 ] . While the latter relies on a polynomial version of Fermat’s Last Theorem due to Newman-Slater [ 28 ] , we leverage on a stronger tool: Mason’s polynomial version of the abc -conjecture from number theory. By combining Mason’s Theorem with a subtle divisibility analysis of the polynomials defined by the network, we are able to describe the kernel of the Jacobian of the parametrization. 2 Related Work Algebraic geometry of deep learning. As anticipated, a line of research in theoretical deep learning – recently termed Neuroalgebraic Geometry [ 25 ] – explores the study of polynomial neural networks through the lens of algebraic geometry. Motivated by the fact that polynomials can approximate arbitrary continuous functions, the goal of Neuroalgebraic Geometry is to approach fundamental problems in machine learning via tools from algebra and geometry. One such problem is identifiability, which represents a core focus of neuroalgebraic geometry. Several models have been analyzed in the literature, ranging from deep fully-connected networks with linear [ 34 , 22 ] , monomial [ 15 , 9 , 18 , 23 , 26 ] , and general polynomial [ 32 , 30 ] activations, to deep convolutional networks with linear [ 16 , 17 , 33 ] , monomial [ 31 , 13 ] and polynomial activations [ 32 ] , to (un-normalized) attention-based networks [ 14 ] . Our work naturally fits into this line of research, since we consider fully-connected networks with monomial activations of large degree. We contribute by describing the critical points of the parametrization – one of the basic questions in Neuroalgebraic Geometry, that has been answered only for linear fully-connected [ 34 ] and polynomial convolutional [ 17 , 31 ] networks. Singular learning theory (SLT). SLT is a discipline in theoretical machine learning focused on singularities 2 2 2 The notion of singularity in SLT does not fully coincide with the one in algebraic geometry – see [ 32 ] for details. – points where the Fisher Information Metric induced in parameter space by a given model degenerates [ 38 , 37 , 2 , 41 ] . According to SLT, singularities induce biases in the learning process. This has been analyzed, mathematically, both from a dynamical [ 40 ] and probabilistic [ 39 ] perspective. Now, singularities are closely related to critical points of the parametrization – we expand on this in Section 4.1 . Thus, our work contributes to the SLT picture by relating singularities to subnetworks (for deep monomial networks). As mentioned in Section 1 , this can be interpreted as a mathematical justification of the implicit simplicity bias of contemporary models. Implicit sparsity bias. The tendency of deep neural networks to converge to sparse weights is well-established. Empirically, this phenomenon has been popularized by the celebrated ‘lottery ticket hypothesis’ [ 10 ] . Theoretically, a line of research [ 42 , 27 , 29 , 3 ] has shown that, for deep diagonal linear networks, the training dynamics, when initialized around the origin, implicitly penalizes the \ell_{1} norm of the weights, inducing sparsity in the representation. A similar bias towards low-rank solutions has been shown for deep linear networks trained with a regularized loss [ 19 , 43 , 36 ] . Closely related to our work, these sparsity biases have been reformulated in terms of subnetworks [ 5 ] , and even analyzed dynamically in simple scenarios [ 20 , 24 ] . While our analysis is not dynamical, our work contributes to sedimenting the fundamental role of subnetworks in the learning process, for an interesting