Gradient Descent as a Shrinkage Operator for Spectral Bias
Simon Lucey · 2025
Read Gradient Descent as a Shrinkage Operator for Spectral Bias by Simon Lucey (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-18207.txt.
Introduction
Neural networks learn some patterns before others, favoring smooth, low frequency structure, a tilt called spectral bias. Simon Lucey reframes that tilt in 2025 through the training process itself, gradient descent. He shows the descent acts like a shrinker, quietly masking parts of the network's response and so choosing how many frequency parts stay active.
That choice ties directly to two knobs every trainer sets: the step size and the number of steps. The shrinking works only for activation shapes that rise steadily, he finds. Wavy activations, by contrast, reach the same control with far fewer steps. The author offers the view as a cleaner handle on why networks learn what they learn first.
Catalog record
| CREATOR | Simon Lucey |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-18207
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Gradient Descent as a Shrinkage Operator for Spectral Bias Simon Lucey Abstract We generalize the connection between activation function and spline regression/smoothing and characterize how this choice may influence spectral bias within a 1D shallow network. We then demonstrate how gradient descent (GD) can be reinterpreted as a shrinkage operator that masks the singular values of a neural network’s Jacobian. Viewed this way, GD implicitly selects the number of frequency components to retain, thereby controlling the spectral bias. An explicit relationship is proposed between the choice of GD hyperparameters (learning rate & number of iterations) and bandwidth (the number of active components). GD regularization is shown to be effective only with monotonic activation functions. Finally, we highlight the utility of non-monotonic activation functions (sinc, Gaussian) as iteration-efficient surrogates for spectral bias. 1 Introduction Spectral bias – the tendency of neural networks to prioritize learning low frequency functions [ 4 ] – is often touted as a potential explanation for the extraordinary generalization properties of modern deep neural networks. Let’s consider the estimation f(x) of a continuous signal y(x) using a finite number of samples \mathbb{D} such that, \arg\min_{f}\frac{1}{2}\sum_{\hat{x}\in\mathbb{D}}||y(\hat{x})-f(\hat{x})||^{2} (1) where f is parameterized using a so called shallow network, f(x)=\sum_{m=0}^{M-1}w_{m}\cdot\eta(x-b_{m}) (2) \mathbf{w}=[w_{0},\ldots,w_{M-1}]^{T} are the network weights, and \mathbf{b}=[b_{0},\ldots,b_{M-1}]^{T} are the bias terms that shift the activation function \eta(x) . M is typically referred to as the width of the network. For the purposes of this paper we will assume we are optimizing Eq. 1 with gradient descent (GD) as is common practice. When using such networks for signal reconstruction we are often faced with seemingly adhoc design questions around: (i) the type of activation function, (ii) the width of the network, and (iii) the choice of learning rate and number of iterations. Our goal in this paper is to better understand how and when these choices explicitly effect spectral bias. Figure 1 : We argue that the implicit regularization offered by gradient descent (GD) can be viewed as a shrinkage operator on the singular values \mathbf{s}=[s_{0},\ldots,s_{N}] of a network’s Jacobian \nabla f(\mathbf{x}) . The operator applies a masking function m(s;\alpha,q) on the singular values, where \alpha is learning rate and q is the number of iterations which dictate the number K of active principal components. The distribution of \mathbf{s} is activation specific, and therefore how effectively it can be regularized through GD. Contributions: In Sec. 3 we generalize the connection between activation function choice and the minimization of gradient energy in spline regression/smoothing. We then show in Sec. 4 how GD can be viewed as a shrinkage operator on the singular values of a network’s Jacobian \nabla f(\mathbf{x}) (see Fig. 1 ). Further, we show that activation functions (Heaviside, tanh) have singular values that relate directly to a frequency basis – specifically the discrete sine transform (DST). For modern activations (e.g. ReLU, GELU, SiLU) the approximation to the frequency basis becomes less strict but is still apparent. An explicit relationship between the number of active principal components and GD hyper-parameters is then established. A heuristic is proposed between (learning rate, number of iterations) and spectral-bandwidth for activations. We are the first, to our knowledge, to make explicit such a relationship. Finally, we demonstrate that the manipulation of (learning rate, number of iterations) – as a strategy for implicit regularization – is only effective for monotonic activations. Non-monotonic activations (sinc, Gaussian, etc.) instead rely on scaling as their method for implicit regularization. We highlight sinc in particular as an iteration efficient surrogate for enforcing spectral bias. 2 Background For the case of least-squares it is understood that GD will converge to the minimum Euclidean norm solution [ 19 ] . The interplay between activation function and GD hyper-parameter choice, however, in this regularization process is not as well understood for neural networks, and is the central focus of this paper. Activation bias : ReLU networks tend to learn low frequency components first when being trained using GD. Sinusoid activations were proposed [ 20 , 18 ] to allow these networks to model higher bandwidth signals. NTK analysis [ 10 ] was used by Tancik et al. [ 20 ] to explain this behaviour and motivate the use of a sinusoid activated first layer – referred to as Fourier positional encoding – within a deep ReLU network. Sitzman et al. [ 18 ] then demonstrated that a fully sinusoid activated deep network – suitably initialized – can achieve even better performance. Ramasinghe et al. [ 14 ] argued that exponential family activations like Gaussian can achieve similar performance without requiring specialized initialization. Recently, it has been noted empirically, that sinc [ 16 ] activations can achieve even better results. A focus of this paper is to make explicit this spectral bias for the case of a 1D shallow network. GD vs. SGD : In spite of previous works [ 1 , 7 ] stating that stochastic sampling is essential – many empirical studies now exist that show full-batch GD optimized networks generalize well. Hoffer et al. [ 9 ] demonstrated with a sufficient number of iterations, full-batch GD generalizes similarly to SGD. Likewise, Geiping et al. [ 6 ] empirically found that generalization behavior persists even without stochastic sampling. For smooth losses – such as the \ell_{2} loss been considered in Eq. 1 – it has been shown [ 13 ] that GD exhbits tighter generalization bounds than SGD and therefore will be of central focus in this paper. ReLU dilemma : A classic activation used throughout modern neural network literature [ 12 ] is \mbox{ReLU}(x)=\max(x,0) . What has surprised much of the ML community in recent years, however, is how poorly this popular activation performs for low dimensional signal reconstruction problems like the one described in Eq. 1 . What makes this even more confusing is seminal work by Zhang et al. [ 26 ] who proved that a 1D shallow ReLU network like the one described in Eq. 2 could perfectly reconstruct training samples \mathbb{D} provided the width and distribution of \mathbf{b} is sufficient. A drawback to this analysis, however, is that it ignores the optimization process required to find the solution – namely GD. 3 Smoothness & Activations Smoothness stems directly from the well established framework of empirical risk minimization (ERM) [ 22 ] , and has been used since the earliest days of ML to quantify generalization (in regression). In summary, given a set of hypotheses (models) that minimizes the empirical risk (with training data), the ERM framework prefers a solution that minimizes the true risk (with respect to the actual data distribution) with a higher probability. When extra prior knowledge is unavailable on the true data distribution, ERM suggests that the best solution would be the one that minimizes the least complex solution that minimizes the empirical risk (under the realizability assumption). This can be primarily achieved through: (i) regularizing the parameters of the model or (ii) regularizing the function output itself. Popular regularizations such as ridge regression fall into the first category, whereas spline regression with regularized derivatives fall into the second category. Connecting (i) and (ii) : Spline regression/smoothing [ 15 ] makes no parametric assumption on f , instead it adds an explicit regularization