Sven: Singular Value Descent as a Computationally Efficient Natural Gradient Method

Samuel Bright-Thonney, Thomas R. Harvey, Andre Lukas, Jesse Thaler · 2026

Read Sven: Singular Value Descent as a Computationally Efficient Natural Gradient Method by Samuel Bright-Thonney, Thomas R. Harvey, Andre Lukas, Jesse Thaler (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-01279.txt.

Introduction

Bright-Thonney and colleagues introduce Sven, a method for neural networks that exploits how loss splits across data points. Rather than collapsing loss to one scalar before updating parameters, Sven treats residuals from each data point as goals to satisfy at once. It finds the smallest update via the loss Jacobian's Moore-Penrose pseudoinverse. That pseudoinverse is approximated by a truncated singular value decomposition keeping only the k top ways. Tests on regression, classification, and small transformer modeling show Sven rivals Adam, Muon, and K-FAC.

Catalog record

CREATORSamuel Bright-Thonney, Thomas R. Harvey, Andre Lukas, Jesse Thaler
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59875
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.01279

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Sven: Singular Value Descent as a Computationally Efficient Natural Gradient Method Samuel Bright-Thonney, Thomas R. Harvey, Andre Lukas, Jesse Thaler Abstract We introduce Sven (Singular Value dEsceNt), a new optimization algorithm for neural networks that exploits the natural decomposition of loss functions into a sum over individual data points, rather than reducing the full loss to a single scalar before computing a parameter update. Sven treats each data point’s residual as a separate condition to be satisfied simultaneously, using the Moore-Penrose pseudoinverse of the loss Jacobian to find the minimum-norm parameter update that best satisfies all conditions at once. In practice, this pseudoinverse is approximated via a truncated singular value decomposition, retaining only the k k most significant directions and incurring a computational overhead of only a factor of k k relative to stochastic gradient descent. This is in comparison to traditional natural gradient methods, which scale as the square of the number of parameters. We show that Sven can be understood as a natural gradient method generalized to the over-parametrized regime, recovering natural gradient descent in the under-parametrized limit. On regression tasks, Sven significantly outperforms standard first-order methods including Adam, converging faster and to a lower final loss, while remaining competitive with LBFGS at a fraction of the wall-time cost. We discuss the primary challenge to scaling, namely memory overhead, and propose mitigation strategies. Beyond standard machine learning benchmarks, we anticipate that Sven will find natural application in scientific computing settings where custom loss functions decompose into several conditions. MIT-CTP/6022 1 Introduction Every standard loss function is a sum. Whether fitting a regression model or training a classifier, the objective decomposes into a sum over data points, with each term encoding an individual condition the model should satisfy. Yet despite this structure, the dominant paradigm in machine learning (ML) discards it immediately. Gradient descent reduces the entire collection of conditions to a single scalar and updates parameters accordingly, treating the decomposition as an implementation detail rather than a source of information. In this paper, we develop a new optimizer for neural networks which takes a global view of this loss decomposition. Our methodology, which we call Sven ( S ingular V alue d E sce N t), can be viewed as an efficient approximation to natural gradient methods, extrapolated to the over-parametrized regime. At each training step, rather than computing a single gradient direction for the total loss, Sven asks a more ambitious question: given the individual residuals of every data point in the batch, what single parameter update would bring each of them closest to zero simultaneously? This is a linear algebra problem, the solution to which makes use of the Moore-Penrose pseudoinverse [ 27 , 28 ] . In practice, computing the full pseudoinverse is expensive, so Sven approximates it via a truncated singular value decomposition (SVD), retaining only the k most significant directions. The result is an optimizer that is aware of the geometry of the loss landscape in a way that standard gradient descent is not. We focus on small scale regression experiments, though scaling to larger models remains an important direction for future work. Scaling this approach presents an unusual challenge: unlike most optimization methods where computational cost is the primary bottleneck, here memory is the limiting factor. The computational overhead (in terms of operations) is only a factor of k larger relative to stochastic gradient descent (SGD), where k is an integer hyperparameter to be discussed later. We discuss strategies for mitigating the memory cost in Appendix C , the most promising of which would require a modification of standard autograd tools that is beyond the scope of this paper. Despite this modest computational overhead, Sven outperforms Adam and other standard first-order methods in our regression experiments, both in convergence speed and final training loss. The remainder of this paper is organized as follows. In Section 2 , we derive the Sven update rule, discuss its relation to natural gradient methods in the under-parametrized limit, and describe its generalization to the over-parametrized regime. Section 3 surveys related work. Section 4 presents our experimental results on 1D regression, polynomial regression, and MNIST classification. Finally, Section 5 concludes and outlines directions for future work. Appendix A provides a pedagogical discussion of natural gradients from a functional perspective, Appendix B puts the present work into the context of functional analysis and differential geometry, Appendix C discusses strategies for mitigating the memory overhead of Sven, and Appendix D contains further experimental details. Beyond standard ML benchmarks, this methodology finds natural application in scientific computing, with an upcoming application to the numerical modular bootstrap presented in forthcoming work [ 4 ] . 2 Methodology In ML, a common problem is the minimization of a loss function composed of several components: L(\theta)=\sum_{\alpha\in\aleph}\ell^{\alpha}(\theta), (1) where each sub-loss \ell^{\alpha}(\theta) is non-negative: \ell^{\alpha}(\theta)\geq 0\quad\forall\,\theta\in\mathbb{R}^{m},\alpha\in\aleph. (2) Each term \ell^{\alpha} encodes a condition the model should satisfy, and the overall loss aggregates these conditions into a single objective. This structure naturally arises, for instance, in the L_{2} -loss for regression and the cross-entropy loss for classification, both of which decompose as a sum over individual data points. From now on, we take the index set \aleph to enumerate individual data points, keeping the more general framework in mind and returning to it when relevant. Algorithm 1 Sven update step. The truncated SVD considers only the k largest singular values, and sets singular values that are a factor of rtol smaller than the largest singular value to zero. Input: Jacobian M , residual vector \mathcal{R} , rank k , tolerance rtol , parameters \theta , learning rate \eta Output: Updated parameters \theta^{\prime} {Compute truncated SVD} U,s,V^{T}\leftarrow\text{TruncatedSVD}(M,k,\text{rtol}) for i=1 to \text{length}(s) do if s_{i}>0 then s_{i}^{-1}\leftarrow 1/s_{i} else s_{i}^{-1}\leftarrow 0 end if end for {Compute Moore-Penrose inverse} M^{+}\leftarrow V\cdot\text{diag}(s^{-1})\cdot U^{T} {Compute parameter update} \delta\theta\leftarrow-\eta\cdot M^{+}\cdot\mathcal{R} return \theta+\delta\theta Despite this structure, minimizing L(\theta) typically proceeds by applying gradient descent to the full loss, disregarding its decomposition (though in practice this is usually done over batches). A more principled approach, however, can explicitly account for the fact that L(\theta) aggregates several distinct conditions. The algorithm we propose, Sven, is given in Algorithm 1 and available on GitHub. 1 1 1 https://github.com/sambt/sven/ We now describe its derivation and its relation to existing methods. 2.1 Inspiration from Least-Squares Regression Although we will generalize to an arbitrary loss shortly, we begin by considering a regression problem, where we use the L_{2} -loss between our model f_{\theta}(x) and some target function g(x) . Such a loss is of the form L(\theta)=\sum_{\alpha\in\mathcal{\aleph}}{\big(\mathcal{R}^{\alpha}(\theta)\big)^{2}}=\sum_{\alpha\in\mathcal{\aleph}}\big(f_{\theta}(\alpha)-g(\alpha)\big)^{2}. (3) Consider the linear expansion of the residual \mathcal{R}^{\alpha}(\theta) around \theta_{0} : L(\theta_{0}+\delta\theta)=\sum_{\alpha\in\aleph}\left(\mathcal{R}^{\alpha}(\theta_{0})+\sum_{i}M^{\alpha}_{i}\,\delta\theta^{i}\right)^{2}+\mathcal{O}\left(|\delta\theta|^{2}\right), (4) where the Jacobian is M^{\alpha}_{\>\>i}\equiv\left.\frac{\partial\mathcal{R}^{\alpha}}{\partial\theta^{i}}\right|_{\theta=\theta_{0}}=\frac{\partial f_{\theta}(\alpha)}{\partial\theta^{i}}, (5) and we will omit the higher order terms from now on, taking them as understood. In such a situation, this can be viewed as a linear least squares regression problem for \delta\theta^{i} . Within the region that the linear expansion is reliable, we seek to solve for all contributions simultaneously. That is, we seek solutions to \mathcal{R}^{\alpha}(\theta_{0})+\sum_{i}M^{\alpha}_{\>\>i}\,\delta\theta^{i}=0, (6) such that the total loss is zero. Although a solution may not exist, the closest approximation to one (in a sense to be made precise shortly) is given by \delta\theta^{i}=-(M^{+})^{i}_{\>\alpha}\,\mathcal{R}^{\alpha}(\theta_{0}), (7) where

Read the complete text