Gradient flow in the kernel learning problem

Yang Li, Feng Ruan · 2025

Read Gradient flow in the kernel learning problem by Yang Li, Feng Ruan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-08550.txt.

Introduction

Yang Li and Feng Ruan continue their study of learning a kernel by following a gradient flow in a 2025 paper. Kernel methods turn data into similarity scores, and choosing the kernel is the problem. The authors identify a canonical Riemannian gradient flow for hunting the flow's stationary points.

With Gaussian noise in the picture, the flow gains a striking property: a whole continuous family of Lyapunov functions, which they read as automatic noise reduction during learning. A long postscript compares the picture with two-layer neural networks and explains the motivation in depth. The authors invite readers to start there. The paper is a theory sequel, refining the geometry of kernel learning rather than adding experiments.

Catalog record

CREATORYang Li, Feng Ruan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-08550

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Gradient flow in the kernel learning problem Yang Li, Feng Ruan Abstract This is a sequel to our paper ‘On the kernel learning problem’. We identify a canonical choice of Riemannian gradient flow, to find the stationary points in the kernel learning problem. In the presence of Gaussian noise variables, this flow enjoys the remarkable property of having a continuous family of Lyapunov functionals, and the interpretation is the automatic reduction of noise. PS. We include an extensive discussion in the postcript explaining the comparison with the 2-layer neural networks. Readers looking for additional motivations are encouraged to read the postscript immediately following the introduction. 1 Introduction 1.1 The kernel ridge regression problem Let V be a real d -dimensional vector space with an inner product |\cdot|_{V}^{2} , and let V^{*} denote its dual space. Given a function k_{V}:V^{*}\to\mathbb{R}^{+} with \int_{V^{*}}k_{V}(\omega)d\omega=1 , we can define a Hilbert space of functions, \mathcal{H}=\{f:V\to\mathbb{C}|\left\lVert f\right\rVert_{\mathcal{H}}^{2}=\int_{V^{*}}\frac{|\hat{f}|^{2}(\omega)}{k_{V}(\omega)}d\omega<+\infty\}, which embeds continuously into C^{0} , and in fact \left\lVert f\right\rVert_{C^{0}}\leq\left\lVert f\right\rVert_{\mathcal{H}} . Following [ 8 ] [ 2 ] , we say \mathcal{H} is a reproducing kernel Hilbert space (RKHS). Let X be a random variable taking value in V\simeq\mathbb{R}^{d} , and Y be a complex valued random variable with \mathbb{E}[|Y|^{2}]<+\infty . We consider the following minimization problem, which we will refer to as kernel ridge regression [ 15 ] . Let \lambda>0 be a small fixed parameter. We define the loss function I(f,U,\lambda)=\frac{1}{2}\mathbb{E}[|Y-f(UX)|^{2}]+\frac{\lambda}{2}\left\lVert f\right\rVert_{\mathcal{H}}^{2}, (1) depending on the parameter U\in\End(V) , and we seek the minimizer for J(U,\lambda)=\inf_{f\in\mathcal{H}}I(f,U,\lambda). The existence of a unique minimizer among all f\in\mathcal{H} follows from the Riesz representation theorem. By definition, the kernel ridge regression problem is a linear problem in infinite dimensions. The minimizer f_{U} for I(f,U,\lambda) can be viewed as the optimal fit for Y as a function of X , given the regularization effect imposed by the Hilbert norm term. By plugging in the zero test function, there is a ‘trivial upper bound’ J(U,\lambda)\leq\frac{1}{2}\mathbb{E}[|Y|^{2}] . 1.2 Kernel learning problem When k_{V} is a rotationally invariant function, namely k_{V}(\omega)=k(|\omega|_{V}^{2}) for some positive valued function k , the kernel ridge regression problem has the following equivalent formulation. Given any (non-degenerate) inner product \Sigma on V , the dual inner product 1 1 1 In index notation, if the inner product on V is represented by (g_{ij}) , then the dual inner product is represented by the inverse matrix (g^{ij}) . on V^{*} is denoted as \Sigma^{-1} . We then introduce a Hilbert space of functions \mathcal{H}_{\Sigma}=\{F:V\to\mathbb{C}|\left\lVert F\right\rVert_{\mathcal{H}_{\Sigma}}^{2}=\sqrt{\det\Sigma}\int_{V^{*}}\frac{|\hat{F}|^{2}}{k(|\omega|_{\Sigma^{-1}}^{2})}d\omega\}, and consider the following minimization problem with respect to F , \mathcal{I}(F,\Sigma,\lambda)=\frac{1}{2}\mathbb{E}[|Y-F(X)|^{2}]+\frac{\lambda}{2}\left\lVert F\right\rVert_{\mathcal{H}_{\Sigma}}^{2}, (2) \mathcal{J}(\Sigma;\lambda):=\min_{F\in\mathcal{H}_{\Sigma}}\frac{1}{2}\mathbb{E}[|Y-F(X)|^{2}]+\frac{\lambda}{2}\left\lVert F\right\rVert_{\mathcal{H}_{\Sigma}}^{2}. In this formulation, the functionals depend only on \Sigma , but not on the ambient inner product |\cdot|_{V}^{2} . Now for each U\in GL(V) , using the ambient inner product |\cdot|_{V}^{2} , we can define the inner product \Sigma on V and the dual inner product \Sigma^{-1} by the formula |x|_{\Sigma}^{2}=|Ux|^{2}_{V}=(x,U^{T}Ux)_{V},\quad|\omega|_{\Sigma^{-1}}=\max_{|x|_{\Sigma}=1}\langle\omega,x\rangle, where U^{T} denotes the adjoint operator of U . By a change of variables [ 15 , Lemma 4.1] , the two formulations are related by F=f\circ U,\quad I(f,U,\lambda)=\mathcal{I}(F,\Sigma,\lambda),\quad J(U,\lambda)=\mathcal{J}(\Sigma;\lambda). (3) The minimizer F_{\Sigma}=f_{U}\circ U gives the optimal fit for Y as a function of a suitable linear transform of X , given the regularization effect imposed by the Hilbert norm term. We denote Sym^{2}_{+} as the space of all non-degenerate inner products on V , which is an open submanifold inside the vector space Sym^{2}V^{*} . In the kernel learning problem , we view \mathcal{J}(\Sigma;\lambda) as a functional on Sym^{2}_{+} , and we seek the local minimizers with respect to \Sigma\in Sym^{2}_{+} , referred to as vacua in [ 15 ] . We emphasize that unlike the kernel ridge regression problem, the dependence of \mathcal{J} on \Sigma is nonlinear . Our motivation is that the vacua reflect two kinds of information: variable selection (which components of X does Y depend on essentially), and scale detection (what are the natural scale parameters in the distribution of X ). Instead of simply learning Y as a function of X in an a priori fixed Hilbert space as in the kernel ridge regression problem, kernel learning searches among a family of Hilbert space norms parametrized by Sym^{2}_{+} , and aims to find the landscape of vacua, and thereby find the most efficient Hilbert space norms adapted to the various regimes of the learning task. More substantial discussions can be found in the companion paper [ 15 ] . Remark 1 . In [ 15 ] , it was explained that there is a partial compactification of Sym^{2}_{+} , where the vacua(=local minimizer of \mathcal{J}(\Sigma;\lambda) ) naturally lives. Morally speaking, this reflects the fact that \Sigma can limit to a degenerate inner product, or it can diverge to infinity while remaining bounded on a subspace. The first situation is relevant for variable selection tasks, and the second situation is sometimes relevant if some marginal distribution of X contains discrete atoms. In this paper, for simplicity we only worry about the first possibility. The degenerate inner products are stratified according to the dimension of the null subspace. The open strata are \mathcal{S}_{l}=\{\Sigma\text{ is a positive semi-definite inner product on $V$ with rank $l$}\}, for l=0,1,\ldots d-1. Each \mathcal{S}_{l} is a smooth manifold. The space of positive semi-definite forms on V is Sym^{2}_{\geq 0}=Sym^{2}_{+}\cup\bigcup_{l\leq d-1}\mathcal{S}_{l}, which is a closed subset of Sym^{2}V^{*} . 1.3 The gradient flow This paper is motivated by the dynamical question of how to find the vacua of \mathcal{J}(\Sigma;\lambda) algorithmically, in terms of a gradient flow on the Riemannian manifold Sym^{2}_{+} . We emphasize that the first variation of \mathcal{J} is naturally a 1-form on Sym^{2}_{+} , and to define the gradient vector field , one has to choose some Riemannian metric on Sym_{+}^{2} ; this choice is part of the design of the gradient flow. The readers interested in the relevance of this paper to 2-layer neural networks, are encouraged to refer to the postscript for some extra motivations and an informal survey of results. The main contribution of this paper is to identify a canonical choice of Riemannian metric designed so that when the random variable X contains many noise Gaussian variable components, the Riemannian gradient flow turns out to possess a continuous family of Lyapunov functionals , whose monotonicity reflects the decrease of the influence of the noise. The definition of our gradient flow only requires the input of the covariance matrix of X , involves no further additional choice, and is manifestly coordinate free. In the rest of the paper, we assume X is a random variable valued in the vector space V\simeq\mathbb{R}^{d} , and Y is a random variable valued in the complex numbers, with \mathbb{E}[|X|^{2}]<+\infty , and \mathbb{E}[|Y|^{2}]<+\infty . Thus we can define the covariance matrix of X , which in coordinate free language is an inner product on V^{*} , {\rm Cov}(X)(\omega,\omega)=\mathbb{E}[\langle\omega,X-\mathbb{E}[X]\rangle^{2}]. The dual inner product on V is the inverse covariance matrix, which canonically gives an inner product on V , and we shall identify it with |\cdot|_{V}^{2} . In index notation, {\rm Cov}(X) is represented by the matrix C^{ij}=\mathbb{E}[(X^{i}-\mathbb{E}X^{i})(X^{j}-\mathbb{E}X^{j})], and the inverse covariance matrix is represented by the inverse matrix C_{ij} . We start with some basic geometry on

Read the complete text