The Riemannian Geometry associated to Gradient Flows of Linear Convolutional Networks El Mehdi Achour achour@mathc.rwth-aachen.de, Kathlén Kohn kathlen@kth.se, Holger Rauhut rauhut@lmu.de Abstract We study geometric properties of the gradient flow for learning deep linear convolutional networks. For linear fully connected networks, it has been shown recently that the corresponding gradient flow on parameter space can be written as a Riemannian gradient flow on function space (i.e., on the product of weight matrices) if the initialization satisfies a so-called balancedness condition. We establish that the gradient flow on parameter space for learning linear convolutional networks can be written as a Riemannian gradient flow on function space regardless of the initialization. This result holds for D D -dimensional convolutions with D ≥ 2 D\geq 2 , and for D = 1 D=1 it holds if all so-called strides of the convolutions are greater than one. The corresponding Riemannian metric depends on the initialization. keywords Deep learning theory, neural tangent kernel, gradient flows, convolutional networks, Riemaniann geometry, neuromanifolds, algebraic invariants, singularities, projective geometry, polynomial factorization 1 Introduction Convergence properties of (stochastic) gradient descent schemes for learning deep neural networks are challenging to understand mainly due to the nonconvexity of the corresponding loss functions. In recent years progress could be achieved in specific simplified settings Arora et al. 2018 ; Jacot et al. 2018 ; Bah et al. 2022 ; Kohn et al. 2022 . One line of work considers deep linear neural networks Arora et al. 2018 ; Bah et al. 2022 ; Kohn et al. 2022 ; Kohn et al. 2024 ; Nguegnang et al. 2024 . While linear neural networks represent linear functions so that their expressivity is somewhat limited, convergence properties of training algorithms are still highly non-trivial and therefore interesting to investigate. Such investigations should be seen as a first step before passing to networks with nonlinear activation function. For fully connected linear networks and the square loss it was shown in Bah et al. 2022 that (essentially) the corresponding gradient flow converges to a global minimizer for almost all initializations, and this result was extended to gradient descent under a suitable upper bound on the step sizes Nguegnang et al. 2024 . Additionally, Bah et al. 2022 revealed that under so-called balanced initialization, the flow of the neural network, i.e., the product of the weight matrices, is independent of its parametrization and follows a Riemannian gradient flow with respect to a suitable Riemannian metric. Expressed differently, the so-called neural tangent kernel (determining the Riemannian metric) is independent of the parameterization. This is a weaker property of the neural tangent kernel than the well-known one that in the limit of infinite width the neural tangent kernel becomes constant Jacot et al. 2018 (for random initialization) – of course, it is then in particular also independent of the parameterization. But in contrast, the mentioned result of Arora et al. 2018 ; Bah et al. 2022 holds for any finite width of the layers. In this paper, we study similar geometric and convergence properties for linear convolutional neural networks, i.e., we pass from the fully connected case to certain structured neural networks. More precisely, we study the neural tangent kernel’s sole dependence on the network as a function (instead of its parameters). Each layer in such a network is a convolution on D -dimensional signals that is given by its filter w_{l} , which is a D -dimensional tensor, and its stride s_{l}\in\mathbb{Z}_{>0}^{D} . Given a D -dimensional input tensor, the convolution computes the inner product of various parts of the input tensor with the filter w_{l} , by moving the filter through the input tensor with step size s_{i,d} in the d -th direction. For a formal definition, see Sections 2 and 3 . For a linear convolutional network with H layers, the following algebraic scalars stay constant throughout gradient flow ( Kohn et al. 2022 , Prop. 5.13) : \displaystyle\delta_{l}:=\|w_{l+1}\|_{F}^{2}-\|w_{l}\|_{F}^{2},\quad\text{ for all }l=1,\ldots,H-1. (1) If all \delta_{1},\ldots,\delta_{H-1} are zero, we say that the filters (w_{1},\ldots,w_{H}) are balanced in analogy to the fully connected case Arora et al. 2018 ; Bah et al. 2022 . One of our main findings is that in most cases the corresponding neural tangent kernel and thereby the associated Riemannian metric is independent of the parameterization of the convolutional network, regardless whether the initialization is balanced or not. The neural tangent kernel only depends on the \delta_{l} (at initialization). 1.1 Learning fully connected and convolutional linear networks via gradient flow For a fixed neural network architecture, we denote by \mu:\Theta\to\mathcal{M} the network parametrization map that assigns to parameters \theta\in\Theta the end-to-end function \mu_{\theta}:=\mu(\theta) . The image of \mu is the neuromanifold \mathcal{M} of the network architecture. We consider both fully connected and convolutional linear neural networks: • Fully connected linear neural networks take the form \mu_{\theta}:\mathbb{R}^{d_{0}}\to\mathbb{R}^{d_{H}},\mu_{\theta}(X)=W_{N}\cdots W_{1}X, where W_{l}\in\mathbb{R}^{d_{l}\times d_{l-1}} and \theta=(W_{1},\ldots,W_{H}) , i.e, \mu_{\theta}=W_{N}\cdots W_{1} . Setting r=\min\{d_{0},\ldots,d_{H}\} , the neuromanifold \mathcal{M} is the manifold of d_{H}\times d_{0} matrices of rank at most r . • A convolutional linear neural network is a composition of linear convolutions. On one-dimensional signals, the convolution in the l -th layer is given by a filter w_{l}\in\mathbb{R}^{k_{l}} and a stride s_{l}\in\mathbb{Z}_{>0} , l=1,\ldots,H . It is a linear map \alpha_{w_{l},s_{l}}:\mathbb{R}^{d_{l-1}}\to\mathbb{R}^{d_{l}} with output dimension d_{l} and input dimension d_{l-1}=s_{l}(d_{l}-1)+k_{l} that sends a vector X\in\mathbb{R}^{d_{l-1}} to the vector with i -th entry \displaystyle\left(\alpha_{w_{l},s_{l}}(X)\right)_{i}=\sum_{j=0}^{k_{l}-1}w_{l,j}X_{is_{l}+j},\quad\text{ for }i=0,1,\ldots,d_{l}-1. (2) The network composes H such convolutions, i.e., f=\alpha_{w_{H},s_{H}}\circ\cdots\circ\alpha_{w_{1},s_{1}} , which is again a convolution of stride s_{1}\cdots s_{H} and filter size k:=k_{1}+\sum_{l=2}^{H}(k_{l}-1)\prod_{i=1}^{l-1}s_{i} . Hence, the network parametrization map \displaystyle\mu:\mathbb{R}^{k_{1}}\times\ldots\times\mathbb{R}^{k_{H}}\to\mathbb{R}^{k} assigns to a filter tuple (w_{1},\ldots,w_{H}) the filter of the end-to-end convolution. Convolutions and linear convolutional neural networks on D -dimensional signals are defined analogously; see ( 14 ) below. Given training data \mathcal{D}=\{(X_{i},Y_{i})\in\mathbb{R}^{d_{0}}\times\mathbb{R}^{d_{H}}\mid i=1,2,\ldots,N\} , one aims at learning a hypothesis function \alpha:\mathbb{R}^{d_{0}}\to\mathbb{R}^{d_{H}} (e.g. a neural network) such that \alpha(X_{i})\approx Y_{i} for i=1,\ldots,N . Given a (differentiable) loss function \ell:\mathbb{R}^{d_{H}}\times\mathbb{R}^{d_{H}}\to\mathbb{R} , the empirical risk of \alpha is defined as \ell_{\mathcal{D}}(\alpha)=\sum_{i=1}^{N}\ell(\alpha(X_{i}),Y_{i}). Below we will exclusively work with the square loss \ell(\alpha(X_{i}),Y_{i})=\|\alpha(X_{i})-Y_{i}\|_{2}^{2}. The empirical square loss in parameter space is then defined as \mathcal{L}_{\mathcal{D}}(\theta):=\ell_{\mathcal{D}}\circ\mu(\theta)=\sum_{i=1}^{N}\|\mu_{\theta}(X_{i})-Y_{i}\|_{2}^{2}. Learning via empirical risk minimization requires minimization of \mathcal{L}_{\mathcal{D}} . In practice, one often uses (variants of) gradient descent or stochastic gradient descent. For a mathematical analysis it is convenient to view gradient descent as an Euler discretization of gradient flow, which we will consider here exclusively. Introducing a time dependence \theta(t),t\in[0,\infty) and given an initialization \theta_{0} , the gradient flow is the solution to the ODE \frac{d}{dt}\theta(t)=-\nabla\mathcal{L}_{\mathcal{D}}(\theta(t)),\quad\theta(0)=\theta_{0}. (3) The main interest of this paper is to study geometric properties of this gradient flow as well as convergence properties, in particular, whether the flow converges to a minimum of \mathcal{L}_{\mathcal{D}} . The neural tangent kernel introduced in Jacot et al. 2018 is a useful tool to this end. Denoting by J_{\theta} the Jacobi matrix of \mu at \theta , the evolution in function space, i.e., the evolution of \mu(\theta(t)) can be written as \frac{d}{dt}\mu(\theta(t))=-J_{\theta(t)}J_{\theta(t)}^{\top}\nabla\ell_{\mathcal{D}}(\mu(\theta(t))), where \nabla\ell(\alpha) denotes the (functional) gradient of \ell_{\mathcal{D}} with respect to \alpha . The above equation motivates to introduce the neural tangent kernel as K_{\theta}:=J_{\theta}J_{\theta}^{\top}. In case of fully connected linear networks where \mu(\theta)=W_{N}\cdots W_{1} , the gradient flow equation reads \frac{d}{dt}W_{l}(t)=-\nabla_{W_{j}}\mathcal{L}_{\mathcal{D}}(W_{1}(t),\ldots,W_{N}(t)),\quad l=1,\ldots,H. (4) It is shown in Arora et al. 2018 that \displaystyle\Delta_{i}:=\Delta_{i}(t)=W_{i+1}^{\top}(t)W_{i+1}(t)-W_{i}(t)W_{i}^{\top}(t),\quad\text{ for all }i=1,\ldots,H-1, (5) is constant in t , i.e., \Delta_{i}(t)=\Delta_{i}(0) for all t\geq 0 and all i=1,\ldots,H-1 . We refer to Marcotte et al. 2024 for more information on such invariants. In function space the corresponding evolution is given by \mu(\theta(t))=W(t)=W_{H}(t)\cdots W_{1}(t). For balanced initialization, i.e., \Delta_{i}(0)=0 for all i=1,\ldots,H-1 , the evolution of W(t) only depends on W(t) itself but not on the individual factors W_{l}(t) . More precisely, it is shown in Arora et al. 2018 that in this case W(t) satisfies the ODE \frac{d}{dt}W(t)=-\sum_{j=1}^{H}(W(t)W(t)^{T})^{\frac{H-j}{H}}\nabla\mathcal{L}_{\mathcal{D}}(W(t))(W(t)^{T}W(t))^{\frac{j-1}{H}}=:-\mathcal{A}_{W}(t)\big(\nabla\mathcal{L}_{\mathcal{D}}(W(t))\big). (6) It is shown in Bah et al. 2022 that the restriction \bar{\mathcal{A}}_{W} of \mathcal{A}_{W} to the tangent space T_{W}(\mathcal{M}_{r}) of the manifold \mathcal{M}_{r} of d_{H}\times d_{0} matrices of rank exactly r is invertible. The bilinear form g_{W}(Y,Z)=\langle\bar{\mathcal{A}}_{W}^{-1}Y,Z\rangle,\quad Y,Z\in T_{W}(\mathcal{M}_{r}) generates a Riemannian metric of class C^{1} on \mathcal{M}_{r} ( Bah et al. 2022 , Propositions 10 and 11) . Moreover, with the associated Riemannian gradient \nabla^{g} we can write the evolution ( 6 ) of W(t) (in case of balanced initialization, i.e., \Delta_{i}(0)=0 , i=1,\ldots,H-1 ) as Riemannian gradient flow (see also Bah et al. 2022 ), \frac{d}{dt}W(t)=-\nabla^{g}\mathcal{L}_{\mathcal{D}}(W(t)). The neural tangent kernel at \theta=(W_{1},\ldots,W_{H}) (not necessarily balanced) is given by its action on Z\in\mathbb{R}^{d_{H}\times d_{H}} via (see, e.g., ( Bah et al. 2022 , Lemma 2) K_{\theta}(Z)=\sum_{l=1}^{H}W_{H}\cdots W_{l+1}W_{l+1}^{\top}\cdots W_{H}^{\top}\,Z\,W_{1}^{\top}\cdots W_{l-1}^{\top}W_{l-1}\cdots W_{1}. (7) If \theta=(W_{1},\ldots,W_{H}) is balanced, i.e., \Delta_{i}=0 , i=1,\ldots,H-1 , and W=W_{H}\cdots W_{1} then K_{\theta}(Z)=\mathcal{A}_{W}(Z), in particular, the neural tangent kernel is independent of the parameterization of W=\mu(\theta) , i.e., independent of the individual factors W_{l} in the product W=W_{H}\cdots W_{1} . Moreover, the Riemannian metric g_{W} on \mathcal{M}_{k} is given by the quadratic form induced by K_{\theta}^{-1} (where the inverse is taken for the restriction on the tangent space T_{W}(\mathcal{M}_{k}) ). It is shown in Bah et al. 2022 that for almost all initialization the gradient flow ( 4 ) converges to a global minimum of \ell_{\mathcal{D}}(\alpha) restricted to the manifold \mathcal{M}_{k} of rank k -matrices where k\leq r:=\min\{d_{l},l=0,\ldots,H\} . For k=r , this corresponds to the global minimum of \mathcal{L}_{\mathcal{D}}(\theta) , but it is currently open whether k=r holds for almost all initializations. For linear convolution networks, where \theta=(w_{1},\ldots,w_{H}) and \mu(\theta) is the corresponding end-to-end convolutional filter (taking into account the strides s_{1},\ldots,s_{H} ), we will study the parameter-independence of neural tangent kernels; see Section 1.2 . Convergence of the corresponding gradient flow for learning linear convolutional neural networks on one-dimensional signals, i.e., D=1 , is studied in Diederen et al. 2025 . It is shown that for certain loss functions, including the square loss, the gradient flow ( 3 ) converges to a critical point of \mathcal{L}_{\mathcal{D}} . 1.2 Main Contributions We prove the following: • For linear convolutional networks on D -dimensional signals where D\geq 2 : For any fixed \delta=(\delta_{1},\ldots,\delta_{H-1})\in\mathbb{R}^{H-1} and any end-to-end function v in the smooth locus of the neuromanifold \mathcal{M} , there are finitely many parameters \theta=(w_{1},\ldots,w_{H}) in \mu^{-1}(v) that satisfy ( 1 ); they all have the same neural tangent kernel K_{\theta} . Hence, the neural tangent kernel only depends on the end-to-end function v and the invariants \delta . – For every fixed \delta , the neural tangent kernels are a Riemannian metric on the smooth locus of \mathcal{M} . – For fixed \delta , the neural tangent kernel is generally parameter dependent on the singular locus of \mathcal{M} . • For linear convolutional networks on one-dimensional signals where the strides s_{1},\ldots,s_{H-1}\in\mathbb{Z}_{>0} are larger than one: All results from the previous bullet points still hold. – It is known that for enough and sufficiently generic training data, every critical point \theta of the squared-error loss is either the zero function \mu(\theta)=0 , or \theta is a regular point of \mu and its image \mu(\theta) is in the smooth locus of \mathcal{M} ( Kohn et al. 2024 , Theorem 2.12) . We show that gradient flow with generic initialization does not converge to the zero function. Note that the zero function corresponds to a saddle point of \mathcal{L}_{\mathcal{D}} (unless Y=0 ). As noted above, it has been shown in Diederen et al. 2025 that (for convolutional networks on one-dimensional signals) gradient flow always converges to a critical point. This means that altogether we can say that we have a metric on the smooth part of \mathcal{M} with convergence to a point in that smooth part, and that this point is a critical point of the loss in both parameter and function space. • For linear convolutional networks on one-dimensional signals where some strides are one: For fixed \delta , the neural tangent kernel is generally parameter dependent. • For fully-connected linear networks, the neural tangent kernel is generally parameter dependent. While this may intuitively be clear given the available results, this has not been rigorously shown before. In this paper, we therefore provide formal proofs of the following facts. – For generic \Delta=(\Delta_{1},\ldots,\Delta_{H-1}) , the neural tangent kernel is generally parameter dependent. – In the balanced case (i.e., all \Delta_{i}=0 ), we explain the underlying geometry for why the neural tangent kernel only depends on the function V at hand: The balanced parameters in \mu^{-1}(V) are related by orthogonal matrices, which are isometries of the ambient Euclidean parameter space. We summarize the key difference between the studied architectures in Table 1 . 1-dim. higher-dim. fully-connected convolutions convolutions Is NTK parameter yes, yes, for all \delta , yes, for all \delta independent for when balanced when all strides >1 and all strides fixed \Delta resp. \delta ? (i.e., \Delta=0 ) Table 1: Linear network architectures studied in this article. Without the assumptions after “when”, the answer to the question becomes “generally no”. Remark 1 . For networks with algebraic activation function, the network parametrization map \mu:\Theta\to\mathcal{M} is algebraic, and so is the map that assigns to parameters \theta\in\Theta the neural tangent kernel K_{\theta} . Therefore, the equality of neural tangent kernels K_{\theta_{1}}=K_{\theta_{2}} is a polynomial condition in (\theta_{1},\theta_{2}) . On irreducible algebraic sets, polynomial conditions hold either everywhere or almost nowhere. Since the invariants ( 5 ) and ( 1 ) are polynomial, the set of parameter pairs (\theta_{1},\theta_{2}) that satisfy the same invariants and give rise to the same function \mu(\theta_{i}) is an algebraic set. Hence, to prove the bullet items above where “the neural tangent kernel is generally parameter dependent”, it is sufficient to exhibit one counterexample. 2 Convolutions on One-Dimensional Signals In this section, we focus on linear networks where each layer is a convolution on one-dimensional signals, see ( 2 ). The composition of such convolutions is equivalent to multiplying sparse polynomials as follows. For any positive integer t and k , we denote by \mathbb{R}[x^{t},y^{t}]_{k-1} the vector space of polynomials that are homogeneous of degree k-1 in the variables (x^{t},y^{t}) . This vector space is isomorphic to \mathbb{R}^{k} via the map \displaystyle\pi_{t}:\mathbb{R}^{k}\to\mathbb{R}[x^{t},y^{t}]_{k-1},v\mapsto v_{0}x^{t(k-1)}+v_{1}x^{t(k-2)}y^{t}+\ldots+v_{k-1}y^{t(k-1)}. The important insight is the following equality: \displaystyle\pi_{1}(\mu(w_{1},\ldots,w_{H}))=\pi_{t_{H}}(w_{H})\cdots\pi_{t_{1}}(w_{1}),\quad\text{ where }t_{l}=s_{1}\cdots s_{l-1}. (8) This allows us to view the network parametrization map \mu as multiplying sparse polynomials. We also see directly from ( 8 ) that layers with k_{l}=1 are irrelevant (i.e., such a layer can be omitted) and that the last stride s_{H} does not affect the map \mu . Therefore, we assume from now on that k_{l}>1 for all l=1,\ldots,H and that s_{H}=1 . The neuromanifold \mathcal{M}:=\mathrm{im}(\mu) is a Euclidean closed, semi-algebraic subset of \mathbb{R}^{k} of dimension \sum_{l=1}^{H}(k_{l}-1)+1 ( Kohn et al. 2024 , Theorem 2.4) . If H>1 , it has singular points. This singular locus consists of 0 and the union of all neuromanifolds that are strictly contained in \mathcal{M} and come from architectures with the same strides s_{1},\ldots,s_{H} ( Kohn et al. 2024 , Theorem 2.9) . This stratification is analogous to the well-known fact that the neuromanifold of a linear fully-connected network consists of all matrices whose rank is upper bounded by some r , and that its singular locus (if r is not the maximal rank) is precisely the matrices of rank at most r-1 . We denote the smooth locus of \mathcal{M} as \mathrm{Reg}(\mathcal{M}) , which is the set of points in \mathcal{M} , which are not singular. Due to the scaling ambiguity of the filters \mu(\lambda_{1}w_{1},\ldots,\lambda_{H}w_{H})=\lambda_{1}\cdots\lambda_{H}\mu(w_{1},\ldots,w_{H}) (for \lambda_{l}\in\mathbb{R} ), we can projectify the map \mu : \displaystyle\mu_{\mathbb{P}}:\mathbb{P}^{k_{1}-1}\times\ldots\times\mathbb{P}^{k_{H}-1}\to\mathbb{P}^{k-1},\quad([w_{1}],\ldots,[w_{H}])\mapsto[\mu(w_{1},\ldots,w_{H})]. Here, we write [v] for the element in projective space \mathbb{P}^{k-1} corresponding to a nonzero vector v\in\mathbb{R}^{k} . The image of the map \mu_{\mathbb{P}} is the projective neuromanifold \mathbb{P}\mathcal{M} . We show now that knowing the invariants in ( 1 ) fixes the scaling ambiguity up to signs. Proposition 2 . Let \delta_{1},\ldots,\delta_{H-1}\in\mathbb{R} , and consider non-zero filters w_{1}\in\mathbb{R}^{k_{1}} , …, w_{H}\in\mathbb{R}^{k_{H}} . Then, there are 2^{H-1} scalar tuples (\lambda_{1},\ldots,\lambda_{H})\in\mathbb{R}^{H} such that \lambda_{1}\cdots\lambda_{H}=1 and \displaystyle\delta_{i}=\|\lambda_{i+1}w_{i+1}\|^{2}-\|\lambda_{i}w_{i}\|^{2},\quad\text{ for all }i=1,\ldots,H-1. These scalar tuples are all equal up to sign changes of the components. Proof. This proof is inspired by the proof of ( Kohn et al. 2022 , Corollary 5.14) . Define C:=\prod_{l=1}^{H}\|w_{l}\|^{2} and \beta_{l}:=\|\lambda_{l}w_{l}\|^{2}=\lambda_{l}^{2}\|w_{l}\|^{2} for all l=1,\ldots,H . Then we have reformulated our problem in that we want to show that the conditions \displaystyle\prod_{l=1}^{H}\beta_{l}=C\text{ and }\beta_{i+1}-\beta_{i}=\delta_{i}\text{ for }i=1,\ldots,H-1 (9) have only one solution \beta_{1},\ldots,\beta_{H} where all \beta_{l} are positive. All \beta_{l} can be computed from \beta_{1} via \beta_{l}=\beta_{1}+\sum_{i=1}^{l-1}\delta_{i} . Hence, the conditions ( 9 ) can be rewritten as a single equation \displaystyle\beta_{1}\cdot(\beta_{1}+\delta_{1})\cdots(\beta_{1}+\delta_{1}+\ldots+\delta_{H-1})=C. (10) Now we choose a sufficiently large n_{1}>0 such that n_{l}:=n_{1}+\sum_{i=1}^{l-1}\delta_{i}\geq 0 for all l=2,\ldots,H . We define the new unknown x:=\beta_{1}-n_{1} such that the equality ( 10 ) becomes \displaystyle(x+n_{1})(x+n_{2})\cdots(x+n_{H})=C. (11) Defining n:=\min\{n_{1},\ldots,n_{H}\}\geq 0 , we observe that the left-hand side of ( 11 ) is positive and monotonically increasing in the range (-n,+\infty) . Since the left-hand side is zero for x=-n and converges to +\infty for x\to\infty and C>0 , the intermediate value theorem tells us that there is precisely one x in the range (-n,+\infty) that yields the equality in ( 11 ). By definition of n , this is the only solution where all \beta_{l}=x+n_{l} are positive. ∎ The last ingredient we need is the following standard fact: Observation 3 . Since \mu is linear in each of the w_{i} , its derivative at a filter tuple \theta=(w_{1},\ldots,w_{H}) is the linear map \displaystyle d_{\theta}\mu:\dot{\theta}=(\dot{w}_{1},\ldots,\dot{w}_{H})\mapsto\mu(\dot{w}_{1},w_{2},\ldots,w_{H})+\ldots+\mu(w_{1},\ldots,w_{H-1},\dot{w}_{H}). It is represented by the Jacobian matrix J=J_{\theta} . It has the block columns structure J=[J_{1}|J_{2}|\cdots|J_{H}] , where J_{i} corresponds to taking the derivative with respect to w_{i} ; in other words, J_{1}\dot{w}_{1}=\mu(\dot{w}_{1},w_{2},\ldots,w_{H}),\ldots,J_{H}\dot{w}_{H}=\mu(w_{1},\ldots,w_{H-1},\dot{w}_{H}). From the latter expressions, we see that the entries of J_{i} are linear in the entries of w_{j} for each j\neq i . We then conclude that K_{\theta}=JJ^{\top}=J_{1}J_{1}^{\top}+\ldots+J_{H}J_{H}^{\top} , where the entries of each term K_{i}:=J_{i}J_{i}^{\top} are homogeneous of degree 2 in the entries of w_{j} for each j\neq i . 2.1 All strides larger than one We now show that the neural tangent kernel only depends on the end-to-end filter v\in\mathcal{M} and the invariants \delta_{1},\ldots,\delta_{H-1} , when the strides in the layers – up to possibly the last one – are larger than one. Theorem 4 . Let the strides s_{1},\ldots,s_{H-1} be larger than one. For any fixed \delta=(\delta_{1},\ldots,\delta_{H-1})\in\mathbb{R}^{H-1} and any end-to-end function v in the smooth locus of the neuromanifold \mathcal{M} , there are 2^{H-1} many parameters \theta=(w_{1},\ldots,w_{H}) in \mu^{-1}(v) that satisfy ( 1 ); they all have the same neural tangent kernel K_{\theta} . Proof. Under the assumption that s_{l}>1 for all l=1,\ldots,H-1 , every point [v] in the smooth locus of \mathbb{P}\mathcal{M} has a unique point in its fiber \mu_{\mathbb{P}}^{-1}([v]) ( Kohn et al. 2024 , Section 5) . In other words, for every end-to-end filter v in the smooth part of \mathcal{M} , there is a unique filter tuple, up to scaling, that parametrizes v . Together with Proposition 2 , this shows that, for fixed \delta_{1},\ldots,\delta_{H-1} , every end-to-end filter v in the smooth locus of the neuromanifold \mathcal{M} has a unique filter tuple \theta=(w_{1},\ldots,w_{H}) , up to signs, such that \mu(\theta)=v and the invariants in ( 1 ) are satisfied. The signs do not affect the neural tangent kernel K_{\theta} since, due to Observation 3 , K_{\theta} is the sum of matrices K_{i} that are homogeneous of degree two in the filters in \theta . ∎ Hence, for any fixed choice of invariants \delta=(\delta_{1},\ldots,\delta_{H-1}) , the neural tangent kernel only depends on the smooth point v\in\mathcal{M} and we denote it by K^{(\delta)}(v) . Example 5 . The following network architecture will serve as a running example. We consider a network with two layers whose filters have sizes k_{1}=3,k_{2}=2 and with strides s_{1}=2,s_{2}=1 . For ease of notation we denote the first filter by a=(a_{0},a_{1},a_{2})\in\mathbb{R}^{3} and the second filter by b=(b_{0},b_{1})\in\mathbb{R}^{2} . The filter of the end-to-end convolution is \displaystyle\mu(a,b)=(a_{0}b_{0},a_{1}b_{0},a_{2}b_{0}+a_{0}b_{1},a_{1}b_{1},a_{2}b_{1}). According to ( Kohn et al. 2022 , Example 4.12) , the neuromanifold is \displaystyle\mathcal{M}=\{v\in\mathbb{R}^{5}\mid v_{0}v_{3}^{2}+v_{1}^{2}v_{4}-v_{1}v_{2}v_{3}=0,\,v_{2}^{2}-4v_{0}v_{4}\geq 0\}. Thus, its Zariski closure is the hypersurface in \mathbb{R}^{5} defined by v_{0}v_{3}^{2}+v_{1}^{2}v_{4}-v_{1}v_{2}v_{3}=0 , that is singular at all points of the form v=(v_{0},0,v_{2},0,v_{4}) . Now, we consider a smooth point v\in\mathcal{M} (i.e., v_{1}\neq 0 or v_{3}\neq 0 ). As in the proof of Theorem 4 , there is – up to scaling – only one choice of filters (a,b) such that \mu(a,b)=v . If both v_{1}\neq 0 and v_{3}\neq 0 , then this choice is \displaystyle a=(\frac{v_{0}}{v_{1}},1,\frac{v_{4}}{v_{3}})\quad\text{ and }\quad b=(v_{1},v_{3}). (12) If precisely one of v_{1} and v_{3} is zero, then the preimage filters are a=(v_{2},v_{3},v_{4}) and b=(0,1) for v_{1}=0 , or a=(v_{0},v_{1},v_{2}) and b=(1,0) for v_{3}=0 . Next, for a fixed \delta\in\mathbb{R} , we compute the possible scalars \lambda such that \delta=\|\frac{1}{\lambda}b\|^{2}-\|\lambda a\|^{2} . We follow the proof of Proposition 2 . For 2-layer networks, we see that ( 10 ) becomes a quadratic equation in \lambda^{2} : \displaystyle\lambda^{2}\|a\|^{2}\cdot(\lambda^{2}\|a\|^{2}+\delta)=\|a\|^{2}\|b\|^{2}. Its only positive solution is \lambda^{2}=\frac{-\delta+\sqrt{\delta^{2}+4\|a\|^{2}\|b\|^{2}}}{2\|a\|^{2}} . Theorem 4 tells us that we can compute K^{(\delta)}(v) as K_{(\lambda a,\frac{1}{\lambda}b)}=\frac{1}{\lambda^{2}}K_{1}+\lambda^{2}K_{2} , where \displaystyle\small K_{1}=\left(\begin{array}[]{c c c c c}b_{0}^{2}&0&b_{0}b_{1}&0&0\\ 0&b_{0}^{2}&0&b_{0}b_{1}&0\\ b_{0}b_{1}&0&b_{0}^{2}+b_{1}^{2}&0&b_{0}b_{1}\\ 0&b_{0}b_{1}&0&b_{1}^{2}&0\\ 0&0&b_{0}b_{1}&0&b_{1}^{2}\end{array}\right)\text{ and }K_{2}=\left(\begin{array}[]{c c c c c}a_{0}^{2}&a_{0}a_{1}&a_{0}a_{2}&0&0\\ a_{0}a_{1}&a_{1}^{2}&a_{1}a_{2}&0&0\\ a_{0}a_{2}&a_{1}a_{2}&a_{0}^{2}+a_{2}^{2}&a_{0}a_{1}&a_{0}a_{2}\\ 0&0&a_{0}a_{1}&a_{1}^{2}&a_{1}a_{2}\\ 0&0&a_{0}a_{2}&a_{1}a_{2}&a_{2}^{2}\end{array}\right). This only depends on v and \delta . For instance, if we assume that both v_{1}\neq 0 and v_{3}\neq 0 , then K_{1} and K_{2} can be expressed only in terms of v using the substitution ( 12 ). Similarly, \lambda^{2} can be written in terms of v and \delta . In the balanced case, this simplifies to \lambda^{2}=\frac{\|b\|}{\|a\|}=\sqrt{\frac{v_{1}^{2}+v_{3}^{2}}{\frac{v_{0}^{2}}{v_{1}^{2}}+1+\frac{v_{4}^{2}}{v_{3}^{2}}}} . \hfill\diamondsuit The neural tangent kernels K^{(\delta)}(v) are the inverse of the pushforward of the following Riemannian metric: Consider \displaystyle\Theta_{\delta}:=\{(w_{1},\ldots,w_{H})\in\mu^{-1}(\mathrm{Reg}(\mathcal{M}))\mid\delta_{i}=\|w_{i+1}\|^{2}-\|w_{i}\|^{2}\text{ for all }i=1,\ldots,H-1\}, where we recall that \mathrm{Reg}(\mathcal{M}) denotes the smooth locus of \mathcal{M} . This space \Theta_{\delta} is a smooth manifold of the same dimension as the neuromanifold \mathcal{M} . Its tangent space at any \theta=(w_{1},\ldots,w_{H})\in\Theta_{\delta} is \displaystyle T_{\theta}\Theta_{\delta}=\{(\dot{w}_{1},\ldots,\dot{w}_{H})\mid\langle w_{i+1},\dot{w}_{i+1}\rangle=\langle w_{1},\dot{w}_{1}\rangle\text{ for all }i=1,\ldots,H-1\}, (13) where \langle\cdot,\cdot\rangle denotes the standard Euclidean inner product. Lemma 6 . For every \theta\in\Theta_{\delta} , the derivative d_{\theta}\mu restricted to the tangent space T_{\theta}\Theta_{\delta} is a bijection onto T_{\mu(\theta)}\mathcal{M} . Proof. Since \mu(\theta) is a smooth point of \mathcal{M} , the derivative d_{\theta}\mu is surjective onto T_{\mu(\theta)}\mathcal{M} , and its kernel is spanned by \dot{\theta}_{1}:=(-w_{1},w_{2},0,\ldots,0),\ldots,\dot{\theta}_{H-1}:=(0,\ldots,0,-w_{H-1},w_{H}) , see Kohn et al. 2024 . Due to \dim(\Theta_{\delta})=\dim(\mathcal{M}) , it is sufficient to show that this kernel intersects the tangent space T_{\theta}\Theta_{\delta} only at 0 . For that, we consider a point \lambda_{1}\dot{\theta}_{1}+\ldots+\lambda_{H-1}\dot{\theta}_{H-1} that is contained in T_{\theta}\Theta_{\delta} . Our aim is to show that all \lambda_{i} must be equal to 0 . We begin by proving inductively that each \lambda_{i} is of the form \lambda_{1}\cdot c_{i} for some positive scalar c_{i} . This is immediate for i=1 . For i=2 , we plug in the point \sum_{j}\lambda_{j}\dot{\theta}_{j} into the first condition from ( 13 ). This yields the equality (\lambda_{1}-\lambda_{2})\|w_{2}\|^{2}=-\lambda_{1}\|w_{1}\|^{2} . Therefore, \lambda_{2}=\lambda_{1}\cdot c_{2} , where c_{2}:=\frac{\|w_{1}\|^{2}+\|w_{2}\|^{2}}{\|w_{2}\|^{2}}>0 . Similarly, for i>2 , the conditions in ( 13 ) yield (\lambda_{i-1}-\lambda_{i})\|w_{i}\|^{2}=-\lambda_{1}\|w_{1}\|^{2} , which can be rewritten as \lambda_{i}=\frac{1}{\|w_{i}\|^{2}}(\lambda_{i-1}\|w_{i}\|^{2}+\lambda_{1}\|w_{1}\|^{2}) . Applying the induction hypothesis, we obtain \lambda_{i}=\lambda_{1}\cdot c_{i} , where c_{i}:=\frac{1}{\|w_{i}\|^{2}}(c_{i-1}\|w_{i}\|^{2}+\|w_{1}\|^{2})>0 . On the other hand, the last condition in ( 13 ) gives us that \lambda_{H-1}\|w_{H}\|^{2}=-\lambda_{1}\|w_{1}\|^{2} . Thus, we have \lambda_{1}\cdot c_{H-1}=\lambda_{H-1}=-\lambda_{1}\cdot\frac{\|w_{1}\|^{2}}{\|w_{H}\|^{2}} , which shows that \lambda_{1} must be 0 , and so all \lambda_{i} are 0 . ∎ Corollary 7 . Let the strides s_{1},\ldots,s_{H-1} be larger than one. The network parametrization map \mu restricted to \Theta_{\delta} is a submersion onto \mathrm{Reg}(\mathcal{M}) with finite fibers. The pushforward of the Euclidean metric on \Theta_{\delta} under \mu is given by the bilinear forms (K^{(\delta)}(v))^{-1} for v\in\mathrm{Reg}(\mathcal{M}) . Proof. The first part of the claim follows immediately from Lemma 6 and Theorem 4 . More concretely, Lemma 6 states that the derivatives of \mu_{\delta}:=\mu|_{\Theta_{\delta}} are bijective. For such a submersion with finite fibers, the pushforward metric can be computed via averaging over the metrics on the fibers: \displaystyle\|\dot{v}\|_{\mu_{\delta}}^{2}:= \displaystyle\frac{1}{|\mu_{\delta}^{-1}(v)|}\sum_{\theta\in\mu_{\delta}^{-1}(v)}\|(d_{\theta}\mu_{\delta})^{-1}(\dot{v})\|^{2} \displaystyle= \displaystyle\frac{1}{|\mu_{\delta}^{-1}(v)|}\sum_{\theta\in\mu_{\delta}^{-1}(v)}((d_{\theta}\mu_{\delta})^{-1}(\dot{v}))^{\top}(d_{\theta}\mu_{\delta})^{-1}(\dot{v}) \displaystyle= \displaystyle\frac{1}{|\mu_{\delta}^{-1}(v)|}\sum_{\theta\in\mu_{\delta}^{-1}(v)}\dot{v}^{\top}(J_{\theta}J_{\theta}^{\top})^{-1}\dot{v}=\frac{1}{|\mu_{\delta}^{-1}(v)|}\sum_{\theta\in\mu_{\delta}^{-1}(v)}\dot{v}^{\top}K_{\theta}^{-1}\dot{v}, where \dot{v}\in T_{v}\mathcal{M} for some v\in\mathrm{Reg}(\mathcal{M}) . By Theorem 4 , all \theta\in\mu_{\delta}^{-1}(v) have the same neural tangent kernel K_{\theta}=K^{(\delta)}(v) , and so we conclude that the pushforward metric is \|\dot{v}\|_{\mu_{\delta}}^{2}=\dot{v}^{\top}(K^{(\delta)}(v))^{-1}\dot{v} . ∎ In particular, Corollary 7 shows that the K^{(\delta)}(v) , for any fixed choice of \delta , form a Riemannian metric on the smooth locus of the neuromanifold \mathcal{M} . This metric can generally not be extended on the singular locus of \mathcal{M} . It was shown in Kohn et al. 2024 that the singular locus of \mathcal{M} consists precisely of those end-to-end filters v that either have several filter tuples (even up to scaling by constants) parametrizing it or such that the unique filter tuple \theta (up to scaling) satisfies that the rank d_{\theta}\mu is less than \dim(\mathcal{M}) . In the latter case, the neural tangent kernel K_{\theta} drops rank. In the former case, the neural tangent kernels K_{\theta_{1}} and K_{\theta_{2}} for \mu(\theta_{1})=\mu(\theta_{2}) are generally distinct as the following example demonstrates. Example 8 . We consider the network architecture from Example 5 . Recall that the singular points of the neuromanifold \mathcal{M} are of the form v=(v_{0},0,v_{2},0,v_{4}) . They are parametrized by filters of the form a=(a_{0},0,a_{2}) and b=(b_{0},b_{1}) . If (a_{0},a_{2}) and (b_{0},b_{1}) are linearly independent, then the resulting end-to-end filter v has two distinct (up to scaling) parametrizations, namely v=\mu(a_{0},0,a_{2},b_{0},b_{1})=\mu(b_{0},0,b_{1},a_{0},a_{2}) . At both parameter tuples, the derivative of \mu is of maximal rank 4 , and so is the neural tangent kernel. However, in general, K_{(a_{0},0,a_{2},b_{0},b_{1})}\neq K_{(b_{0},0,b_{1},a_{0},a_{2})} , even for fixed \delta -invariant. One concrete balanced (i.e., \delta=0 ) example is (a_{0},a_{2})=(1,2) and (b_{0},b_{1})=(2,1) . In this case, \displaystyle\small K_{(a_{0},0,a_{2},b_{0},b_{1})}=\left(\begin{array}[]{c c c c c}5&0&4&0&0\\ 0&4&0&2&0\\ 4&0&10&0&4\\ 0&2&0&1&0\\ 0&0&4&0&5\end{array}\right)\text{ and }K_{(b_{0},0,b_{1},a_{0},a_{2})}=\left(\begin{array}[]{c c c c c}5&0&4&0&0\\ 0&1&0&2&0\\ 4&0&10&0&4\\ 0&2&0&4&0\\ 0&0&4&0&5\end{array}\right) If (a_{0},a_{2}) and (b_{0},b_{1}) are linearly dependent, they are the same up to scaling. This means that, up to scaling, v has only one preimage under \mu . That preimage (a_{0},0,a_{2},b_{0},b_{1}) is a critical point of the map \mu , i.e., the rank of the neural tangent kernel K_{(a_{0},0,a_{2},b_{0},b_{1})} is less than the generic rank 4 . \hfill\diamondsuit Moreover, for varying \delta , the metrics K^{(\delta)} on \mathrm{Reg}(\mathcal{M}) are generally distinct. Example 9 . We consider the same network architecture as in Example 5 . Suppose v\in\mathcal{M} is such that v_{0} and v_{1} are not equal to 0. Such a v is in particular a smooth point of the neuromanifold \mathcal{M} . We show that for different \delta we obtain different K^{(\delta)}(v) . We fix filters a and b such that \mu(a,b)=v . Our assumptions on v imply that a_{0}\neq 0 and a_{1}\neq 0 . By Example 5 , we have K^{(\delta)}(v)=K_{(\lambda a,\frac{1}{\lambda}b)} , where \lambda^{2}=\frac{-\delta+\sqrt{\delta^{2}+4\|a\|^{2}\|b\|^{2}}}{2\|a\|^{2}} . In particular, the entry of K^{(\delta)}(v) at position (2,1) is equal to \lambda^{2}a_{0}a_{1}\neq 0 . The function which to \delta\in\mathbb{R} associates \frac{-\delta+\sqrt{\delta^{2}+4\|a\|^{2}\|b\|^{2}}}{2\|a\|^{2}} is strictly decreasing and therefore bijective. Hence, distinct \delta ’s yield distinct values for \lambda^{2} and thus for the (2,1) -entry of K^{(\delta)}(v) . \hfill\diamondsuit 2.2 Convergence of gradient flow In this section, we focus on the squared-error loss. We still assume that the strides in the network layers are larger than one. We show that, for a sufficient amount of generic training data and generic initialization, gradient flow converges to a critical point \theta in parameter space such that \mu(\theta) is a critical point on the smooth locus of the neuromanifold \mathcal{M} . Given training data \mathcal{D}=\{(X_{i},Y_{i})\in\mathbb{R}^{d_{0}}\times\mathbb{R}^{d_{H}}\mid i=1,2,\ldots,N\} , the squared-error loss of an end-to-end convolution \alpha is \ell_{\mathcal{D}}(\alpha)=\sum_{i=1}^{N}\|\alpha(X_{i})-Y_{i}\|^{2} . We denote the squared-error loss in parameter space by \mathcal{L}_{\mathcal{D}}:=\ell_{\mathcal{D}}\circ\mu . Recall that we write k for the size of the end-to-end filters in \mathcal{M} . Theorem 10 . Let the strides s_{1},\ldots,s_{H-1} be larger than one, and let N\geq k . For almost all data \mathcal{D}\in(\mathbb{R}^{d_{0}}\times\mathbb{R}^{d_{H}})^{N} and almost all initializations, minimizing the squared-error loss \mathcal{L}_{\mathcal{D}} via gradient flow converges to a critical point \theta of \mathcal{L}_{\mathcal{D}} such that \mu(\theta) is a smooth point of \mathcal{M} and a critical point of \ell_{\mathcal{D}}|_{\mathrm{Reg}(\mathcal{M})} . Proof. It is proven in Diederen et al. 2025 that gradient flow converges to a critical point of \mathcal{L}_{\mathcal{D}} . Moreover, under the assumptions that the strides are larger than one, N\geq k , and sufficiently generic data, ( Kohn et al. 2024 , Theorem 2.12) shows that every critical point \theta of \mathcal{L}_{\mathcal{D}} either satisfies \mu(\theta)=0 or \mu(\theta) is a smooth point of \mathcal{M} that is a critical point of \ell_{\mathcal{D}} restricted to that smooth locus. Hence, it is enough to exclude \mu(\theta)=0 for generic initializations. We explain this separately in Proposition 12 . ∎ To prove Proposition 12 , we start by rewriting the squared-error loss in terms of filters instead of convolutions. Under the assumption N\geq k , it is shown in ( Kohn et al. 2024 , Corollary 7.2) that, for almost all input training data X_{1},\ldots,X_{N}\in\mathbb{R}^{d_{0}} , minimizing \ell_{\mathcal{D}}(\alpha) is equivalent to minimizing \displaystyle\ell_{A,u}(w):=(w-u)^{\top}A(w-u), where w is the filter of the convolution \alpha , u is a data filter that depends on the training data \mathcal{D} , and A is a symmetric positive-definite matrix that depends on the input training data X_{1},\ldots,X_{N} . We denote the corresponding loss in parameter space by \mathcal{L}_{A,u}:=\ell_{A,u}\circ\mu . When the data \mathcal{D} is generic, the vector Au is generic as well (see ( Kohn et al. 2024 , Section 7) ). We use this to prove the technical Lemma 11 , that is based on the following standard construction from algebraic geometry. Given a k -dimensional vector space V , its projectivization \mathbb{P}(V) is a (k-1) -dimensional projective space. The projectivization \mathbb{P}(V^{\ast}) of the dual vector space V^{\ast} consists of all the hyperplanes in \mathbb{P}(V) . Given a subvariety X\subseteq\mathbb{P}(V) , we say that a hyperplane H\subseteq\mathbb{P}(V) is tangent to X if there is a smooth point x\in X such that H contains the embedded tangent space T_{x}X\subseteq\mathbb{P}(V) . Thinking of the hyperplanes in \mathbb{P}(V) as points of the dual projective space \mathbb{P}(V^{\ast}) , the set of hyperplanes that are tangent to X is a subset of \mathbb{P}(V^{\ast}) . Its Zariski closure is a proper subvariety of \mathbb{P}(V^{\ast}) , called the dual variety of X [Chapter 1] Gelfand et al. 1994 . Lemma 11 . Let \theta=(w_{1},\ldots,w_{H}) with w_{i}\in\mathbb{R}^{k_{i}}\setminus\{0\} , and let N\geq k . For almost all N -tuples of data \mathcal{D} , the vector Au is not orthogonal to the image of the derivative d_{\theta}\mu . Proof. Let us first consider the case that \mu(\theta) is a smooth point of the neuromanifold \mathcal{M} . Then, the image of d_{\theta}\mu is the tangent space of \mathcal{M} at \mu(\theta) . A vector v=(v_{0},\ldots,v_{k-1}) is orthogonal to this tangent space if and only if the hyperplane P_{v}:=\{(h_{0},\ldots,h_{k-1})\mid\sum_{i}v_{i}h_{i}=0\} contains the tangent space. Due to the genericity of the data \mathcal{D} , the vector Au and the hyperplane P_{Au} are generic. Since the dual variety of the Zariski closure \mathbb{P}\overline{\mathcal{M}} of the projective neuromanifold \mathbb{P}\mathcal{M} is a proper subvariety of (\mathbb{P}^{k-1})^{\ast} , the generic hyperplane P_{Au} , considered as a point in the dual space (\mathbb{P}^{k-1})^{\ast} , is not contained in the dual variety of \mathbb{P}\overline{\mathcal{M}} . Hence, the hyperplane P_{Au} is not tangent to \mathbb{P}\overline{\mathcal{M}} , and so the vector Au is not orthogonal to T_{\mu(\theta)}\mathcal{M}=\mathrm{im}(d_{\theta}\mu) . Now, we consider arbitrary \theta where all layer filters are nonzero. We denote by r the rank of the derivative d_{\theta}\mu . Since d_{\theta}\mu(\theta)=H\cdot\mu(\theta)\neq 0 , we have that r\geq 1 . We further denote by X_{r} the set of all filter tuples \theta^{\prime} such that the rank of d_{\theta^{\prime}}\mu is r , and by Y_{r} the Zariski closure of \mu(X_{r}) . Note that Y_{\dim\mathcal{M}}=\overline{\mathcal{M}} . If \mu(\theta) is a smooth point of Y_{r} , then \mathrm{im}(d_{\theta}\mu) is the tangent space T_{\mu(\theta)}Y_{r} . Otherwise, since \mathrm{im}(d_{\theta}\mu) has dimension r , this linear space is contained in the Zariski closure of the set of all tangent spaces at smooth points of Y_{r} . Therefore, the dual variety of \mathbb{P}Y_{r} contains all hyperplanes that contain the r -dimensional \mathrm{im}(d_{\theta}\mu) . Since, as above, the hyperplane P_{Au} is a generic point in (\mathbb{P}^{k-1})^{\ast} , it is not contained in the dual variety of \mathbb{P}Y_{r} . Thus, it does not contain \mathrm{im}(d_{\theta}\mu) , and so Au is not orthogonal to \mathrm{im}(d_{\theta}\mu) . ∎ Proposition 12 . Let N\geq k . For almost all data \mathcal{D}\in(\mathbb{R}^{d_{0}}\times\mathbb{R}^{d_{H}})^{N} and almost all initializations, minimizing \mathcal{L}_{\mathcal{D}} via gradient flow converges to a point \theta with \mu(\theta)\neq 0 . Proof. For generic weight initialization, the \delta_{i} (for i=1,\ldots,H-1 ) are generic as well, and also the \sum_{i=k}^{l}\delta_{i} . Since the invariants ( 1 ) stay constant during gradient flow, at the point \theta=(w_{1},\ldots,w_{H}) of convergence, at most one of the filters w_{l} can be zero. Indeed, if two filters w_{k} and w_{l} are equal to zero then \sum_{i=k}^{l}\delta_{i} has to be zero, which is not the case because of genericity. If none of the filters is zero, we are done. Therefore, we assume now that precisely one of the filters, say w_{l} , is zero. We now consider the Hessian matrix \mathcal{H} at \theta of the loss function \mathcal{L}_{A,u} that is equivalent to the squared-error loss. The Hessian matrix consists of blocks, each corresponding to a pair of filters (w_{i},w_{j}) . We denote these blocks by \mathcal{H}_{(w_{i},w_{j})} . Those blocks encode bilinear functions \displaystyle d^{2}_{w_{i},w_{j}}\mathcal{L}_{A,u}:(\dot{w}_{i},\dot{w}_{j})\mapsto 2\left(\mu_{i}(\dot{w}_{i})^{\top}A\mu_{j}(\dot{w}_{j})-\mu_{i,j}(\dot{w}_{i},\dot{w}_{j})^{\top}Au\right), where \mu_{i}(\dot{w}_{i})=\mu(w_{1},\ldots,w_{i-1},\dot{w}_{i},w_{i+1},\ldots,w_{H}) , \mu_{j} is analogously defined, and similarly \mu_{i,j} substitutes both w_{i} and w_{j} by \dot{w}_{i} and \dot{w}_{j} (if i=j , then \mu_{i,i}=0 ). In the case that neither i nor j are equal to l , then our assumption w_{l}=0 implies that d^{2}_{w_{i},w_{j}}\mathcal{L}_{A,u}=0 . Thus, also the corresponding Hessian block \mathcal{H}_{(w_{i},w_{j})} is zero. Now we show that not all Hessian blocks with j=l and i\neq l are zero. Those blocks encode bilinear forms d^{2}_{w_{i},w_{l}}\mathcal{L}_{A,u}=-2\mu_{i,l}^{\top}Au . We assume for contradiction that all those forms are zero. We pick a nonzero filter c\in\mathbb{R}^{k_{l}} and consider the map \tilde{\mu}:(w^{\prime}_{1},\ldots,w^{\prime}_{l-1},w^{\prime}_{l+1},\ldots,w^{\prime}_{H})\mapsto\mu(w^{\prime}_{1},\ldots,w^{\prime}_{l-1},c,w^{\prime}_{l+1},\ldots,w^{\prime}_{H}) . We also write \tilde{\theta} for the filter tuple that is obtained from \theta by omitting w_{l} . Then, d_{\tilde{\theta}}\tilde{\mu}(\dot{w}_{1},\ldots,\dot{w}_{l-1},\dot{w}_{l+1},\ldots,\dot{w}_{H})^{\top}Au=\sum_{i\neq l}\mu_{i,l}(\dot{w}_{i},c)^{\top}Au=0 for all \dot{w}_{i}\in\mathbb{R}^{k_{i}} . This means that the vector Au is orthogonal to the image of the derivative d_{\tilde{\theta}}\tilde{\mu} . However, this contradicts Lemma 11 applied to the (H-1) -layer network parametrization map \tilde{\mu} . Therefore, one of the blocks \mathcal{H}_{(w_{i},w_{l})} with i\neq l has a non-zero entry. We denote the coordinates of that entry in the Hessian matrix \mathcal{H} by (s,t) (such that the index t corresponds to the filter w_{l} ). We consider the s -th and t -th standard basis vectors e_{s} and e_{t} of \mathbb{R}^{\sum_{j=1}^{H}k_{j}} , and the vector v=\alpha e_{s}+e_{t} for some scalar \alpha\in\mathbb{R} . We have \displaystyle v^{\top}\mathcal{H}v=\alpha^{2}\mathcal{H}_{ss}+2\alpha\mathcal{H}_{st}+\mathcal{H}_{tt}=2\alpha\mathcal{H}_{st}+\mathcal{H}_{tt}. Since \mathcal{H}_{st}\neq 0 , we can choose \alpha such that v^{\top}\mathcal{H}v<0 . Hence, \theta is a critical point for which the Hessian has at least one negative eigenvalue. This is called a strict saddle , and we know from Bah et al. 2022 that, for almost all initializations, gradient flow avoids strict saddles. Thus, \theta with one of the layer filters equal to zero could not have been the point of convergence. ∎ 2.3 Some strides equal to one A key fact that made Theorem 4 and its proof possible is that a generic point v in the neuromanifold has a unique point (up to scaling) in its fiber \mu^{-1}(v) . In other words, the generic non-empty fiber of the projective network parametrization map \mu_{\mathbb{P}} is a singleton. An algebraic map with that property is called birational ; it means that the map is bijective almost everywhere. When some of the strides s_{1},\ldots,s_{H-1} are equal to one, the projective network parametrization map \mu_{\mathbb{P}} is no longer birational Kohn et al. 2024 . This means that it is no longer true that a generic point v in the neuromanifold has a unique point (up to scaling) in its fiber \mu^{-1}(v) . In fact, the fibers of \mu_{\mathbb{P}} are now generally finite but larger than one. This can be seen via the interpretation of \mu as polynomial multiplication given in ( 8 ): If some stride s_{l} with 1\leq l0}^{D} . The convolution is a linear map \alpha_{w,\boldsymbol{s}} that produces an output tensor of format d^{(1)}\times\ldots\times d^{(D)} from an input tensor X of format (s^{(1)}(d^{(1)}-1)+k^{(1)})\times\ldots\times(s^{(D)}(d^{(D)}-1)+k^{(D)}) as follows: \displaystyle(\alpha_{w,\boldsymbol{s}}(X))_{i_{1},\ldots,i_{D}}=\sum_{j_{1}=0}^{k^{(1)}-1}\cdots\sum_{j_{D}=0}^{k^{(D)}-1}w_{j_{1},\ldots,j_{D}}X_{i_{1}s^{(1)}+j_{1},\ldots,i_{D}s^{(D)}+j_{D}} (14) for i_{m}=0,1,\ldots,d^{(m)}-1 . A linear convolutional network composes H such convolutions, which results again in a convolution on D -dimensional signals. Writing s^{(m)}_{l} and k^{(m)}_{l} for the strides and filter sizes in the l -th layer, the end-to-end convolution has stride (s^{(1)}_{1}\cdots s^{(1)}_{H},\ldots,s^{(D)}_{1}\cdots s^{(D)}_{H}) and filter format k^{(1)}\times\ldots\times k^{(D)} , where k^{(m)}:=k^{(m)}_{1}+\sum_{l=2}^{H}(k^{(m)}_{l}-1)\prod_{i=1}^{l-1}s^{(m)}_{i} . The network parametrization map assigns to a tuple of filter tensors (w_{1},\ldots,w_{H}) the filter of the end-to-end convolution: \displaystyle\mu:\mathbb{R}^{k_{1}^{(1)}\times\ldots\times k_{1}^{(D)}}\times\ldots\times\mathbb{R}^{k_{H}^{(1)}\times\ldots\times k_{H}^{(D)}}\to\mathbb{R}^{k^{(1)}\times\ldots\times k^{(D)}}. As in the case of one-dimensional signals, the end-to-end filter can be equivalently computed via polynomial multiplication. This time we need D pairs of variables (\boldsymbol{x},\boldsymbol{y}):=((x_{1},y_{1}),\ldots,(x_{D},y_{D})) . For tuples \boldsymbol{t}=(t^{(1)},\ldots,t^{(D)}) and \boldsymbol{k}=(k^{(1)},\ldots,k^{(D)}) of positive integers, we denote by \mathbb{R}[(\boldsymbol{x},\boldsymbol{y})^{\boldsymbol{t}}]_{\boldsymbol{k}-\boldsymbol{1}} the vector space of polynomials that are homogeneous of degree k^{(m)}-1 in the variables (x_{m}^{t^{(m)}},y_{m}^{t^{(m)}}) for m=1,2,\ldots,D . We can identify this space of polynomials with the space of tensors v in \mathbb{R}^{k^{(1)}\times\ldots\times k^{(D)}} via the map \displaystyle\pi_{\boldsymbol{t}}:v\mapsto\sum_{i_{1}=0}^{k^{(1)}-1}\cdots\sum_{i_{D}=0}^{k^{(D)}-1}v_{i_{1},\ldots,i_{D}}x_{1}^{t^{(1)}(k^{(1)}-1-i_{1})}y_{1}^{t^{(1)}i_{1}}\cdots x_{D}^{t^{(D)}(k^{(D)}-1-i_{D})}y_{D}^{t^{(D)}i_{D}}. Lemma 14 . Writing \boldsymbol{1}:=(1,\ldots,1) and t_{l}^{(m)}:=s_{1}^{(m)}\cdots s_{l-1}^{(m)} , we have \displaystyle\pi_{\boldsymbol{1}}(\mu(w_{1},\ldots,w_{H}))=\pi_{\boldsymbol{t}_{H}}(w_{H})\cdots\pi_{\boldsymbol{t}_{1}}(w_{1}). Proof. We abbreviate ( 14 ) by (\alpha_{w,\boldsymbol{s}}(X))_{\boldsymbol{i}}=\sum_{\boldsymbol{j}=\boldsymbol{0}}^{\boldsymbol{k}-\boldsymbol{1}}w_{\boldsymbol{j}}X_{\boldsymbol{i}\boldsymbol{s}+\boldsymbol{j}} . The assertion follows inductively from composing two such convolutions: \displaystyle(\alpha_{w_{2},\boldsymbol{s}_{2}}\circ\alpha_{w_{1},\boldsymbol{s}_{1}})(X)_{\boldsymbol{i}} \displaystyle=\sum_{\boldsymbol{j}=\boldsymbol{0}}^{\boldsymbol{k}_{2}-\boldsymbol{1}}w_{2,\boldsymbol{j}}\alpha_{w_{1},\boldsymbol{s}_{1}}(X)_{\boldsymbol{i}\boldsymbol{s}_{2}+\boldsymbol{j}}=\sum_{\boldsymbol{j}=\boldsymbol{0}}^{\boldsymbol{k}_{2}-\boldsymbol{1}}w_{2,\boldsymbol{j}}\sum_{\boldsymbol{l}=\boldsymbol{0}}^{\boldsymbol{k}_{1}-\boldsymbol{1}}w_{1,\boldsymbol{l}}X_{(\boldsymbol{i}\boldsymbol{s}_{2}+\boldsymbol{j})\boldsymbol{s}_{1}+\boldsymbol{l}} \displaystyle=\sum_{\boldsymbol{m}=0}^{(\boldsymbol{k}_{2}-\boldsymbol{1})\boldsymbol{s}_{1}+\boldsymbol{k}_{1}-\boldsymbol{1}}v_{\boldsymbol{m}}X_{\boldsymbol{i}\boldsymbol{s}_{2}\boldsymbol{s}_{1}+\boldsymbol{m}}, where I_{\boldsymbol{m}}:=\{(\boldsymbol{j},\boldsymbol{l})\mid\boldsymbol{0}\leq\boldsymbol{j}<\boldsymbol{k}_{2},\;\boldsymbol{0}\leq\boldsymbol{l}<\boldsymbol{k}_{1},\;\boldsymbol{j}\boldsymbol{s}_{1}+\boldsymbol{l}=\boldsymbol{m}\} and v_{\boldsymbol{m}}:=\sum_{(\boldsymbol{j},\boldsymbol{l})\in I_{\boldsymbol{m}}}w_{2,\boldsymbol{j}}w_{1,\boldsymbol{l}} . Note that v_{\boldsymbol{m}} coincides with the coefficient of the polynomial \pi_{\boldsymbol{s}_{1}}(w_{2})\pi_{\boldsymbol{1}}(w_{1}) , where the degrees of the variables y_{1},\ldots,y_{D} are m_{1},\ldots,m_{D} . ∎ The above lemma lets us interpret the network parametrization map \mu as the multiplication of polynomials, and we can think of the neuromanifold \mathcal{M} as the set of all polynomials in the variables x_{1},y_{1},\ldots,x_{D},y_{D} that can be factored as Q_{H}\cdots Q_{1} with Q_{l}\in\mathbb{R}[(\boldsymbol{x},\boldsymbol{y})^{\boldsymbol{t}_{l}}]_{\boldsymbol{k}_{l}-\boldsymbol{1}} . Similarly to the case of one-dimensional signals, we see that a filter size k^{(m)}_{l} being 1 means that the variables x_{m},y_{m} do not appear in the l -th layer. Corollary 15 . Let D>1 , and assume that every layer has at least two filter sizes larger than one. Then the projectivization of \mu is birational, up to possibly reordering some of the layer factors. In other words, almost every end-to-end filter in the neuromanifold \mathcal{M} has a unique factorization into layer filters, up to scalar multiplication in each layer and possibly reordering some of the layers. Proof. Our assumption means that, for every layer l , at least two variable pairs (x_{m_{1}},y_{m_{1}}) and (x_{m_{2}},y_{m_{2}}) appear in the polynomials in \mathbb{R}[(\boldsymbol{x},\boldsymbol{y})^{\boldsymbol{t}_{l}}]_{\boldsymbol{k}_{l}-\boldsymbol{1}} that correspond to the l -th layer filter. Hence, the polynomials in \mathbb{R}[(\boldsymbol{x},\boldsymbol{y})^{\boldsymbol{t}_{l}}]_{\boldsymbol{k}_{l}-\boldsymbol{1}} are generically irreducible. (This is a standard fact from algebraic geometry, following from Bertini’s theorem; see the proof of ( Kohn et al. 2022 , Corollary 4.16) for a formal argument.) Therefore, for a generic filter tuple \theta=(w_{1},\ldots,w_{H}) , the unordered set of filters \{w_{1},\ldots,w_{H}\} can be recovered (up to scalars) from the end-to-end filter \mu(\theta) by computing the factorization of \pi_{\boldsymbol{1}}(\mu(\theta)) into its irreducible factors \{\pi_{\boldsymbol{t}_{1}}(w_{1}),\ldots,\pi_{\boldsymbol{t}_{H}}(w_{H})\} . ∎ Observation 16 . a) The ordering of the filters in the above argument is not uniquely determined if and only if the spaces \mathbb{R}[(\boldsymbol{x},\boldsymbol{y})^{\boldsymbol{t}_{l}}]_{\boldsymbol{k}_{l}-\boldsymbol{1}} and \mathbb{R}[(\boldsymbol{x},\boldsymbol{y})^{\boldsymbol{t}_{L}}]_{\boldsymbol{k}_{L}-\boldsymbol{1}} for two layers l1 . b) The neural tangent kernel is not affected by the swap described in a). To see this, we consider \theta=(w_{1},\ldots,w_{H}) and \theta^{\prime} that is obtained from \theta by swapping w_{l} with w_{L} . Observation 3 also holds for convolutions on higher-dimensional signals as it only depends on the multilinearity of \mu . So we consider the block structure of the Jacobian matrices J_{\theta}=[J_{\theta,1}|\cdots|J_{\theta,H}] and J_{\theta^{\prime}}=[J_{\theta^{\prime},1}|\cdots|J_{\theta^{\prime},H}] . By Observation 3 and Lemma 14 , the matrix J_{\theta,i} encodes the linear map that multiplies \pi_{\boldsymbol{t}_{i}}(\cdot) with all polynomials \pi_{\boldsymbol{t}_{j}}(w_{j}) for j\neq i . Therefore, ( 15 ) implies that J_{\theta^{\prime},l}=J_{\theta,L} , J_{\theta^{\prime},L}=J_{\theta,l} , and J_{\theta^{\prime},i}=J_{\theta,i} for all i\notin\{l,L\} . Thus, we conclude K_{\theta^{\prime}}=\sum_{i=1}^{H}J_{\theta^{\prime},i}J_{\theta^{\prime},i}^{\top}=\sum_{i=1}^{H}J_{\theta,i}J_{\theta,i}^{\top}=K_{\theta} . c) \mathbb{R}[(\boldsymbol{x},\boldsymbol{y})^{\boldsymbol{t}_{l}}]_{\boldsymbol{k}_{l}-\boldsymbol{1}} coincides with \mathbb{R}[(\boldsymbol{x},\boldsymbol{y})^{\boldsymbol{t}_{L}}]_{\boldsymbol{k}_{L}-\boldsymbol{1}} if and only if \boldsymbol{k}_{l}=\boldsymbol{k}_{L} and, for all m=1,\ldots,D with k^{(m)}_{l}>1 , we have t^{(m)}_{l}=t^{(m)}_{L} . The latter condition means that the strides s^{(m)}_{l},\ldots,s^{(m)}_{L-1} are equal to one. As in the case of one-dimensional signals, the scaling ambiguity that appears in Corollary 15 is fixed (up to signs) by knowing the invariants in ( 1 ). Proposition 17 . Let \delta_{1},\ldots,\delta_{H-1}\in\mathbb{R} , and consider non-zero filter tensors w_{1}\in\mathbb{R}^{k_{1}^{(1)}\times\ldots\times k_{1}^{(D)}} , …, w_{H}\in\mathbb{R}^{k_{H}^{(1)}\times\ldots\times k_{H}^{(D)}} . Then, there are 2^{H-1} scalar tuples (\lambda_{1},\ldots,\lambda_{H})\in\mathbb{R}^{H} such that \lambda_{1}\cdots\lambda_{H}=1 and \displaystyle\delta_{i}=\|\lambda_{i+1}w_{i+1}\|_{F}^{2}-\|\lambda_{i}w_{i}\|_{F}^{2},\quad\text{ for all }i=1,\ldots,H-1. These scalar tuples are all equal up to signs. Proof. The same proof as for Proposition 2 applies. ∎ Putting everything together, we can now conclude for convolutions on higher-dimensional signals that the neural tangent kernel only depends on the end-to-end filter v\in\mathcal{M} and the invariants \delta_{1},\ldots,\delta_{H-1} , no matter whether some strides are equal to one. Theorem 18 . Let D>1 , and assume that every layer has at least two filter sizes larger than one. For any fixed \delta=(\delta_{1},\ldots,\delta_{H-1})\in\mathbb{R}^{H-1} and any end-to-end function v in the smooth locus of the neuromanifold \mathcal{M} , there are 2^{H-1} many parameters \theta=(w_{1},\ldots,w_{H}) in \mu^{-1}(v) that satisfy ( 1 ); they all have the same neural tangent kernel K_{\theta} . For any fixed \delta , these neural tangent kernels are a Riemmanian metric on the smooth locus of \mathcal{M} . Proof. We see from Lemma 14 that every fiber of the projectivization \mu_{\mathbb{P}} of \mu is finite. Moreover, Corollary 15 says that, after possibly modding out some permutations of the input layers (cf. Observation 16 ), the map \mu_{\mathbb{P}} becomes birational (i.e., its generic fiber is a singleton). For such a map, a standard fact from algebraic geometry ( Kohn et al. 2017 , Lemma 3.2) tells us that the image of \mu_{\mathbb{P}} (over the complex numbers) is smooth precisely at those points that have a single preimage under \mu_{\mathbb{P}} (modulo the necessary layer permutations) and where the derivative of \mu_{\mathbb{P}} at that preimage has maximal rank. In other words, for every end-to-end filter v in the smooth locus of \mathcal{M} , there is a unique filter tuple \theta , up to scalars and possibly some layer permutations, that parametrizes v ; moreover, d_{\theta}\mu has maximal rank. By Proposition 17 , the invariants \delta fix the scalars in the filter tuple \theta , up to signs. The signs and the possible layer permutations do not affect the neural tangent kernel K_{\theta} , by Observations 3 and 16 . Finally, the proofs of Lemma 6 and Corollary 7 hold verbatim. ∎ 4 Fully-Connected Networks We now move to the case of fully-connected linear networks. As seen in Section 1.1 , the NTK is parameter-independent for balanced initialization. Moreover, the polynomials ( 5 ) generate all algebraic relations that stay constant along the gradient flow curve for generic initialization ( Marcotte et al. 2024 ) . Hence, a natural question is whether, like for convolutional networks with strides larger than 1, for a generic choice of \Delta_{i} , the NTK is also parameter-independent. While the form ( 7 ) of the NTK suggests that this is not the case in general, we show this rigorously here. We first start by proving it for a particular choice of \Delta for one hidden layer. Example 19 . For \Delta_{1}=diag(0,15/4) , the neural tangent kernel is parameter-dependent. Indeed, let the end-to-end product matrix be equal to the identity W=I . Consider V_{2}=\operatorname{diag}(1,2) and V_{1}=\operatorname{diag}(1,1/2) . Hence V_{2}V_{1}=I_{2}=W and the balancedness constant matrix is indeed equal to V_{2}^{T}V_{2}-V_{1}V_{1}^{T}=\operatorname{diag}(0,15/4)=\Delta_{1} . Consider then U_{2}=\left(\begin{array}[]{c c}0&2\\ 1&0\end{array}\right) and U_{1}=\left(\begin{array}[]{c c}0&1\\ 1/2&0\end{array}\right) , we have again U_{2}U_{1}=I_{2} and U_{2}^{T}U_{2}-U_{1}U_{1}^{T}=\operatorname{diag}(0,15/4)=\Delta_{1} . However, using the formula ( 7 ) for the NTK in this case, we have K^{(\Delta_{1})}_{V_{2},V_{1}}=\nabla_{W}\ell(W)V_{1}^{T}V_{1}+V_{2}V_{2}^{T}\nabla_{W}\ell(W)\neq\nabla_{W}\ell(W)U_{1}^{T}U_{1}+U_{2}U_{2}^{T}\nabla_{W}\ell(W)=K^{(\Delta_{1})}_{U_{2},U_{1}} . Hence the neural tangent kernel is parameter-dependent. Using Remark 1 , we conclude that this is the case for almost all \Delta_{1} . We can also easily generalize this example to any depth and width of the network. We now go back to the balanced case, and provide a geometric reason for why the NTK is parameter-independent in this case. The following result shows that the balanced weight matrices that lead to a fixed product matrix W differ only by orthogonal group action. The latter are isometries of the Euclidean metric on the parameter space and thus do not affect the NTK. Moreover, analogously to Corollary 7 , the inverses of the NTK are the pushforward of the Euclidean metric on the space \Theta_{0} of balanced matrix tuples under the network parametrization map \mu , which is simply the quotient metric under the orthogonal action. Theorem 20 . Let (W_{H},\ldots,W_{1})\in GL_{d}(\mathbb{R})^{H} such that W=W_{H}\cdots W_{1} , and \Delta_{i}=W_{i+1}^{\top}W_{i+1}-W_{i}W_{i}^{\top}=0 for all i\in\{1,\ldots,H-1\} . Then the set of weights which are balanced and have as a product W is equal to \displaystyle\mu^{-1}(W)\cap\Theta_{0}=\left\{(W_{H}G_{H-1},G_{H-1}^{T}W_{H-1}G_{H-2},\ldots,G_{2}^{T}W_{2}G_{1},G_{1}^{T}W_{1})\mbox{ s.t. }G_{H-1},\ldots,G_{1}\in\mathcal{O}_{d}(\mathbb{R})\right\} (16) Proof. Consider the map \mu:GL_{d}(\mathbb{R})^{H}\to GL_{d}(\mathbb{R}) defined by \mu(W_{H},\ldots,W_{1})=W_{H}\cdots W_{1} . Then the fibers of a generic W can be written as (W_{H}G_{H-1},G_{H-1}^{-1}W_{H-1}G_{H-2},\ldots,G_{2}^{-1}W_{2}G_{1},G_{1}^{-1}W_{1}), where (W_{H},\ldots,W_{1}) are any fixed matrices such that their product is equal to W . Let us prove that the fiber such that all the layers are balanced leads then to all G_{i} being orthogonal. It was shown in Arora et al. 2018 that we have for all balanced weights (W^{\prime}_{H},\ldots,W^{\prime}_{1}) in the fiber of W , for all j\in\{1,\ldots,H\} the following equations W^{\prime}_{H}\cdots W^{\prime}_{j}(W^{\prime}_{H}\cdots W^{\prime}_{j})^{T}=(WW^{T})^{\frac{H-j+1}{H}}. Let (W_{H},\ldots,W_{1}) be balanced matrices in the fiber of W , and G_{1},\ldots,G_{H-1} invertible matrices such that (W_{H}G_{H-1},G_{H-1}^{-1}W_{H-1}G_{H-2},\ldots,G_{2}^{-1}W_{2}G_{1},G_{1}^{-1}W_{1}) are also balanced matrices (in the fiber of W ). We have, for j\in\{1,\ldots,H-1\} , \displaystyle W_{H}\cdots W_{j+1}G_{j}G_{j}^{T}(W_{H}\cdots W_{j+1})^{T} \displaystyle=(WW^{T})^{\frac{H-j}{H}} \displaystyle\text{ and }\quad\quad W_{H}\cdots W_{j+1}(W_{H}\cdots W_{j+1})^{T} \displaystyle=(WW^{T})^{\frac{H-j}{H}}. Hence by unicity of the symmetric PSD root of a symmetric PSD matrix we have \displaystyle W_{H}\cdots W_{j+1}G_{j}G_{j}^{T}(W_{H}\cdots W_{j+1})^{T}=W_{H}\cdots W_{j+1}(W_{H}\cdots W_{j+1})^{T}. Therefore, since all weight matrices are invertible we obtain G_{j}G_{j}^{T}=I so that the G_{j} , j=1,\ldots,H-1 are orthogonal. Conversely if some weight layers are balanced and in the fiber of W , then one can easily check that for G_{j} orthogonal, the layers are also balanced. ∎ 5 Conclusion In this paper, we investigated how the gradient flow on parameter space influences the dynamics of the end-to-end function in linear networks. We showed that, for a broad class of linear convolutional networks, the neural tangent kernel – and thus the corresponding function-space ODE – can be expressed independently of the individual layer parameters, relying instead on a global quantity defined at initialization, leading to a Riemaniann gradient flow. In contrast, we demonstrated that this property does not generally extend to fully connected linear networks, highlighting a fundamental structural difference between the two architectures. These findings raise intriguing questions about whether similar principles might emerge in the context of more practical, nonlinear networks.