Uniform Spectral Growth and Convergence of Muon in LoRA-Style Matrix Factorization Changmin Kang, Jihun Yun, Baekrok Shin, Yeseul Cho, Chulhee Yun Abstract Spectral gradient descent (SpecGD) orthogonalizes the matrix parameter updates and has inspired practical optimizers such as Muon. They often perform well in large language model (LLM) training, but their dynamics remain poorly understood. In the low-rank adaptation (LoRA) setting, where weight updates are parameterized as a product of two low-rank factors, we find a distinctive spectral phenomenon under Muon in LoRA fine-tuning of LLMs: singular values of the LoRA product show near-uniform growth across the spectrum, despite orthogonalization being performed on the two factors separately. Motivated by this observation, we analyze spectral gradient flow (SpecGF)—a continuous-time analogue of SpecGD—in a simplified LoRA-style matrix factorization setting and prove “equal-rate” dynamics: all singular values grow at equal rates up to small deviations. Consequently, smaller singular values attain their target values earlier than larger ones, sharply contrasting with the largest-first stepwise learning observed in standard gradient flow. Moreover, we prove that SpecGF in our setting converges to global minima from almost all initializations, provided the factor norms remain bounded; with ℓ 2 \ell_{2} regularization, we obtain global convergence. Lastly, we corroborate our theory with experiments in the same setting. † † affiliationnotice: Equal contribution 1 Introduction Classical optimization algorithms for training neural networks, such as stochastic gradient descent ( 32 ) and Adam(W) ( 24 ; 29 ) update network parameters in a coordinate-wise manner, ignoring the structure of parameters. In contrast, several modern approaches in deep learning exploit the structure of matrix parameters, focusing on how or what to update during training. One line of work follows the former direction, and the Muon optimizer ( 21 ) is one of representative examples. At each iteration t , Muon minimizes a loss function f(\mathbf{W}) in the weight matrix \mathbf{W} as \displaystyle\begin{split}\mathbf{M}_{t}&=\nabla f(\mathbf{W}_{t})+\mu\mathbf{M}_{t-1},\\ \mathbf{W}_{t+1}&=\mathbf{W}_{t}-\eta_{t}\mathcal{T}(\mathbf{M}_{t}),\end{split} (1) where \mu\in[0,1] tunes the momentum, \eta_{t} is the step size, and \mathcal{T} orthogonalizes the momentum (see Equation 5 ). In practice, Newton-Schulz iteration approximates \mathcal{T} by mapping most singular values of normalized \mathbf{M}_{t} to an interval [0.7,1.3] . If \mu=0 , the update follows spectral gradient descent (SpecGD, 8 ; 31 ). Orthogonalized updates induce the spectrum of matrices to be more isotropic ( 38 ; 3 ) . By leveraging such geometric structure, Muon has gained significant attention for training large language models (LLM), often outperforming Adam(W) ( 26 ; 38 ) . Another line of work targets what parameters are updated. This category includes Low-rank adaptation (LoRA, 18 ), which freezes the pretrained model and injects a low-rank update. LoRA parameterizes a weight update \mathbf{W}\in\mathbb{R}^{m\times n} as a product of low-rank factor matrices \mathbf{A}\mathbf{B} where \mathbf{A}\in\mathbb{R}^{m\times r} and \mathbf{B}\in\mathbb{R}^{r\times n} with r\ll\min\{m,n\} . This substantially reduces the number of trainable parameters while preserving strong fine-tuning performance. Combining these two intriguing approaches modifies both the parameterization and optimizer, potentially leading to distinct training dynamics and implicit biases. While recent studies combining Muon-style optimizers with LoRA have also reported promising empirical results ( 2 ; 4 ) , theoretical explanations about their dynamics and implicit bias remain absent. In Section 3 , we fine-tune language models using the Muon optimizer together with the LoRA parameterization to examine the dynamics, and we observe the Muon’s unique feature persists even after applying LoRA. Although each factor receives its own orthogonalized updates, all spectral components of the product \mathbf{A}\mathbf{B} evolve nearly uniformly. On the contrary, the dynamics of linear models with multiple layers trained under gradient descent exhibit incremental learning caused by saddle-to-saddle dynamics—the model learns singular values in a stepwise manner, from largest to smallest ( 5 ; 14 ; 25 ) . Motivated by this empirical observation, in subsequent sections, we deliver rigorous analyses of the dynamics induced from the orthogonalized updates under the LoRA setting. To this end, we model the matrix factorization objective induced by LoRA fine-tuning through the lens of spectral gradient flow (SpecGF) , a continuous-time model of SpecGD. We summarize our contributions as follows: Uniform Spectral Growth. Even though SpecGF orthogonalizes the gradients of the LoRA components \mathbf{A} and \mathbf{B} rather than their product, we show that the singular values of \mathbf{A}\mathbf{B} still evolve at a uniform rate under SpecGF ( Theorem 5.7 ). The dynamics make smaller singular values converge before larger ones. This sharply contrasts with standard gradient flow which exhibits largest-first stepwise learning dynamics ( 5 ; 14 ) . Convergence Guarantee. We prove that if SpecGF converges, it almost surely converges to global minima provided that the factor norms remain bounded ( Theorem 6.2 ); with \ell_{2} regularization, SpecGF globally converges. In addition, every global minimum is Lyapunov stable, and SpecGF converges to global minima exponentially fast ( Proposition 6.7 ). Empirical Validation. On matrix factorization, we confirm that SpecGF exhibits uniform growth and smallest-first convergence order, in contrast to the largest-first behavior of standard gradient flow. 2 Related Works Orthogonalized Optimizers. A line of research on the optimizers for matrix parameters, rather than vectorized ones, has been spotlighted for its empirical success. This includes Muon ( 21 ) and its variants, such as Dion ( 2 ) and MuonBP ( 23 ) that enhanced the efficiency, as well as preconditioner-equipped ones like Shampoo ( 16 ) and SOAP ( 36 ) . Muon normalizes the singular values of the matrix-valued updates, leading to a faster convergence and higher performance ( 26 ; 38 ) compared to Adam ( 24 ) . Recent studies have begun to analyze orthogonalized gradients through the lens of spectral gradient descent (SpecGD, 8 ), which can be viewed as Muon without momentum. Specifically, 12 study implicit bias and max-margin behavior under spectral descent, while 3 analyze generalization benefits in imbalanced classification. These results are informative but limited to data assumptions. Low-Rank Adaptation (LoRA). Low-rank adaptation (LoRA) is an efficient approach for fine-tuning large language models (LLM). This approach injects trainable low-rank factors into weight updates while keeping the base model fixed ( 18 ) , motivated by the observation that fine-tuning updates often lie in a low intrinsic-dimensional subspace ( 1 ) . LoRA reduces trainable parameters from this low-rank structure without sacrificing much downstream performance. Several recent studies report improved performance when applying Muon-style optimizers in LoRA fine-tuning ( 2 ; 4 ) . However, theoretical guarantees for convergence and training dynamics in this combined setting also remain largely unexplored. Training Dynamics. For deep linear models with standard gradient flow, large singular values are typically learned earlier than smaller ones ( 5 ; 14 ) , aligning with broader notions of spectral bias ( 10 ; 39 ) . Gradient-based methods can also exhibit saddle-to-saddle trajectories in matrix factorization problems ( 19 ) . Orthogonalized optimizers such as Muon or SpecGD induce different dynamics. 3 prove that, for linear and bilinear models under classification tasks with MSE loss, every singular value of the parameters increases at the same rate until saturation. Such a uniform growth across all spectral components leads to a better generalization of SpecGD compared to gradient descent under data imbalance. However, their analysis is limited to a strong initialization such as spectral initialization ( 41 ) . 30 demonstrate that Muon encourages a more uniform spectrum than AdamW in the trainable matrices. (a) RoBERTa-Base with Muon (b) RoBERTa-Base with AdamW (c) LLaMA-3.2-1B with Muon (d) LLaMA-3.2-1B with AdamW Figure 1 : Evolution of the singular values of the LoRA \mathbf{A}\mathbf{B} adapter applied to the query matrix in the first self-attention layer. 3 Empirical Observation: Uniform Growth in LoRA with Muon We begin with an empirical observation that motivates our theoretical analysis. Given pretrained weight \mathbf{W}_{0}\in\mathbb{R}^{m\times n} , we consider the fine-tuning problem \min_{\mathbf{W}\in\mathbb{R}^{m\times n}}\ell(\mathbf{W}_{0}+\mathbf{W}), (2) where \ell:\mathbb{R}^{m\times n}\to\mathbb{R} denotes the loss function and \mathbf{W} represents the trainable weight update. The LoRA approach reparameterizes \mathbf{W} as \mathbf{A}\mathbf{B} with low-rank factors \mathbf{A}\in\mathbb{R}^{m\times r} and \mathbf{B}\in\mathbb{R}^{r\times n} , where r\leq\min\{m,n\} . The factors are initialized as \mathbf{A}(0)=\bm{0} , and \mathbf{B}(0) is a small random matrix ( 18 ; 17 ) . When training LoRA adapters with Muon, we observe that the singular values of \mathbf{A}\mathbf{B} grow considerably more uniformly across the spectrum. This is a non-trivial phenomenon because Muon orthogonalizes the updates of \mathbf{A} and \mathbf{B} separately. Such a spectral growth in the product \mathbf{A}\mathbf{B} suggests some form of alignment between the matrices \mathbf{A} and \mathbf{B} , established during the training process. Experimental Setup. We fine-tune RoBERTa-Base ( 27 ) on SST-2 datasets from the GLUE benchmark ( 37 ) . For a larger-scale experiment, we train LLaMA-3.2-1B ( 15 ) on the Alpaca dataset ( 34 ) . For both cases, LoRA is applied to the query and value matrices with rank r=8 . We compare Muon and AdamW optimizers, tracking the singular values of the LoRA product throughout training. Additional training details are described in Section E.2 . LLM Fine-tuning Results. Figure 1 illustrates the evolution of the singular values of the LoRA \mathbf{A}\mathbf{B} adapter applied to the query matrix in the first self-attention layer. Figures 1(c) and 1(a) show results with the Muon optimizer: all singular values exhibit near-uniform evolution, maintaining parallel trajectories throughout training. In contrast, Figures 1(d) and 1(b) show that the AdamW optimizer does not necessarily attend to every spectral component equally; it may focus on larger singular values, aligning with the “largest-first” learning dynamics observed in standard gradient descent. The complete singular value trajectories and the effective rank ( 33 ) for all matrices are provided in Section E.2 . Across all layers, Muon promotes all singular values of LoRA \mathbf{A}\mathbf{B} adapter to evolve near-uniformly; thus, their effective rank is consistently close to the LoRA rank r=8 . Such dynamics are not always observed for AdamW across all matrices; the consistency is degraded. This empirical evidence confirms qualitatively different learning behavior of Muon from conventional optimizers. Modeling for Theory. The near-uniform evolution of singular values for all LoRA \mathbf{A}\mathbf{B} adapters stems from Muon, but such interactions are highly intractable due to the intricate interactions among the components of LLM. To rigorously analyze the dynamics from Muon with LoRA, we construct and study a simplified setting that captures the LoRA approach and orthogonalized matrix updates. (a) SpecGF (b) Vanilla GF Figure 2 : Comparison of singular value evolutions. While SpecGF induces uniform growth of spectrum of \mathbf{A}\mathbf{B} , GF induces the largest-first dynamics. Recall that Equation 2 aims to find the optimal update \mathbf{W}^{\star} during fine-tuning. A second-order Taylor expansion of \ell around \mathbf{W}_{0}+\mathbf{W}^{\star} yields: \displaystyle\phantom{=}\ell(\mathbf{W}_{0}+\mathbf{W})-\ell(\mathbf{W}_{0}+\mathbf{W}^{\star}) \displaystyle\approx\frac{1}{2}\left\langle\mathbf{W}-\mathbf{W}^{\star},\nabla^{2}\ell(\mathbf{W}_{0}+\mathbf{W}^{\star})(\mathbf{W}-\mathbf{W}^{\star})\right\rangle. Therefore, the fine-tuning objective can be approximated as the minimization of the quadratic term. To theoretically understand the dynamics of LoRA, we constrain the problem by relaxing the Hessian \nabla^{2}\ell(\mathbf{W}_{0}+\mathbf{W}^{\star}) to an identity: \min_{\mathbf{A},\mathbf{B}}\Big\{\mathcal{L}(\mathbf{A},\mathbf{B})\coloneqq\frac{1}{2}\left\lVert\mathbf{A}\mathbf{B}-\mathbf{Y}\right\rVert_{\mathrm{F}}^{2}\Big\}, (3) where \mathbf{A}\mathbf{B} and \mathbf{Y}\in\mathbb{R}^{m\times n} respectively take the place of \mathbf{W} and \mathbf{W}^{\star} . We train (\mathbf{A},\mathbf{B}) with SpecGD ( Equation 1 with \mu=0 ) with loss function in Equation 3 . We set (m,n,r)=(60,70,5) and then construct \mathbf{Y}=\mathbf{U}_{r}\bm{\Sigma}\mathbf{V}_{r}^{\top} where \mathbf{U}_{r}\in\mathbb{R}^{m\times r} and \mathbf{V}_{r}\in\mathbb{R}^{n\times r} are random orthonormal matrices, and \bm{\Sigma}=\mathrm{Diag}(8,5,3,1.5,0.7) . We initialize \mathbf{A}(0)=\mathbf{0} and \mathbf{B}(0) with i.i.d. Gaussian entries of scale \gamma=10^{-3} . We run SpecGD with a learning rate \eta=0.01 . Notably, the singular values of \mathbf{A}\mathbf{B} grow at a uniform rate; see Figure 2 . On the contrary, plain gradient descent exhibits incremental learning. It is expected that the singular values of a single matrix uniformly grow under orthogonalized updates. However, the surprising finding is that such uniform growth is also observed for the product \mathbf{A}\mathbf{B} , even though \mathbf{A} and \mathbf{B} are updated via orthogonalization separately. Our observation naturally leads to the following question: Key Question. Why do the singular values of \mathbf{A}\mathbf{B} evolve uniformly although the gradient of \mathbf{A} and \mathbf{B} are orthogonalized separately? To address this question, we study Equation 3 with spectral gradient methods. Our theoretical framework, developed in the following section, explains the observed uniform growth and the smallest-first convergence order. 4 Theoretical Setup Notation. We use bold uppercase letters (e.g., \mathbf{X} ) for real matrices, bold lowercase letters (e.g., \mathbf{x} ) for real vectors, and lowercase letters (e.g., x ) for real scalars. Let \mathbf{1}\in\mathbb{R}^{r} denote the all-ones vector and \mathbf{I}_{d} the d\times d identity matrix. We write \mathsf{GL}(r) for the set of invertible r\times r real matrices, and \mathrm{Diag}:\mathbb{R}^{d}\to\mathbb{R}^{d\times d} for the diagonal operator. We denote the Frobenius, spectral, and nuclear norms by \left\lVert\cdot\right\rVert_{\mathrm{F}} , \lVert\cdot\rVert_{2} , and \lVert\cdot\rVert_{*} , respectively. For matrices \mathbf{M} and \mathbf{N} , we write \left\langle\mathbf{M},\mathbf{N}\right\rangle\coloneqq\mathrm{Tr}(\mathbf{M}\mathbf{N}^{\top}) and denote the pseudoinverse of \mathbf{M} by \mathbf{M}^{\dagger} . The k -th largest singular value of \mathbf{M} is denoted by \sigma_{k}(\mathbf{M}) , and [k]\coloneqq\{1,\dots,k\} . 4.1 Problem Setup To align with the LoRA setting, we assume r\leq\mathrm{rank}(\mathbf{Y}) to solve Equation 3 . If \mathbf{A} and \mathbf{B} depend on time t , we may write \mathcal{L}(t)\coloneqq\mathcal{L}(\mathbf{A}(t),\mathbf{B}(t)) . We initialize each factor as \mathbf{A}(0)=\bm{0} and \mathbf{B}(0) at random, which coincides with the standard LoRA initialization where each entry of \mathbf{B}(0) is sampled from a Gaussian distribution ( 18 ; 17 ) . We analyze the continuous-time analog of SpecGD, spectral gradient flow (SpecGF) , defined by: \displaystyle\begin{split}\dot{\mathbf{A}}(t)&=-\mathcal{T}\left(\nabla_{\mathbf{A}}\mathcal{L}(t)\right)\\ &=-\mathcal{T}\left((\mathbf{A}(t)\mathbf{B}(t)-\mathbf{Y})\mathbf{B}(t)^{\top}\right),\\ \dot{\mathbf{B}}(t)&=-\mathcal{T}\left(\nabla_{\mathbf{B}}\mathcal{L}(t)\right)\\ &=-\mathcal{T}\left(\mathbf{A}(t)^{\top}(\mathbf{A}(t)\mathbf{B}(t)-\mathbf{Y})\right),\end{split} (4) where \mathcal{T} is an orthogonalization operator applied to the matrix gradients. For a matrix \mathbf{M}\in\mathbb{R}^{m\times n} with compact SVD \mathbf{U}_{\mathbf{M}}\mathbf{S}_{\mathbf{M}}\mathbf{V}_{\mathbf{M}}^{\top} and \mathbf{S}_{\mathbf{M}}\coloneqq\mathrm{Diag}\{\sigma_{1},\ldots,\sigma_{k}\}\in\mathbb{R}^{k\times k} , the exact orthogonalization is written as \mathcal{T}(\mathbf{M})\coloneqq\big((\mathbf{M}\mathbf{M}^{\top})^{\dagger}\big)^{1/2}\!\mathbf{M}=\mathbf{U}_{\mathbf{M}}\mathbf{V}_{\mathbf{M}}^{\top}. (5) Notice that \mathcal{T} normalizes all the nonzero singular values to 1 . However, \mathcal{T} is not analytic at rank-deficient points since it involves discontinuous singular-vector selections. To obtain an analytic vector field, we introduce a smoothed operator \mathcal{T}_{\beta} with \beta>0 : \mathcal{T}_{\beta}(\mathbf{M})\coloneqq\left(\mathbf{M}\mathbf{M}^{\top}+\beta\mathbf{I}_{m}\right)^{-1/2}\!\mathbf{M}. (6) Adding \beta\mathbf{I} makes the inverse square root well-defined, and makes SpecGF analytic everywhere; see Corollary D.2 for details. We can write \mathcal{T}_{\beta}(\mathbf{M}) as \mathcal{T}_{\beta}(\mathbf{M})=\mathbf{U}_{\mathbf{M}}\mathbf{S}_{\beta}\mathbf{V}_{\mathbf{M}}^{\top},\;\mathbf{S}_{\beta}\coloneqq\mathrm{Diag}\left\{\frac{\sigma_{i}}{\sqrt{\sigma_{i}^{2}+\beta}}\right\}_{i\in[k]}. Thus, \mathcal{T}_{\beta} maps all positive inputs to values near 1 , similar to NS iterations, and boils down to \mathcal{T} as \beta\to 0 . Throughout the paper, “SpecGF with \mathcal{N} ” refers to the flow where both \dot{\mathbf{A}} and \dot{\mathbf{B}} use the operator \mathcal{N}\in\{\mathcal{T},\mathcal{T}_{\beta}\} for orthogonalization. We use the compact SVD of \mathbf{Y} . Let r^{\star}=\mathrm{rank}(\mathbf{Y}) . Then \mathbf{Y}=\mathbf{U}_{r^{\star}}\bm{\Sigma}\mathbf{V}_{r^{\star}}^{\top} , where \bm{\Sigma}\in\mathbb{R}^{r^{\star}\times r^{\star}} contains the nonzero singular values in non-increasing order and \mathbf{U}_{r^{\star}} , \mathbf{V}_{r^{\star}} consist of the corresponding singular vectors. 5 Uniform Growth of Singular Values We analyze how the singular values of \mathbf{W}(t)=\mathbf{A}(t)\mathbf{B}(t) evolve from SpecGF with \mathcal{T}_{\beta} under LoRA initialization. A key finding is that, all active singular values grow at the same rate, a phenomenon we call equal-rate dynamics. This synchronized behavior implies that the smallest singular value reaches its target first. 5.1 Alignment Yields Decoupled Dynamics We start with the simple case—rank-1 \mathbf{Y} and r=1 —that leads to the decoupling of the dynamics of the product \mathbf{A}(t)\mathbf{B}(t) into those of scalars. The matrices reduce to \mathbf{Y}=\sigma\mathbf{u}\mathbf{v}^{\top},\quad\mathbf{A}\in\mathbb{R}^{m\times 1},\quad\mathbf{B}\in\mathbb{R}^{1\times n}, where \mathbf{u} and \mathbf{v} are unit vectors in \mathbb{R}^{m} and \mathbb{R}^{n} , respectively. Given a random unit vector \mathbf{w} in \mathbb{R}^{n} and small \gamma>0 , LoRA training starts at \mathbf{A}(0)=\bm{0},\mathbf{B}(0)=\gamma\mathbf{w}^{\top}. Notice that \left\langle\mathbf{v},\mathbf{w}\right\rangle\neq 0 almost surely. This simple case reduces the dynamics of two vectors to those of scalars: Theorem 5.1 (Informal) . Assume \left\langle\mathbf{v},\mathbf{w}\right\rangle\neq 0 and decompose \mathbf{w} into \mathbf{v} and \mathbf{z} with \left\langle\mathbf{v},\mathbf{z}\right\rangle=0 . Then, for all t\geq 0 , \mathbf{A}(t)=a(t)\mathbf{u},\quad\mathbf{B}(t)=b(t)\mathbf{v}^{\top}+c(t)\mathbf{z}^{\top}. If \gamma>0 is sufficiently small, then under SpecGF with \mathcal{T}_{\beta} , a(t)b(t)\to\sigma,\quad c(t)\to 0. Moreover, as \gamma\to 0 , both |a(t)-b(t)| and \lvert\dot{a}(t)-\dot{b}(t)\rvert vanish to 0 at any t\geq 0 . For the proof of Theorem 5.1 , see Section A.1 . If the initialization scale is small, the decoupling from \mathbf{A}(t)\mathbf{B}(t) to a(t),b(t) , and c(t) results in both the convergence to global minima and a similar rate increment in both factors. This decoupling extends to general matrices under SpecGF with spectral initialization ( 41 ) , which aligns the singular spaces of \mathbf{A}(0) and \mathbf{B}(0) with that of \mathbf{Y} . Writing \mathbf{Y}=\mathbf{U}_{r^{\star}}\bm{\Sigma}\mathbf{V}_{r^{\star}}^{\top} and letting \mathbf{U}_{r},\mathbf{V}_{r} be the first r columns of \mathbf{U}_{r^{\star}},\mathbf{V}_{r^{\star}} , the initialization takes \mathbf{A}(0)=\mathbf{U}_{r}\bm{\Sigma}_{\mathbf{A}}\mathbf{Q}^{\top} and \mathbf{B}(0)=\mathbf{Q}\bm{\Sigma}_{\mathbf{B}}\mathbf{V}_{r}^{\top} , where \bm{\Sigma}_{\mathbf{A}},\bm{\Sigma}_{\mathbf{B}} are nonnegative diagonal and \mathbf{Q}\in\mathbb{R}^{r\times r} is orthogonal. The dynamics then decouple into r independent pairs of singular values (\sigma_{i}(\mathbf{A}(t)),\sigma_{i}(\mathbf{B}(t))) . If \bm{\Sigma}_{\mathbf{A}} and \bm{\Sigma}_{\mathbf{B}} are sufficiently small, SpecGF with \mathcal{T}_{\beta} keeps both |\sigma_{i}(\mathbf{A}(t))-\sigma_{i}(\mathbf{B}(t))| and |\dot{\sigma}_{i}(\mathbf{A}(t))-\dot{\sigma}_{i}(\mathbf{B}(t))| small for all i ; see Section A.2 . 5.2 General case: Approximate as Near-Diagonal The analyses in the decoupled case utilize alignments between LoRA factors, \mathbf{A}(t) and \mathbf{B}(t) . However, such alignments are not applicable to Equation 3 for LoRA initialization. In this section, we consider the general case without alignments at initialization. Despite the lack of alignment, we show that under small initialization, all singular values of \mathbf{A}(t)\mathbf{B}(t) tend to grow at nearly the same rate, even though \mathbf{A} and \mathbf{B} are updated via separately orthogonalized gradients. Our approach introduces a coordinate system aligned with the target \mathbf{Y} , and we argue that orthogonalization largely separates the growth speed from the current singular-value scale, leading to near-uniform growth across singular values. Core Variables. We focus on the case r=r_{\star} for simplicity; the case r\cdots>\sigma_{r}>0 with \sigma_{i}=O(1) . Let \mathbf{V}_{\perp}\in\mathbb{R}^{n\times(n-r)} denote the orthogonal complement of \mathbf{V} (full right singular vectors), so that [\mathbf{V}_{r}\mid\mathbf{V}_{\perp}] is orthonormal. We define the core variables : \displaystyle\big(\mathbf{X}(t),\mathbf{Z}(t),\mathbf{Z}_{\perp}(t)\big)\coloneqq\big(\mathbf{U}_{r}^{\top}\mathbf{A}(t),\mathbf{B}(t)\mathbf{V}_{r},\mathbf{B}(t)\mathbf{V}_{\perp}\big). The core product is \mathbf{G}(t)\coloneqq\mathbf{X}(t)\mathbf{Z}(t) , and under \mathbf{A}(0)=\mathbf{0} , we have \mathbf{A}(t)=\mathbf{U}_{r}\mathbf{X}(t) for all t\geq 0 . We denote the diagonal entries of \mathbf{G}(t) by d_{i}(t):=[\mathbf{G}(t)]_{ii} for all i\in[r] . Further, we let d_{\min}(t)\coloneqq\min_{i\in[r]}d_{i}(t) . For a square matrix \mathbf{M} , we write \mathrm{Off}(\mathbf{M})\coloneqq\mathbf{M}\odot(\mathbf{1}\mathbf{1}^{\top}-\mathbf{I}_{r}) where \odot denotes the Hadamard (element-wise) product. Key Concepts. Under the reparametrization, we introduce two tolerances: (i) the alignment tolerance \delta>0 ; and (ii) the target tolerance \varepsilon>0 . • Alignment tolerance \delta>0 : We define \delta -alignment by \displaystyle\left\lVert\mathrm{Off}(\mathbf{G}(t))\right\rVert_{\mathrm{F}}\leq\delta\text{ and }\left\lVert\mathbf{X}(t)\mathbf{Z}_{\perp}(t)\right\rVert_{\mathrm{F}}\leq\delta, (7) i.e., \mathbf{G}(t) is near-diagonal for a sufficiently small \delta>0 . • Target tolerance \varepsilon>0 : We define d_{i}(t):=[\mathbf{G}(t)]_{ii} , e_{i}(t):=d_{i}(t)-\sigma_{i} . The active set is defined by \displaystyle\mathcal{I}_{\varepsilon}(t):=\{i\in[r]:|e_{i}(t)|>\varepsilon\}. (8) We call each index i\in[r] a mode , corresponding to the i -th singular value of \mathbf{A}\mathbf{B} . A mode i is active at time t if i\in\mathcal{I}_{\varepsilon}(t) , meaning it has not yet converged to its target \sigma_{i} . Note that, given a convergence to a global minima ( Section 6 ), for any \delta>0 there exists a finite t_{0}>0 such that \delta -alignment holds for all t\geq t_{0} since \mathcal{L}=\tfrac{1}{2}(\|\mathbf{G}(t)-\bm{\Sigma}\|_{F}^{2}+\|\mathbf{X}\mathbf{Z}_{\perp}\|_{F}^{2}) . Why Near-Diagonal? When \mathbf{G}(t) is diagonal, its diagonal entries d_{i}(t) exactly correspond to the singular values of \mathbf{A}(t)\mathbf{B}(t) , greatly simplifying the analysis. Although \mathbf{G}(t) is not exactly diagonal in general, under \delta -alignment ( 7 ) the diagonal entries can still approximate the singular value well. This is guaranteed by the following lemma. Lemma 5.2 (Diagonal approximation) . Under \delta -alignment ( 7 ), it follows that |\sigma_{i}(\mathbf{A}(t)\mathbf{B}(t))-d_{i}(t)|=O(\delta) , for all i\in[r] , where \sigma_{i}(\cdot) denotes the i -th singular value. In other words, d_{i}(t) approximates the singular value of \sigma_{i}(\mathbf{A}(t)\mathbf{B}(t)) up to O(\delta) error; see Lemma B.4 for the proof of Lemma 5.2 . Hence, it suffices to analyze the dynamics of d_{i}(t) to understand the singular values of \mathbf{A}(t)\mathbf{B}(t) . However, under LoRA initialization \mathbf{A}(0)=\bm{0} , the gradient \nabla_{\mathbf{B}}\mathcal{L} is very close to zero, thus \mathcal{T}_{\beta} operates in its near-zero regime, where the update is dominated by the \beta -regularization. We therefore introduce a short time \tau>0 such that the diagonal entries become positive enough. Lemma 5.3 (Initial growth) . Let \tau\asymp\frac{\sigma_{r}}{\sigma_{1}\sqrt{r}} and \tau\gamma<\sigma_{r}-\varepsilon holds for a small \varepsilon such that \varepsilon\ll\sigma_{r} . With probability at least 1-e^{-cr} for some absolute constant c>0 , it follows that d_{\min}(\tau)\geq d_{0}\asymp\tau\gamma>0 . Moreover, all modes are active at \tau : \mathcal{I}_{\varepsilon}(\tau)=[r] and d_{i}(\tau)<\sigma_{i} for all i . By Lemma 5.3 , we are now able to analyze the training dynamics starting at t=\tau , where the active set equals [r] . Since this regime corresponds to the early training, focusing on t\geq\tau incurs no loss of generality in characterizing the overall training behaviors; see Lemma B.5 for details. Similar to time \tau , we define a termination time as T:=\inf\{t\geq\tau:\mathcal{I}_{\varepsilon}(t)=\emptyset\} . Note that if SpecGF converges to a global minimum (cf. Proposition 6.7 ), the termination time T is necessarily finite. We prove uniform growth through the following steps: • Alignment from Small Initialization ( Proposition 5.4 ). Under Gaussian initialization with small \gamma , the \delta -alignment (near-diagonal) holds throughout training for \delta=O(\gamma) . • Square-Root Dynamics ( Lemma 5.5 and Lemma 5.6 ). In the near-diagonal regime with d_{\min}(t)>0 , the square-root coordinates s_{i}(t)=\sqrt{d_{i}(t)} evolve at approximately unit rate: \dot{s}_{i}(t)\approx 1 . • Uniform Growth ( Theorem 5.7 ). All active modes grow at the same rate in \sqrt{\cdot} -coordinates, leading to the “equal-rate” learning. 5.2.1 Alignment from Small Initialization A key insight is that under small initialization, the trajectory satisfies \delta -alignment throughout training with \delta=O(\gamma) . Proposition 5.4 (High-probability alignment) . Under Gaussian initialization \mathbf{A}(0)=\mathbf{0} , \mathbf{B}(0)=\gamma\mathbf{N} with \mathbf{N}_{ij}\stackrel{{\scriptstyle\mathrm{i.i.d.}}}{{\sim}}\mathcal{N}(0,1) , the following holds on [0,T] with probability at least 1-e^{-cr^{2}}-e^{-crn} for some absolute constant c>0 , \displaystyle\sup_{t\leq T}\left\lVert\mathrm{Off}(\mathbf{G}(t))\right\rVert_{\mathrm{F}}=O(\gamma),~~\sup_{t\leq T}\left\lVert\mathbf{X}(t)\mathbf{Z}_{\perp}(t)\right\rVert_{\mathrm{F}}=O(\gamma). This proposition mitigates the effect of off-diagonal interactions, which allows us to simplify the subsequent analysis. In particular, given a sufficiently small initialization scale \gamma>0 , we can directly use the diagonal entries d_{i}(t) as surrogates of the singular values of \mathbf{A}(t)\mathbf{B}(t) by Lemma 5.2 . See Proposition B.3 for the proof of Proposition 5.4 . 5.2.2 Square-Root Dynamics Our analysis of uniform growth requires that d_{\min}(t) remain bounded away from 0 along the trajectory. The following lemma guarantees the lower bound for all t\geq\tau . Lemma 5.5 (Persistence of Non-degeneracy) . For a sufficiently small tolerance such that \varepsilon\ll\sigma_{r} , d_{\min}(t)\geq d_{0}>0 holds for all t\in[\tau,T] . See Lemma B.9 for details. Given d_{\min}(t)>0 , we can analyze the evolution of s_{i}(t)=\sqrt{d_{i}(t)} . Lemma 5.6 (Square-root dynamics) . Suppose d_{\min}(t)\geq d_{0}>0 and \beta=O(\varepsilon^{3}) . Then for each active mode i\in\mathcal{I}_{\varepsilon}(t) with e_{i}(t)\leq-\varepsilon , the square-root coordinate s_{i}(t)=\sqrt{d_{i}(t)} satisfies \dot{s}_{i}(t)=1+O(\varepsilon^{1/4}) . \mathcal{T}_{\beta} decouples the speed from the current value d_{i} , unlike plain GF where \dot{d}_{i}\propto e_{i} ; see Lemma B.8 for details. 5.2.3 Main Result: Uniform Growth Putting these ingredients together, we show that all active modes grow uniformly. Theorem 5.7 (Uniform growth) . For a target tolerance such that \varepsilon\ll\sigma_{r} , consider SpecGF with \mathcal{T}_{\beta} where \beta=O(\varepsilon^{3}) under initialization \mathbf{A}(0)=\mathbf{0} , \mathbf{B}(0)=\gamma\mathbf{N} where \gamma=O(\varepsilon^{1/2}) and \mathbf{N}_{ij}\overset{\text{i.i.d}}{\sim}\mathcal{N}(0,1) . Then for all t\in[\tau,T] and all active modes i\in\mathcal{I}_{\varepsilon}(t) with e_{i}(t)\leq-\varepsilon , the following holds with probability at least 1-e^{-cr} for some absolute constant c>0 : (i) Approximate unit speed: \displaystyle\frac{\mathrm{d}}{\mathrm{d}t}\sqrt{d_{i}(t)}=1+O(\varepsilon^{1/4}) . (ii) Uniform growth: For any two active modes i,j\in\mathcal{I}_{\varepsilon}(t) , \displaystyle\left\lvert\frac{\mathrm{d}}{\mathrm{d}t}\sqrt{d_{i}(t)}-\frac{\mathrm{d}}{\mathrm{d}t}\sqrt{d_{j}(t)}\right\rvert=O(\varepsilon^{1/4}). Corollary 5.8 (Smallest singular value learns first) . All \sqrt{d_{i}(t)} trajectories are nearly parallel. Since all modes start near zero and \sigma_{r} requires the least growth, the smallest singular value reaches its target first. Our analysis naturally extends to the underparameterized setting r0 and b\in[\frac{1}{2},1) such that for all \mathbf{x}\in W , |F(\mathbf{x})-F(\mathbf{p})|^{b}\leq C\|\nabla F(\mathbf{x})\| . If the SpecGF iteration does not diverge to infinity in norm, it should have an accumulation point (\bar{\mathbf{A}},\bar{\mathbf{B}}) ; we show (\mathbf{A}(t),\mathbf{B}(t))\to(\bar{\mathbf{A}},\bar{\mathbf{B}}) . By Equation 9 and continuity of \mathcal{L} , \mathcal{L}(t) decreases to \mathcal{L}(\bar{\mathbf{A}},\bar{\mathbf{B}}) . Once (\mathbf{A}(t),\mathbf{B}(t)) is arbitrarily close to (\bar{\mathbf{A}},\bar{\mathbf{B}}) , it remains in that vicinity forever by Equation 9 and Lemma 6.5 . ∎ Stability of Global Minima. We state that every global minimum is Lyapunov stable under SpecGF with \mathcal{T}_{\beta} . Lyapunov stability is standard for describing whether trajectories that start near an equilibrium remain near it over time. Definition 6.6 (Lyapunov stability) . A critical point \bar{\mathbf{x}}\in\mathbf{R}^{d} is Lyapunov stable if for every neighborhood U_{\epsilon}\subset\mathbf{R}^{d} of \bar{\mathbf{x}} , there exists a neighborhood U_{\delta}\subset\mathbf{R}^{d} of \bar{\mathbf{x}} such that for all \mathbf{x}(t) with \mathbf{x}(0)\in U_{\delta} , \mathbf{x}(t)\in U_{\epsilon} for all t\geq 0 . If then, any starting point near some global minima cannot go far away from that global minimum. By Proposition 6.4 , bounded SpecGF iterations should converge to a critical point. The stability of the global minima is proved by Lemma 6.5 ; see Theorem D.15 for details. Therefore, the local convergence of SpecGF with \mathcal{T}_{\beta} is always guaranteed. Convergence Rate. The convergence rate of SpecGF with \mathcal{T}_{\beta} readily follows by Lemma 6.5 . Assume that SpecGF converges to a global minimum. We show that the loss function is essentially Morse-Bott, whose optimal exponent in the Łojasiewicz inequality is \frac{1}{2} , leading to exponential convergence; see Section D.5 for detailed proof. Proposition 6.7 (Informal) . Assume that the singular values of \mathbf{Y} are mutually distinct. If SpecGF with \mathcal{T}_{\beta} converges to a global minimum of \mathcal{L} , then it exponentially converges in terms of \mathcal{L}(t) . The assumption of distinct singular values of the target matrix is also considered in previous studies ( 14 ; 20 ) . Remark 6.8 . Assume that both \nabla_{\mathbf{A}}\mathcal{L}(t) and \nabla_{\mathbf{B}}\mathcal{L}(t) are full-rank throughout SpecGF with \mathcal{T} . Then SpecGF with \mathcal{T} is analytic, and the same arguments for SpecGF with \mathcal{T}_{\beta} hold. Moreover, if SpecGF with \mathcal{T} converges, then it should converge in finite time ; see Proposition D.18 for details. Global Convergence via Regularization. Adding a \ell_{2} regularizer guarantees global convergence of SpecGF with \mathcal{T}_{\beta} . All theoretical claims for SpecGF with \mathcal{T}_{\beta} also apply to the regularized SpecGF with \mathcal{T}_{\beta} . Moreover, boundedness of \mathbf{A} and \mathbf{B} readily follows from the regularization; see Section D.6 for details. Therefore, regularized SpecGF with \mathcal{T}_{\beta} globally converges to the global minima almost surely. 7 Empirical Validation of Our Theory In this section, we validate our theoretical results. In particular, we confirm the uniform growth of singular values under SpecGF and the equal-rate learning behavior. Figure 3 : Loss comparison. Setup. We use the same setup as in Figure 2 . We set \beta=10^{-8} (for our theory) or \beta=0 (for exact orthonormalization). To analyze the dynamics, we track core variables \mathbf{X} and \mathbf{Z} with the diagonal entries d_{i}=[\mathbf{G}]_{ii} and \sigma_{i}(\mathbf{A}\mathbf{B}) . Results. We highlight several key observations: 1. Convergence. In Figure 3 , both methods successfully minimize the loss, confirming the convergence guarantee of Theorem 6.2 . Notably, SpecGF ( \beta=10^{-8} ) converges significantly faster than vanilla GF. 2. Diagonal approximation. In Figure 4 , the left and right panels show that d_{i}(t)=[\mathbf{X}(t)\mathbf{Z}(t)]_{ii} closely tracks \sigma_{i}(\mathbf{A}(t)\mathbf{B}(t)) , validating Lemma 5.2 . This confirms that the diagonal entries of the core product serve as accurate surrogates for the singular values of \mathbf{A}(t)\mathbf{B}(t) . 3. Uniform Growth. Under SpecGF, the curves \sqrt{d_{i}(t)} in Figure 4 exhibit nearly identical slopes, demonstrating our Theorem 5.7 . In contrast, vanilla GF shows non-uniform growth. We present further experimental results in Section E.1 by modifying the training settings of Figure 2 . Across various settings, the uniform growth of singular values of the product of LoRA factors persists. (a) Vanilla GF (b) SpecGF ( \beta=10^{-8} ) (c) SpecGF ( \beta=0 ) Figure 4 : Comparison of SpecGF and vanilla GF on matrix factorization with a rank- 5 target. 8 Conclusion We analyzed spectral gradient flow (SpecGF) for LoRA-style matrix factorization and proved that singular values of the product \mathbf{A}\mathbf{B} exhibit uniform growth, leading to a “equal-rate” learning that contrasts with the largest-first behavior of standard gradient flow. We also established that SpecGF almost surely converges to global minima; with \ell_{2} regularization, global convergence is also guaranteed. We validated our theory via experiments on matrix factorization and LLM fine-tuning. A key limitation is that global convergence without regularization remains open due to the absence of known invariants for SpecGF. Extending our continuous-time analysis to discrete-time SpecGD and more general network architectures is an important future work. Impact Statement This paper presents work whose goal is to advance the field of machine learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here. Supplementary Materials Appendix A Speed of singular values of \mathbf{A} and \mathbf{B} In this section, we show that the singular values of \mathbf{A} and \mathbf{B} increase at the same speed if the whole matrix evolution can be well-decoupled into scalar evolutions. A.1 Rank-1 Case Next, we consider the case with random initialization but simplify the problem to rank-1 \mathbf{Y} and r=1 case: \mathbf{Y}=\sigma\mathbf{u}\mathbf{v}^{\top},\quad\mathbf{A}\in\mathbb{R}^{m\times 1},\quad\mathbf{B}\in\mathbb{R}^{1\times n}. There, \mathbf{u} and \mathbf{v} are unit vectors in \mathbb{R}^{m} and \mathbb{R}^{n} , respectively. A.1.1 SpecGF with \mathcal{T}_{\beta} We note that the dynamics exist uniquely (from analyticity of \mathcal{T}_{\beta} ) and globally (from boundedness of velocity by \mathcal{T}_{\beta} ). In addition, for this rank-1 case, SpecGF with \mathcal{T}_{\beta} is: \displaystyle\mathcal{T}_{\beta}(\nabla_{\mathbf{A}}\mathcal{L})=\frac{\nabla_{\mathbf{A}}\mathcal{L}}{\sqrt{\left\lVert\nabla_{\mathbf{A}}\mathcal{L}\right\rVert_{\mathrm{F}}^{2}+\beta}},\quad\mathcal{T}_{\beta}(\nabla_{\mathbf{B}}\mathcal{L})=\frac{\nabla_{\mathbf{B}}\mathcal{L}}{\sqrt{\left\lVert\nabla_{\mathbf{B}}\mathcal{L}\right\rVert_{\mathrm{F}}^{2}+\beta}}. We sample a random unit vector \mathbf{w} in \mathbb{R}^{n} , then \alpha\triangleq\left\langle\mathbf{v},\mathbf{w}\right\rangle\neq 0 with probability 1. Assumption A.1 . \alpha\neq 0 . A typical LoRA training would start at \mathbf{A}(0)=\bm{0},\quad\mathbf{B}(0)=\gamma\mathbf{w}^{\top}, for sufficiently small \gamma>0 with gradients \displaystyle\nabla_{\mathbf{A}}\mathcal{L} \displaystyle=(\mathbf{A}\mathbf{B}-\mathbf{Y})\mathbf{B}^{\top}, \displaystyle\nabla_{\mathbf{B}}\mathcal{L} \displaystyle=\mathbf{A}^{\top}(\mathbf{A}\mathbf{B}-\mathbf{Y}). We first show that \mathbf{A}(t)\in\mathrm{span}\{\mathbf{u}\} and \mathbf{B}(t)^{\top}\in\mathrm{span}\{\mathbf{v},\mathbf{w}\} at all t\geq 0 for SpecGF with \mathcal{T} . Lemma A.2 . For all t\geq 0 , \mathbf{A}(t)\in\mathrm{span}\{\mathbf{u}\} and \mathbf{B}(t)^{\top}\in\mathrm{span}\{\mathbf{v},\mathbf{w}\} hold. Proof. Assume that \mathbf{A}(t)=a(t)\mathbf{u} and \mathbf{B}(t)=b_{0}(t)\mathbf{v}^{\top}+c_{0}(t)\mathbf{w}^{\top} for some a(t),b_{0}(t),c_{0}(t)\in\mathbb{R} . We directly compute the gradients: \displaystyle\nabla_{\mathbf{A}}\mathcal{L}(t) \displaystyle=(\mathbf{A}(t)\mathbf{B}(t)-\mathbf{Y})\mathbf{B}(t)^{\top}=\mathbf{u}\left((a(t)b_{0}(t)-\sigma)(b_{0}(t)+c_{0}(t)\alpha)+a(t)c_{0}(t)(b_{0}(t)\alpha+c_{0}(t))\right), \displaystyle\nabla_{\mathbf{B}}\mathcal{L}(t) \displaystyle=\mathbf{A}(t)^{\top}(\mathbf{A}(t)\mathbf{B}(t)-\mathbf{Y})=a(t)(a(t)b_{0}(t)-\sigma)\mathbf{v}^{\top}+a(t)^{2}c_{0}(t)\mathbf{w}^{\top}. Normalization only modifies the coefficients, so the updates \dot{\mathbf{A}}(t)\in\mathrm{span}\{\mathbf{u}\} and \dot{\mathbf{B}}(t)^{\top}\in\mathrm{span}\{\mathbf{v},\mathbf{w}\} hold. The proof is concluded as \mathbf{A}(0)\in\mathrm{span}\{\mathbf{u}\} and \mathbf{B}(0)^{\top}\in\mathrm{span}\{\mathbf{v},\mathbf{w}\} . ∎ Without loss of generality, assume \alpha>0 . If \alpha<0 , one can simply flip the sign of \mathbf{u} and \mathbf{v} and the argument analogously holds. If \alpha=1 , then \mathbf{w}=\mathbf{v} and it is straightforward to see that \displaystyle\nabla_{\mathbf{A}}\mathcal{L}(t) \displaystyle=(\mathbf{A}(t)\mathbf{B}(t)-\mathbf{Y})\mathbf{B}(t)^{\top}=\mathbf{u}(a(t)b_{0}(t)-\sigma)b_{0}(t), \displaystyle\nabla_{\mathbf{B}}\mathcal{L}(t) \displaystyle=\mathbf{A}(t)^{\top}(\mathbf{A}(t)\mathbf{B}(t)-\mathbf{Y})=a(t)(a(t)b_{0}(t)-\sigma)\mathbf{v}^{\top}. Hence, if \gamma^{2}<\sigma , then the dynamics of a(t) and b_{0}(t) are exactly equal to those in Proposition A.14 . Henceforth, consider the case \alpha\in(0,1) . For the convenience of later analysis, define \mathbf{z}\triangleq\frac{1}{\sqrt{1-\alpha^{2}}}(\mathbf{w}-\alpha\mathbf{v}), (10) which gives \mathbf{w}=\alpha\mathbf{v}+\sqrt{1-\alpha^{2}}\mathbf{z} and \left\langle\mathbf{v},\mathbf{z}\right\rangle=0 . Thus, \mathbf{B}(t)=(b_{0}(t)+\alpha c_{0}(t))\mathbf{v}^{\top}+\sqrt{1-\alpha^{2}}c_{0}(t)\mathbf{z}^{\top} . For brevity, we write the parameters as \mathbf{A}(t)=a(t)\mathbf{u},\quad\mathbf{B}(t)\triangleq b(t)\mathbf{v}^{\top}+c(t)\mathbf{z}^{\top}. (11) The goal is to show that 1) a(t)b_{0}(t)\to\sigma with neither a(t)\to\infty nor b_{0}(t)\to\infty and 2) c_{0}(t)\to 0 ; this is equivalent to showing that 1) a(t)b(t)\to\sigma with neither a(t)\to\infty nor b(t)\to\infty and 2) c(t)\to 0 . The remaining part is to show that a(t)\approx b(t) ; due to randomness in the initialization, \dot{a}(t) cannot be exactly equal to \dot{b}(t) . Instead, we show that |a(t)-b(t)| and |\dot{a}(t)-\dot{b}(t)| are both uniformly small over t , and the gap vanishes to 0 as \gamma\to 0 . We derive the time derivative of the coefficients. From \displaystyle\mathcal{L}(t) \displaystyle=\frac{1}{2}\left\lVert\mathbf{A}(t)\mathbf{B}(t)-\mathbf{Y}\right\rVert_{\mathrm{F}}^{2}=\frac{1}{2}\left\lVert\mathbf{u}\left((a(t)b(t)-\sigma)\mathbf{v}^{\top}+a(t)c(t)\mathbf{z}^{\top}\right)\right\rVert_{\mathrm{F}}^{2} \displaystyle=\frac{1}{2}\lVert(a(t)b(t)-\sigma)\mathbf{v}^{\top}+a(t)c(t)\mathbf{z}^{\top}\rVert_{2}^{2}\triangleq\frac{1}{2}\lVert\mathbf{d}(t)\rVert_{2}^{2}, with \mathbf{d}(t)=a(t)\mathbf{B}(t)-\sigma\mathbf{v}^{\top} . Whenever \lVert\mathbf{d}(t)\rVert_{2}=0 , a(t)b(t)=\sigma and c(t)=0 and the dynamics terminate. From now on, the interval of time of our interest is [0,T_{\mathrm{max}}) where T_{\mathrm{max}}=\inf\{t:\lVert\mathbf{d}(t)\rVert_{2}=0\} ; T_{\mathrm{max}} could be possibly infinite. We derive the loss gradient, with m(t)\triangleq(a(t)b(t)-\sigma)b(t)+a(t)c(t)^{2} , \displaystyle\nabla_{\mathbf{A}}\mathcal{L}(t) \displaystyle=\mathbf{u}\left(m(t)\right), \displaystyle\nabla_{\mathbf{B}}\mathcal{L}(t) \displaystyle=a(t)\mathbf{d}(t), and we obtain \displaystyle\dot{a}(t) \displaystyle=-\frac{m(t)}{\sqrt{m(t)^{2}+\beta}} \displaystyle\dot{b}(t) \displaystyle=-\frac{a(t)(a(t)b(t)-\sigma)}{\sqrt{a(t)^{2}\lVert\mathbf{d}(t)\rVert_{2}^{2}+\beta}} \displaystyle\dot{c}(t) \displaystyle=-\frac{a(t)^{2}c(t)}{\sqrt{a(t)^{2}\lVert\mathbf{d}(t)\rVert_{2}^{2}+\beta}}. As c(0)=\sqrt{1-\alpha^{2}}\gamma>0 , it directly follows that c(t)\geq 0 for all t\geq 0 and c(t) is non-increasing. In addition, we have a(0)=0 with \dot{a}(0)>0 and b(0)=\alpha\gamma>0 . The simultaneous non-negativity of a and b at t=0 makes them keep non-negative for all t\geq 0 under continuity. If b hits 0 while a(t)\geq 0 , it gains non-negative speed; same for a . The only way for them to be negative is that both a and b go across 0 at the same time, but then the dynamics become stationary. Therefore, we can safely argue that a(t),b(t),c(t)\geq 0 . We return our attention to the loss function. We know that \mathcal{L}(t) is non-increasing, hence \lVert\mathbf{d}(t)\rVert_{2} is also non-increasing. Precisely, \displaystyle\frac{\mathrm{d}}{\mathrm{d}t}\frac{1}{2}\lVert\mathbf{d}(t)\rVert_{2}^{2} \displaystyle=\left\langle\mathbf{d}(t),\dot{\mathbf{d}}(t)\right\rangle \displaystyle=\left\langle\mathbf{d}(t),\dot{a}(t)\mathbf{B}(t)+a(t)\dot{\mathbf{B}}(t)\right\rangle \displaystyle=\dot{a}(t)\left\langle\mathbf{d}(t),\mathbf{B}(t)\right\rangle+a(t)\left\langle\mathbf{d}(t),\dot{\mathbf{B}}(t)\right\rangle \displaystyle=-\frac{m(t)}{\sqrt{m(t)^{2}+\beta}}\cdot m(t)-a(t)\frac{a(t)\lVert\mathbf{d}(t)\rVert_{2}^{2}}{\sqrt{a(t)^{2}\lVert\mathbf{d}(t)\rVert_{2}^{2}+\beta}} \displaystyle=-\frac{m(t)^{2}}{\sqrt{m(t)^{2}+\beta}}-\frac{a(t)^{2}\lVert\mathbf{d}(t)\rVert_{2}^{2}}{\sqrt{a(t)^{2}\lVert\mathbf{d}(t)\rVert_{2}^{2}+\beta}}\leq 0. Notice that m(0)=-\sigma b(0)<0 , thus \frac{\mathrm{d}}{\mathrm{d}t}\lVert\mathbf{d}(0)\rVert_{2}^{2}<0 . Hence, for t>0 , \lVert\mathbf{d}(t)\rVert_{2}^{2}<\lVert\mathbf{d}(0)\rVert_{2}^{2} . Then, \lVert\mathbf{d}(t)\rVert_{2}<\sigma for t>0 also implies that a(t),b(t)>0 ; if any of two is 0 at t , then \lVert\mathbf{d}(t)\rVert_{2}=\lVert(a(t)b(t)-\sigma)\mathbf{v}^{\top}+a(t)c(t)\mathbf{z}^{\top}\rVert_{2}\geq\sigma . Lemma A.3 . m(t)<0 . Proof. Recall that m(t)=(a(t)b(t)-\sigma)b(t)+a(t)c(t)^{2} and m(0)=-\sigma b(0)<0 . If there exists some time \tau\triangleq\min\left\{t:m(t)\geq 0\right\} , then m(\tau)=0 and \dot{a}(\tau)=0 . In addition, from \displaystyle m(t) \displaystyle=\left\langle\mathbf{d}(t),\mathbf{B}(t)\right\rangle \displaystyle=a(t)\left\lVert\mathbf{B}(t)\right\rVert_{\mathrm{F}}^{2}-\sigma b(t). \displaystyle\dot{m}(t) \displaystyle=\dot{a}(t)\left\lVert\mathbf{B}(t)\right\rVert_{\mathrm{F}}^{2}+a(t)\frac{\mathrm{d}}{\mathrm{d}t}\left\lVert\mathbf{B}(t)\right\rVert_{\mathrm{F}}^{2}-\sigma\dot{b}(t) \displaystyle=\dot{a}(t)\left\lVert\mathbf{B}(t)\right\rVert_{\mathrm{F}}^{2}-2a(t)^{2}\frac{m(t)}{\sqrt{a(t)^{2}\lVert\mathbf{d}(t)\rVert_{2}^{2}+\beta}}-\sigma\dot{b}(t). Plugging \tau vanishes the first two terms on the right-hand side, giving \dot{m}(\tau)=-\sigma\dot{b}(\tau)<0 , a contradiction: If \dot{b}(\tau)=0 , then we have a(\tau)b(\tau)=\sigma and c(\tau)=0 . ∎ Lemma A.3 directly implies that a(t)b(t)<\sigma , thus a(t) and b(t) are strictly increasing and positive. Therefore, all the scalars a(t),b(t) , and c(t) are bounded, thus SpecGF with \mathcal{T}_{\beta} converges. From 6 , the critical point is either the origin or \mathbf{Y} , the point where a(t)b(t)=\sigma and c(t)=0 . For t>0 , \lVert\mathbf{d}(t)\rVert_{2}^{2}<\lVert\mathbf{d}(0)\rVert_{2}^{2} and thus the origin cannot be the converging point. Therefore, SpecGF with \mathcal{T}_{\beta} converges to the global minima, and we see that \lVert\mathbf{d}(t)\rVert_{2}\to 0 , thus a(t)b(t)\to\sigma and c(t)\to 0 . This recovers the limiting behavior of b_{0}(t)=b(t)-\alpha c_{0}(t) and c_{0}(t)=\frac{1}{\sqrt{1-\alpha^{2}}}c(t) . We now bound \Delta(t)\triangleq a(t)-b(t) . Recall that \lVert\mathbf{d}(t)\rVert_{2}=\sqrt{\left(a(t)b(t)-\sigma\right)^{2}+\left(a(t)c(t)\right)^{2}}. 1. a(t)\geq b(t) : Notice that, from -m(t)=-(a(t)b(t)-\sigma)b(t)-a(t)c(t)^{2}\leq-(a(t)b(t)-\sigma)b(t)\leq-(a(t)b(t)-\sigma)a(t) , \dot{a}(t)=-\frac{m(t)}{\sqrt{m(t)^{2}+\beta}}=\frac{-m(t)}{\sqrt{(-m(t))^{2}+\beta}}\leq\frac{-(a(t)b(t)-\sigma)a(t)}{\sqrt{(a(t)b(t)-\sigma)^{2}a(t)^{2}+\beta}} Then \displaystyle\dot{\Delta}(t) \displaystyle=\dot{a}(t)-\dot{b}(t) \displaystyle\leq\frac{-(a(t)b(t)-\sigma)a(t)}{\sqrt{(a(t)b(t)-\sigma)^{2}a(t)^{2}+\beta}}+\frac{a(t)(a(t)b(t)-\sigma)}{\sqrt{a(t)^{2}\lVert\mathbf{d}(t)\rVert_{2}^{2}+\beta}} \displaystyle=\frac{-(a(t)b(t)-\sigma)a(t)\left(a(t)^{2}\lVert\mathbf{d}(t)\rVert_{2}^{2}-a(t)^{2}(a(t)b(t)-\sigma)^{2}\right)}{\sqrt{a(t)^{2}\lVert\mathbf{d}(t)\rVert_{2}^{2}+\beta}\sqrt{(a(t)b(t)-\sigma)^{2}a(t)^{2}+\beta}\left(\sqrt{a(t)^{2}\lVert\mathbf{d}(t)\rVert_{2}^{2}+\beta}+\sqrt{(a(t)b(t)-\sigma)^{2}a(t)^{2}+\beta}\right)} \displaystyle=\frac{-(a(t)b(t)-\sigma)a(t)^{3}\left(a(t)c(t)\right)^{2}}{\sqrt{a(t)^{2}\lVert\mathbf{d}(t)\rVert_{2}^{2}+\beta}\sqrt{(a(t)b(t)-\sigma)^{2}a(t)^{2}+\beta}\left(\sqrt{a(t)^{2}\lVert\mathbf{d}(t)\rVert_{2}^{2}+\beta}+\sqrt{(a(t)b(t)-\sigma)^{2}a(t)^{2}+\beta}\right)} \displaystyle=-\dot{c}(t)\frac{-(a(t)b(t)-\sigma)a(t)^{3}c(t)}{\sqrt{(a(t)b(t)-\sigma)^{2}a(t)^{2}+\beta}\left(\sqrt{a(t)^{2}\lVert\mathbf{d}(t)\rVert_{2}^{2}+\beta}+\sqrt{(a(t)b(t)-\sigma)^{2}a(t)^{2}+\beta}\right)} \displaystyle\leq-\dot{c}(t)\frac{-(a(t)b(t)-\sigma)a(t)^{3}c(t)}{-(a(t)b(t)-\sigma)a(t)\left(a(t)\lVert\mathbf{d}(t)\rVert_{2}\right)}\leq-\dot{c}(t). 2. a(t)0 ; notice that T is independent of \gamma . \gamma>0 satisfies \gamma^{2}<\frac{\alpha\sigma}{4(1-\alpha^{2})^{3/2}}\quad\text{and}\quad\gamma<\min\left\{\frac{\sigma/2}{\sqrt{1-\alpha^{2}}(\alpha+T)},1\right\}. A.4 translates to c(0)^{3}\leq\frac{\sigma}{4}b(0),\quad c(0)\left(b(0)+T\right)\leq\frac{\sigma}{2}. The first one is straightforward; the second one holds as \displaystyle c(0)\left(b(0)+T\right)=\gamma\sqrt{1-\alpha^{2}}\left(\gamma\alpha+T\right)\leq\gamma\sqrt{1-\alpha^{2}}\left(\alpha+T\right)<\frac{\sigma}{2}. We see that \displaystyle-\dot{\Delta}(t) \displaystyle=\dot{b}(t)-\dot{a}(t) \displaystyle\leq-\frac{a(t)(a(t)b(t)-\sigma)}{\sqrt{(a(t)b(t)-\sigma)^{2}a(t)^{2}+\beta}}+\frac{m(t)}{\sqrt{(a(t)b(t)-\sigma)^{2}b(t)^{2}+\beta}} \displaystyle=-\frac{a(t)(a(t)b(t)-\sigma)}{\sqrt{(a(t)b(t)-\sigma)^{2}a(t)^{2}+\beta}}+\frac{b(t)(a(t)b(t)-\sigma)}{\sqrt{(a(t)b(t)-\sigma)^{2}b(t)^{2}+\beta}}+\frac{a(t)c(t)^{2}}{\sqrt{(a(t)b(t)-\sigma)^{2}b(t)^{2}+\beta}} \displaystyle\leq\frac{a(t)c(t)^{2}}{\sqrt{(a(t)b(t)-\sigma)^{2}b(t)^{2}+\beta}}\leq\frac{a(t)c(t)^{2}}{\sqrt{\beta}}. We aim to bound the integral of a(t)c(t)^{2} . To this end, define t_{c} when a gets larger than c : t_{c}\triangleq\inf\left\{t:a(t)\geq c(t)\right\}0 . • On t\geq t_{c} , a(t)c(t)^{2}=-\dot{c}(t)\frac{c(t)\sqrt{a(t)^{2}\lVert\mathbf{d}(t)\rVert_{2}^{2}+\beta}}{a(t)}\leq-\dot{c}(t)\frac{c(t)}{a(t)}\left(a(t)\lVert\mathbf{d}(t)\rVert_{2}+\sqrt{\beta}\right)\leq-\dot{c}(t)\left(c(t)\sigma+\sqrt{\beta}\right). This implies \displaystyle\int_{t_{c}}^{T_{\mathrm{max}}}a(t)c(t)^{2}\mathrm{d}t\leq\sigma\int_{t_{c}}^{T_{\mathrm{max}}}-\dot{c}(t)c(t)\mathrm{d}t+\sqrt{\beta}\int_{t_{c}}^{T_{\mathrm{max}}}-\dot{c}(t)\mathrm{d}t\leq\frac{\sigma}{2}c(0)^{2}+\sqrt{\beta}c(0). • For t\leq t_{c} , we show that T\geq t_{c} from A.4 . Then, the integral of a(t)c(t)^{2} on [0,t_{c}] is bounded by that on [0,T] . Suppose by contradiction that t_{c}>T . On [0,T]\subset[0,t_{c}] , a(t)0 . A is computed as follows: a(t)^{2}-\Delta(t)a(t)=a(t)b(t)\leq\sigma, and thus a(t)^{2}\leq\sigma+|\Delta(t)|a(t)\leq\sigma+C(\gamma)a(t), giving A=\frac{C(\gamma)+\sqrt{C(\gamma)^{2}+4\sigma}}{2}. Notice that A\to\sqrt{\sigma} as \gamma\to 0 . Proposition A.5 . \left\lvert\dot{a}(t)-\dot{b}(t)\right\rvert\leq\frac{1}{\sqrt{\beta}}\left(C(\gamma)\sigma+A(1-\alpha^{2})\gamma^{2}\right)+\frac{A^{4}(1-\alpha^{2})\gamma^{2}}{2\beta}. The right-hand side goes to 0 as \gamma\to 0 . Proof. Recall \displaystyle\dot{a}(t) \displaystyle=-\frac{m(t)}{\sqrt{m(t)^{2}+\beta}},\quad m(t)=(a(t)b(t)-\sigma)b(t)+a(t)c(t)^{2}, \displaystyle\dot{b}(t) \displaystyle=-\frac{a(t)(a(t)b(t)-\sigma)}{\sqrt{a(t)^{2}\lVert\mathbf{d}(t)\rVert_{2}^{2}+\beta}}. Then, \displaystyle\left\lvert\dot{a}(t)-\dot{b}(t)\right\rvert \displaystyle\leq\left\lvert-\frac{m(t)}{\sqrt{m(t)^{2}+\beta}}-\frac{a(t)(\sigma-a(t)b(t))}{\sqrt{a(t)^{2}(\sigma-a(t)b(t))^{2}+\beta}}\right\rvert \displaystyle\phantom{=}+\left\lvert\frac{a(t)(\sigma-a(t)b(t))}{\sqrt{a(t)^{2}(\sigma-a(t)b(t))^{2}+\beta}}-\frac{a(t)(\sigma-a(t)b(t))}{\sqrt{a(t)^{2}\lVert\mathbf{d}(t)\rVert_{2}^{2}+\beta}}\right\rvert \displaystyle\leq\frac{1}{\sqrt{\beta}}\left\lvert-m(t)-a(t)(\sigma-a(t)b(t))\right\rvert \displaystyle\phantom{=}+a(t)(\sigma-a(t)b(t))\left(\frac{1}{\sqrt{a(t)^{2}(\sigma-a(t)b(t))^{2}+\beta}}-\frac{1}{\sqrt{a(t)^{2}\left((\sigma-a(t)b(t))^{2}+(a(t)c(t))^{2}\right)+\beta}}\right) \displaystyle\leq\frac{1}{\sqrt{\beta}}\left\lvert(b(t)-a(t))(\sigma-a(t)b(t))-a(t)c(t)^{2}\right\rvert+a(t)(\sigma-a(t)b(t))\frac{a(t)^{4}c(t)^{2}}{2\left(a(t)^{2}(\sigma-a(t)b(t))^{2}+\beta\right)^{\frac{3}{2}}} \displaystyle\leq\frac{1}{\sqrt{\beta}}\left(|\Delta(t)|(\sigma-a(t)b(t))+a(t)c(t)^{2}\right)+\frac{a(t)^{4}c(t)^{2}}{2\left(a(t)^{2}(\sigma-a(t)b(t))^{2}+\beta\right)} \displaystyle\leq\frac{1}{\sqrt{\beta}}\left(C(\gamma)\sigma+Ac(0)^{2}\right)+\frac{A^{4}c(0)^{2}}{2\beta}=\frac{1}{\sqrt{\beta}}\left(C(\gamma)\sigma+A(1-\alpha^{2})\gamma^{2}\right)+\frac{A^{4}(1-\alpha^{2})\gamma^{2}}{2\beta}. The second inequality follows from the Lipschtiz continuity of x\mapsto\frac{x}{\sqrt{x^{2}+\beta}} with constant \frac{1}{\sqrt{\beta}} . The third inequality follows from the mean value theorem applied to x\mapsto\frac{1}{\sqrt{x}} . ∎ A.1.2 SpecGF with \mathcal{T} We note that the dynamics exist globally (from boundedness of velocity by \mathcal{T} ). In addition, for this rank-1 case, SpecGF with \mathcal{T} is identical to the normalized gradient descent: \displaystyle\mathcal{T}(\nabla_{\mathbf{A}}\mathcal{L})=\frac{\nabla_{\mathbf{A}}\mathcal{L}}{\left\lVert\nabla_{\mathbf{A}}\mathcal{L}\right\rVert_{\mathrm{F}}},\quad\mathcal{T}(\nabla_{\mathbf{B}}\mathcal{L})=\frac{\nabla_{\mathbf{B}}\mathcal{L}}{\left\lVert\nabla_{\mathbf{B}}\mathcal{L}\right\rVert_{\mathrm{F}}}. Just as SpecGF with \mathcal{T}_{\beta} , we sample a random unit vector \mathbf{w} in \mathbb{R}^{n} , then \alpha\triangleq\left\langle\mathbf{v},\mathbf{w}\right\rangle\neq 0 with probability 1. Assumption A.6 . \alpha\neq 0 . For now, assume \alpha>0 ; we will justify it shortly after. A typical LoRA training would start at \mathbf{A}_{0}=\bm{0},\quad\mathbf{B}_{0}=\gamma\mathbf{w}^{\top}, for sufficiently small \gamma>0 with gradients \displaystyle\nabla_{\mathbf{A}}\mathcal{L} \displaystyle=(\mathbf{A}\mathbf{B}-\mathbf{Y})\mathbf{B}^{\top}, \displaystyle\nabla_{\mathbf{B}}\mathcal{L} \displaystyle=\mathbf{A}^{\top}(\mathbf{A}\mathbf{B}-\mathbf{Y}). However, \mathbf{A} being zero induces a zero gradient for \mathbf{B} , an ill-defined situation for SpecGF. This motivates us to initialize the SpecGF at the next iteration of LoRA training with step size \eta : \mathbf{A}_{1}=\eta\mathbf{u},\quad\mathbf{B}_{1}=\gamma\mathbf{w}^{\top}. Thus the trajectory is intact but with nonzero gradients. For simplicity, we take \eta=\gamma . SpecGF with \mathcal{T} starts at \mathbf{A}(0)=\gamma\mathbf{u},\quad\mathbf{B}(0)=\gamma\mathbf{w}^{\top}. (12) We first show that \mathbf{A}_{t}\in\mathrm{span}\{\mathbf{u}\} and \mathbf{B}_{t}^{\top}\in\mathrm{span}\{\mathbf{v},\mathbf{w}\} at all t\geq 0 for SpecGD with \mathcal{T} . Lemma A.7 . For all t\geq 0 , \mathbf{A}_{t}\in\mathrm{span}\{\mathbf{u}\} and \mathbf{B}_{t}^{\top}\in\mathrm{span}\{\mathbf{v},\mathbf{w}\} hold. Proof. Assume that \mathbf{A}_{t}=a\mathbf{u} and \mathbf{B}_{t}=b\mathbf{v}^{\top}+c\mathbf{w}^{\top} for some a,b,c\in\mathbb{R} . We directly compute the gradients: \displaystyle\nabla_{\mathbf{A}}\mathcal{L}(t) \displaystyle=(\mathbf{A}_{t}\mathbf{B}_{t}-\mathbf{Y})\mathbf{B}_{t}^{\top}=\mathbf{u}\left((ab-\sigma)(b+c\alpha)+ac(b\alpha+c)\right), \displaystyle\nabla_{\mathbf{B}}\mathcal{L}(t) \displaystyle=\mathbf{A}_{t}^{\top}(\mathbf{A}_{t}\mathbf{B}_{t}-\mathbf{Y})=a(ab-\sigma)\mathbf{v}^{\top}+a^{2}c\mathbf{w}^{\top}. Normalization only modifies the coefficients, so the updates \Delta\mathbf{A}_{t}\in\mathrm{span}\{\mathbf{u}\} and \Delta\mathbf{B}_{t}^{\top}\in\mathrm{span}\{\mathbf{v},\mathbf{w}\} hold. The proof is concluded as \mathbf{A}_{0}\in\mathrm{span}\{\mathbf{u}\} and \mathbf{B}_{0}^{\top}\in\mathrm{span}\{\mathbf{v},\mathbf{w}\} . ∎ Hence, for SpecGF with \mathcal{T} , we assume that there exists a solution that is the continuous version of the solution for SpecGD with \mathcal{T} ; we study only this solution. Indeed, such \mathbf{A}(t) and \mathbf{B}(t) are the solutions for the rank-1 case with suitable coefficient functions. Assumption A.8 . There exist scalars a(t),b_{0}(t) , and c_{0}(t) in \mathbb{R} that are absolutely continuous and \mathbf{A}(t)=a(t)\mathbf{u} and \mathbf{B}(t)=b_{0}(t)\mathbf{v}^{\top}+c_{0}(t)\mathbf{w}^{\top} . We require absolute continuity to define the derivative at almost every t . In addition, we assume that the initialization scale \gamma is sufficiently small. Assumption A.9 . \gamma^{2}<\alpha\sigma . Without loss of generality, assume \alpha>0 . If \alpha<0 , one can simply flip the sign of \mathbf{u} and \mathbf{v} and the argument analogously holds. If \alpha=1 , then \mathbf{w}=\mathbf{v} and it is straightforward to see that \displaystyle\nabla_{\mathbf{A}}\mathcal{L}(t) \displaystyle=(\mathbf{A}(t)\mathbf{B}(t)-\mathbf{Y})\mathbf{B}(t)^{\top}=\mathbf{u}(a(t)b_{0}(t)-\sigma)b_{0}(t), \displaystyle\nabla_{\mathbf{B}}\mathcal{L}(t) \displaystyle=\mathbf{A}(t)^{\top}(\mathbf{A}(t)\mathbf{B}(t)-\mathbf{Y})=a(t)(a(t)b_{0}(t)-\sigma)\mathbf{v}^{\top}. Hence, if \gamma^{2}<\sigma , then a(t) and b_{0}(t) get a unit update, which yields a(t)=\gamma+\max\{T_{\gamma},t\},\quad b_{0}(t)=\gamma+\max\{T_{\gamma},t\} (13) where T_{\gamma}>0 satisfies (T_{\gamma}+\gamma)^{2}=\sigma . Moreover, in this case, a(t)=b_{0}(t) for all t\geq 0 . Henceforth, consider the case \alpha\in(0,1) . For the convenience of later analysis, define \mathbf{z}\triangleq\frac{1}{\sqrt{1-\alpha^{2}}}(\mathbf{w}-\alpha\mathbf{v}), (14) which gives \mathbf{w}=\alpha\mathbf{v}+\sqrt{1-\alpha^{2}}\mathbf{z} and \left\langle\mathbf{v},\mathbf{z}\right\rangle=0 . Thus, \mathbf{B}(t)=(b_{0}(t)+\alpha c_{0}(t))\mathbf{v}^{\top}+\sqrt{1-\alpha^{2}}c_{0}(t)\mathbf{z}^{\top} . For brevity, we write the parameters as \mathbf{A}(t)=a(t)\mathbf{u},\quad\mathbf{B}(t)\triangleq b(t)\mathbf{v}^{\top}+c(t)\mathbf{z}^{\top}. (15) The goal is to show that 1) a(t)b_{0}(t)\to\sigma with neither a(t)\to\infty nor b_{0}(t)\to\infty and 2) c_{0}(t)\to 0 ; this is equivalent to showing that 1) a(t)b(t)\to\sigma with neither a(t)\to\infty nor b(t)\to\infty and 2) c(t)\to 0 . The remaining part is to show that a(t)\approx b(t) ; due to randomness in the initialization, \dot{a}(t) cannot be exactly equal to \dot{b}(t) even if we employ \mathcal{T} . Instead, we show that |a(t)-b(t)| and |\dot{a}(t)-\dot{b}(t)| are both uniformly small over t , and the gap vanishes to 0 as \gamma\to 0 . We derive the time derivative of the coefficients. From \displaystyle\mathcal{L}(t) \displaystyle=\frac{1}{2}\left\lVert\mathbf{A}(t)\mathbf{B}(t)-\mathbf{Y}\right\rVert_{\mathrm{F}}^{2}=\frac{1}{2}\left\lVert\mathbf{u}\left((a(t)b(t)-\sigma)\mathbf{v}^{\top}+a(t)c(t)\mathbf{z}^{\top}\right)\right\rVert_{\mathrm{F}}^{2} \displaystyle=\frac{1}{2}\lVert(a(t)b(t)-\sigma)\mathbf{v}^{\top}+a(t)c(t)\mathbf{z}^{\top}\rVert_{2}^{2}\triangleq\frac{1}{2}\lVert\mathbf{d}(t)\rVert_{2}^{2}, with \mathbf{d}(t)=a(t)\mathbf{B}(t)-\sigma\mathbf{v}^{\top} . Whenever \lVert\mathbf{d}(t)\rVert_{2}=0 , a(t)b(t)=\sigma and c(t)=0 and the dynamics terminate. From now on, the interval of time of our interest is [0,T_{\mathrm{max}}) where T_{\mathrm{max}}=\inf\{t:\lVert\mathbf{d}(t)\rVert_{2}=0\} ; T_{\mathrm{max}} could be possibly infinite. We derive the loss gradient, with m(t)\triangleq(a(t)b(t)-\sigma)b(t)+a(t)c(t)^{2} , \displaystyle\nabla_{\mathbf{A}}\mathcal{L}(t) \displaystyle=\mathbf{u}\left(m(t)\right), \displaystyle\nabla_{\mathbf{B}}\mathcal{L}(t) \displaystyle=a(t)\mathbf{d}(t), and we obtain \displaystyle\dot{a}(t) \displaystyle=-\mathrm{sgn}\left(m(t)\right) \displaystyle\dot{b}(t) \displaystyle=-\mathrm{sgn}(a(t))\frac{a(t)b(t)-\sigma}{\lVert\mathbf{d}(t)\rVert_{2}} \displaystyle\dot{c}(t) \displaystyle=-\mathrm{sgn}(a(t))\frac{a(t)c(t)}{\lVert\mathbf{d}(t)\rVert_{2}}. Since all the time derivatives are bounded by 1, due to \mathcal{T} , we see that a,b,c are all (Lipschitz) continuous in t . As c(0)=\sqrt{1-\alpha^{2}}\gamma>0 , it directly follows that c(t)\geq 0 for all t\geq 0 and c(t) is non-increasing. In addition, we have a(0)=\gamma with \dot{a}(0)=1 (from A.9 ) and b(0)=\alpha\gamma>0 . The simultaneous non-negativity of a and b at t=0 makes them keep non-negative for all t\geq 0 under continuity. If b hits 0 while a(t)\geq 0 , it gains non-negative speed; same for a . The only way for them to be negative is that both a and b go across 0 at the same time, but then the dynamics become stationary. Therefore, we can safely argue that a(t),b(t),c(t)\geq 0 . We return our attention to the loss function. We know that \mathcal{L}(t) is non-increasing, hence \lVert\mathbf{d}(t)\rVert_{2} is also non-increasing. Precisely, \displaystyle\frac{\mathrm{d}}{\mathrm{d}t}\frac{1}{2}\lVert\mathbf{d}(t)\rVert_{2}^{2} \displaystyle=\left\langle\mathbf{d}(t),\dot{\mathbf{d}}(t)\right\rangle \displaystyle=\left\langle\mathbf{d}(t),\dot{a}(t)\mathbf{B}(t)+a(t)\dot{\mathbf{B}}(t)\right\rangle \displaystyle=\dot{a}(t)\left\langle\mathbf{d}(t),\mathbf{B}(t)\right\rangle+a(t)\left\langle\mathbf{d}(t),\dot{\mathbf{B}}(t)\right\rangle \displaystyle=-\mathrm{sgn}\left(m(t)\right)m(t)-a(t)\mathrm{sgn}(a(t))\frac{\lVert\mathbf{d}(t)\rVert_{2}^{2}}{\lVert\mathbf{d}(t)\rVert_{2}} \displaystyle=-\lvert m(t)\rvert-|a(t)|\lVert\mathbf{d}(t)\rVert_{2}\leq 0. Notice that \frac{\mathrm{d}}{\mathrm{d}t}\lVert\mathbf{d}(0)\rVert_{2}^{2}<0 . Hence, for t>0 , \lVert\mathbf{d}(t)\rVert_{2}^{2}<\lVert\mathbf{d}(0)\rVert_{2}^{2} . Then, \lVert\mathbf{d}(t)\rVert_{2}<\sigma for t>0 also implies that a(t),b(t)>0 ; if any of two is 0 at t , \lVert\mathbf{d}(t)\rVert_{2}=\lVert(a(t)b(t)-\sigma)\mathbf{v}^{\top}+a(t)c(t)\mathbf{z}^{\top}\rVert_{2}\geq\sigma . Therefore, we may discard the sign of a(t) in the time derivatives for t>0 : \displaystyle\dot{a}(t) \displaystyle=-\mathrm{sgn}\left(m(t)\right) \displaystyle\dot{b}(t) \displaystyle=-\frac{a(t)b(t)-\sigma}{\lVert\mathbf{d}(t)\rVert_{2}} \displaystyle\dot{c}(t) \displaystyle=-\frac{a(t)c(t)}{\lVert\mathbf{d}(t)\rVert_{2}}. Lemma A.10 . m(t)<0 . Proof. Recall that m(t)=(a(t)b(t)-\sigma)b(t)+a(t)c(t)^{2} and m(0)=-\sigma b(0)<0 ; moreover, m(t) is (bounded) Lipschitz continuous. If there exists some time \tau\triangleq\min\left\{t:m(t)\geq 0\right\} , then m(\tau)=0 and \dot{a}(\tau)=0 . In addition, from \displaystyle m(t) \displaystyle=\left\langle\mathbf{d}(t),\mathbf{B}(t)\right\rangle \displaystyle=a(t)\left\lVert\mathbf{B}(t)\right\rVert_{\mathrm{F}}^{2}-\sigma b(t). \displaystyle\dot{m}(t) \displaystyle=\dot{a}(t)\left\lVert\mathbf{B}(t)\right\rVert_{\mathrm{F}}^{2}+a(t)\frac{\mathrm{d}}{\mathrm{d}t}\left\lVert\mathbf{B}(t)\right\rVert_{\mathrm{F}}^{2}-\sigma\dot{b}(t) \displaystyle=\dot{a}(t)\left\lVert\mathbf{B}(t)\right\rVert_{\mathrm{F}}^{2}-2a(t)\frac{m(t)}{\lVert\mathbf{d}(t)\rVert_{2}}-\sigma\dot{b}(t). Plugging \tau vanishes the first two terms on the right-hand side, giving \dot{m}(\tau)=-\sigma\dot{b}(\tau)<0 , a contradiction: If \dot{b}(\tau)=0 , then we have a(\tau)b(\tau)=\sigma and c(\tau)=0 . ∎ Lemma A.10 directly implies that a(t)b_{0}(t)<\sigma , thus a(t) and b_{0}(t) are strictly increasing and positive. Therefore, all the scalars a(t),b_{0}(t) , and c_{0}(t) are bounded. Let \Delta(t)\triangleq a(t)-b(t) . We will bound |\Delta(t)-\Delta(0)| for t>0 . Recall that \lVert\mathbf{d}(t)\rVert_{2}=\sqrt{\left(a(t)b(t)-\sigma\right)^{2}+\left(a(t)c(t)\right)^{2}}. Then, \dot{\Delta}(t)=\dot{a}(t)-\dot{b}(t)=1+\frac{a(t)b(t)-\sigma}{\lVert\mathbf{d}(t)\rVert_{2}}\geq 0. Furthermore, \lVert\mathbf{d}(t)\rVert_{2}+a(t)b(t)-\sigma\leq a(t)c(t) yields \dot{\Delta}(t)=1+\frac{a(t)b(t)-\sigma}{\lVert\mathbf{d}(t)\rVert_{2}}\leq-\dot{c}(t). We see that \left\lvert\dot{\Delta}(t)\right\rvert\leq-\dot{c}(t) , hence the total variation of \Delta is at most c(0) , that is, |\Delta(t)-\Delta(0)|\leq c(0)=\gamma\sqrt{1-\alpha^{2}}. Plugging \Delta(0)=-\gamma\alpha , we see that |\Delta(t)|\leq\gamma\alpha+\gamma\sqrt{1-\alpha^{2}}\triangleq C(\gamma). (16) Observe that the difference between the “singular values” are uniformly small, and especially the difference vanishes to 0 as \gamma\to 0 . It remains to show that SpecGF with \mathcal{T} converges to the global minima. There exists d_{\infty}\in[0,\sigma) such that \lVert\mathbf{d}(t)\rVert_{2}\to d_{\infty} as t\to\infty . If d_{\infty}\neq 0 , we show that this leads to a contradiction combined with \limsup\frac{\mathrm{d}}{\mathrm{d}t}\frac{1}{2}\lVert\mathbf{d}(t)\rVert_{2}^{2}=0 . Proposition A.11 . d_{\infty}=0 . Proof. Suppose not, that is, d_{\infty}>0 . Since a,b,c are bounded Lipschitz continuous, so is \lVert\mathbf{d}(t)\rVert_{2} . Thus, \lVert\mathbf{d}(t)\rVert_{2} is differentiable a.e., and there exists a sequence \{t_{n}\} where \frac{\mathrm{d}}{\mathrm{d}t}\lVert\mathbf{d}(t_{n})\rVert_{2}\to 0\quad\Rightarrow\quad\frac{\mathrm{d}}{\mathrm{d}t}\lVert\mathbf{d}(t_{n})\rVert_{2}^{2}\to 0. From \displaystyle\frac{\mathrm{d}}{\mathrm{d}t}\frac{1}{2}\lVert\mathbf{d}(t_{n})\rVert_{2}^{2}=-\lvert(a(t_{n})b(t_{n})-\sigma)b(t_{n})+a(t_{n})c(t_{n})^{2}\rvert-|a(t_{n})|\lVert\mathbf{d}(t_{n})\rVert_{2}\leq 0, we obtain a(t_{n}),b(t_{n})\to 0. However, this implies that \lVert\mathbf{d}(t_{n})\rVert_{2}=\sqrt{\left(a(t_{n})b(t_{n})-\sigma\right)^{2}+\left(a(t_{n})c(t_{n})\right)^{2}}\to\sigma . ∎ Therefore, we see that \lVert\mathbf{d}(t)\rVert_{2}\to 0 , thus a(t)b(t)\to\sigma and c(t)\to 0 . This recovers the limiting behavior of b_{0}(t)=b(t)-\alpha c_{0}(t) and c_{0}(t)=\frac{1}{\sqrt{1-\alpha^{2}}}c(t) : a(t)b_{0}(t)\to\sigma,\quad c_{0}(t)\to 0. (17) Synchronized speed of singular values. We know that a(t) is bounded; denote the upper bound of a(t) as A>0 . A is computed as follows: a(t)^{2}-\Delta(t)a(t)=a(t)b(t)\leq\sigma, and thus a(t)^{2}\leq\sigma+|\Delta(t)|a(t)\leq\sigma+C(\gamma)a(t), giving A=\frac{C(\gamma)+\sqrt{C(\gamma)^{2}+4\sigma}}{2}. Notice that A\to\sqrt{\sigma} as \gamma\to 0 . Proposition A.12 . \left\lvert\dot{a}(t)-\dot{b}(t)\right\rvert\leq\frac{A\gamma\sqrt{1-\alpha^{2}}}{2(\sigma-a(t)b(t))}. Specifically, whenever \sigma-a(t)b(t)\geq K\sigma, for some 00 , the fraction is extremely small, making \theta quickly decrease towards 0. A.2 Spectral initialization Case We first show a closed formula for singular values of \mathbf{A}(t) and \mathbf{B}(t) if the singular matrices of \mathbf{Y} are directly applied at the initialization. Recall that \mathbf{Y}=\mathbf{U}_{r^{\star}}\mathbf{S}\mathbf{V}_{r^{\star}}^{\top} is the compact SVD of \mathbf{Y} . Let \mathbf{U}_{r} and \mathbf{V}_{r} consist of the first r columns of \mathbf{U}_{r^{\star}} and \mathbf{V}_{r^{\star}} , respectively. The spectral initialization ( 41 ) sets \mathbf{A}(0)=\mathbf{U}_{r}\bm{\Sigma}_{\mathbf{A},0}\mathbf{Q}^{\top},\quad\mathbf{B}(0)=\mathbf{Q}\bm{\Sigma}_{\mathbf{B},0}\mathbf{V}_{r}^{\top}, where \bm{\Sigma}_{\mathbf{A},0} and \bm{\Sigma}_{\mathbf{B},0} are non-negative diagonal matrices. 41 set the diagonal matrices to be the square root of the singular value matrix of the one-step gradient of full fine-tuning; any non-negative diagonal matrices suffice for our case. Proposition A.13 . Let \mathbf{A}(0)=\mathbf{U}_{r}\bm{\Sigma}_{\mathbf{A},0}\mathbf{Q}^{\top},\quad\mathbf{B}(0)=\mathbf{Q}\bm{\Sigma}_{\mathbf{B},0}\mathbf{V}_{r}^{\top}, where \bm{\Sigma}_{\mathbf{A},0} and \bm{\Sigma}_{\mathbf{B},0} are non-negative diagonal matrices with \lVert\bm{\Sigma}_{\mathbf{A},0}\rVert_{2},\lVert\bm{\Sigma}_{\mathbf{B},0}\rVert_{2}<\sqrt{\sigma_{\mathrm{min}}(\mathbf{Y})} and \mathbf{Q}\in\mathbb{R}^{r\times r} is any orthogonal matrix. Then, for all t\geq 0 , the SpecGF iteration with \mathcal{T} can be written as \mathbf{A}(t)=\mathbf{U}_{r}\bm{\Sigma}_{A}(t)\mathbf{Q}^{\top},\quad\mathbf{B}(t)=\mathbf{Q}\bm{\Sigma}_{B}(t)\mathbf{V}_{r}^{\top} with \bm{\Sigma}_{A}(0)=\bm{\Sigma}_{\mathbf{A},0} and \bm{\Sigma}_{B}(0)=\bm{\Sigma}_{\mathbf{B},0} . Moreover, \bm{\Sigma}_{A}(t)\bm{\Sigma}_{B}(t)\to\bm{\Sigma}_{r} and \dot{\bm{\Sigma}}_{A}(t)=\dot{\bm{\Sigma}}_{B}(t)=\mathbf{I}_{r} , where \bm{\Sigma}_{r}\in\mathbb{R}^{r\times r} is the leading principle submatrix of \bm{\Sigma} , unless \mathbf{A}(0) and \mathbf{B}(0) do not have zero diagonal values at the same coordinate. Proof. Let \sigma_{\mathbf{A},i}(t) and \sigma_{\mathbf{B},i}(t) be the i -th singular value of \bm{\Sigma}_{\mathbf{A}}(t) and \bm{\Sigma}_{\mathbf{B}}(t) , respectively. Similarly, denote the i -th singular value of \mathbf{S} as \sigma_{i} . If \mathbf{A}(t)=\mathbf{U}_{r}\bm{\Sigma}_{A}(t)\mathbf{Q}^{\top},\quad\mathbf{B}(t)=\mathbf{Q}\bm{\Sigma}_{B}(t)\mathbf{V}_{r}^{\top}, then \displaystyle\nabla_{\mathbf{A}}\mathcal{L}(t) \displaystyle=(\mathbf{A}(t)\mathbf{B}(t)-\mathbf{Y})\mathbf{B}(t)^{\top}=\mathbf{U}_{r}(\bm{\Sigma}_{A}(t)\bm{\Sigma}_{B}(t)-\bm{\Sigma}_{r})\bm{\Sigma}_{B}(t)^{\top}\mathbf{Q}^{\top}, \displaystyle\nabla_{\mathbf{B}}\mathcal{L}(t) \displaystyle=\mathbf{A}(t)^{\top}(\mathbf{A}(t)\mathbf{B}(t)-\mathbf{Y})=\mathbf{Q}\bm{\Sigma}_{A}(t)^{\top}(\bm{\Sigma}_{A}(t)\bm{\Sigma}_{B}(t)-\bm{\Sigma}_{r})\mathbf{V}_{r}^{\top}. Thus, \displaystyle\mathcal{T}(\nabla_{\mathbf{A}}\mathcal{L}(t)) \displaystyle=\mathbf{U}_{r}\bm{\Gamma}_{A}(t)\mathbf{Q}^{\top}=\mathbf{U}_{r}\mathrm{Diag}\{\mathrm{sgn}((\sigma_{\mathbf{A},i}(t)\sigma_{\mathbf{B},i}(t)-\sigma_{i})\sigma_{\mathbf{B},i}(t))\}\mathbf{Q}^{\top}, \displaystyle\mathcal{T}(\nabla_{\mathbf{B}}\mathcal{L}(t)) \displaystyle=\mathbf{Q}\bm{\Gamma}_{B}(t)\mathbf{V}_{r}^{\top}=\mathbf{Q}\mathrm{Diag}\{\mathrm{sgn}((\sigma_{\mathbf{A},i}(t)\sigma_{\mathbf{B},i}(t)-\sigma_{i})\sigma_{\mathbf{A},i}(t))\}\mathbf{V}_{r}^{\top}, for i\in[r] , where \bm{\Gamma}_{A}(t) and \bm{\Gamma}_{B}(t) are the rectangular diagonal matrices. Hence, only the singular values of \mathbf{A}(t) and \mathbf{B}(t) are updated, and we can write \bm{\Sigma}_{A}(0)=\bm{\Sigma}_{\mathbf{A},0},\quad\bm{\Sigma}_{B}(0)=\bm{\Sigma}_{\mathbf{B},0}. Since the initialization is small, we see that all the signs in \bm{\Gamma}_{A}(t) and \bm{\Gamma}_{B}(t) are negative before saturation, or \sigma_{\mathbf{A},i}(t)\sigma_{\mathbf{B},i}(t)=\sigma_{i} . Therefore, all the singular values have a speed of 1 until the saturation, and the solution is \displaystyle\bm{\Sigma}_{\mathbf{A}}(t) \displaystyle=\mathrm{Diag}\left\{\sigma_{\mathbf{A},i}(0)+\max\{t,T_{i}\}\right\}_{i} \displaystyle\bm{\Sigma}_{\mathbf{B}}(t) \displaystyle=\mathrm{Diag}\left\{\sigma_{\mathbf{B},i}(0)+\max\{t,T_{i}\}\right\}_{i}, where for each i , T_{i}>0 satisfies \left(\sigma_{\mathbf{A},i}(0)+T_{i}\right)\left(\sigma_{\mathbf{B},i}(0)+T_{i}\right)=\sigma_{i}.\qed This implies that if \bm{\Sigma} is already in non-increasing order, then SpecGF with \mathcal{T} converges to the best rank- r approximation of \mathbf{Y} . We have a similar result for SpecGF with \mathcal{T}_{\beta} . Proposition A.14 . Let \mathbf{A}(0)=\mathbf{U}_{r}\bm{\Sigma}_{\mathbf{A},0}\mathbf{Q}^{\top},\quad\mathbf{B}(0)=\mathbf{Q}\bm{\Sigma}_{\mathbf{B},0}\mathbf{V}_{r}^{\top}, where \bm{\Sigma}_{\mathbf{A},0} and \bm{\Sigma}_{\mathbf{B},0} are positive invertible diagonal matrices with \lVert\bm{\Sigma}_{\mathbf{A},0}\rVert_{2},\lVert\bm{\Sigma}_{\mathbf{B},0}\rVert_{2}<\gamma<\sqrt{\sigma_{\mathrm{min}}(\mathbf{Y})} for \gamma>0 and \mathbf{Q}\in\mathbb{R}^{r\times r} is any orthogonal matrix. Let \sigma_{\mathbf{A},i} and \sigma_{\mathbf{B},i} be the i -th singular value of \bm{\Sigma}_{\mathbf{A},0} and \bm{\Sigma}_{\mathbf{B},0} , respectively. Assume that for all i\in[r] , \quad\left\lvert\log\frac{\sigma_{\mathbf{A},i}}{\sigma_{\mathbf{B},i}}\right\rvert<\gamma. Then, for all t\geq 0 , the SpecGF iteration with \mathcal{T}_{\beta} can be written as \mathbf{A}(t)=\mathbf{U}_{r}\bm{\Sigma}_{A}(t)\mathbf{Q}^{\top},\quad\mathbf{B}(t)=\mathbf{Q}\bm{\Sigma}_{B}(t)\mathbf{V}_{r}^{\top} with \bm{\Sigma}_{A}(0)=\bm{\Sigma}_{\mathbf{A},0} and \bm{\Sigma}_{B}(0)=\bm{\Sigma}_{\mathbf{B},0} . Moreover, \bm{\Sigma}_{A}(t)\bm{\Sigma}_{B}(t)\to\bm{\Sigma}_{r} , where \bm{\Sigma}_{r}\in\mathbb{R}^{r\times r} is the leading principle submatrix of \bm{\Sigma} , unless \mathbf{A}(0) and \mathbf{B}(0) do not have zero diagonal values at the same coordinate. In addition, both \left\lvert\bm{\Sigma}_{A}(t)-\bm{\Sigma}_{B}(t)\right\rvert and \left\lvert\dot{\bm{\Sigma}}_{A}(t)-\dot{\bm{\Sigma}}_{B}(t)\right\rvert go to 0 as \gamma\to 0 . Proof. We note that the dynamics exist uniquely (from analyticity of \mathcal{T}_{\beta} ) and globally (from boundedness of velocity by \mathcal{T}_{\beta} ). Let \sigma_{\mathbf{A},i}(t) and \sigma_{\mathbf{B},i}(t) be the i -th singular value of \bm{\Sigma}_{\mathbf{A}}(t) and \bm{\Sigma}_{\mathbf{B},}(t) , respectively. Denote the i -th singular value of \mathbf{S} as \sigma_{i} . If \mathbf{A}(t)=\mathbf{U}_{r}\bm{\Sigma}_{A}(t)\mathbf{Q}^{\top},\quad\mathbf{B}(t)=\mathbf{Q}\bm{\Sigma}_{B}(t)\mathbf{V}_{r}^{\top}, then \displaystyle\nabla_{\mathbf{A}}\mathcal{L}(t) \displaystyle=(\mathbf{A}(t)\mathbf{B}(t)-\mathbf{Y})\mathbf{B}(t)^{\top}=\mathbf{U}_{r}(\bm{\Sigma}_{A}(t)\bm{\Sigma}_{B}(t)-\bm{\Sigma}_{r})\bm{\Sigma}_{B}(t)^{\top}\mathbf{Q}^{\top}, \displaystyle\nabla_{\mathbf{B}}\mathcal{L}(t) \displaystyle=\mathbf{A}(t)^{\top}(\mathbf{A}(t)\mathbf{B}(t)-\mathbf{Y})=\mathbf{Q}\bm{\Sigma}_{A}(t)^{\top}(\bm{\Sigma}_{A}(t)\bm{\Sigma}_{B}(t)-\bm{\Sigma}_{r})\mathbf{V}_{r}^{\top}. Thus, \displaystyle\mathcal{T}_{\beta}(\nabla_{\mathbf{A}}\mathcal{L}(t)) \displaystyle=\mathbf{U}_{r}\bm{\Gamma}_{A}(t)\mathbf{Q}^{\top}=\mathbf{U}_{r}\mathrm{Diag}\{f_{\beta}((\sigma_{\mathbf{A},i}(t)\sigma_{\mathbf{B},i}(t)-\sigma_{i})\sigma_{\mathbf{B},i}(t))\}\mathbf{Q}^{\top}, \displaystyle\mathcal{T}_{\beta}(\nabla_{\mathbf{B}}\mathcal{L}(t)) \displaystyle=\mathbf{Q}\bm{\Gamma}_{B}(t)\mathbf{V}_{r}^{\top}=\mathbf{Q}\mathrm{Diag}\{f_{\beta}((\sigma_{\mathbf{A},i}(t)\sigma_{\mathbf{B},i}(t)-\sigma_{i})\sigma_{\mathbf{A},i}(t))\}\mathbf{V}_{r}^{\top}, for i\in[r] , where \bm{\Gamma}_{A}(t) and \bm{\Gamma}_{B}(t) are the rectangular diagonal matrices and f_{\beta}(x)\triangleq\frac{x}{\sqrt{x^{2}+\beta}}. Hence, only the singular values of \mathbf{A}(t) and \mathbf{B}(t) are updated, and we can write \bm{\Sigma}_{A}(0)=\bm{\Sigma}_{\mathbf{A},0},\quad\bm{\Sigma}_{B}(0)=\bm{\Sigma}_{\mathbf{B},0}. Notice that the overall dynamics are decoupled into r different scalar dynamics: \displaystyle\dot{\sigma}_{\mathbf{A},i}(t) \displaystyle=-f_{\beta}((\sigma_{\mathbf{A},i}(t)\sigma_{\mathbf{B},i}(t)-\sigma_{i})\sigma_{\mathbf{B},i}(t)), \displaystyle\dot{\sigma}_{\mathbf{B},i}(t) \displaystyle=-f_{\beta}((\sigma_{\mathbf{A},i}(t)\sigma_{\mathbf{B},i}(t)-\sigma_{i})\sigma_{\mathbf{A},i}(t)). So it suffices to demonstrate about one i\in[r] ; for simplicity, we fix i and simplify the notation: \sigma_{\mathbf{A},i}(t)\triangleq a(t),\quad\sigma_{\mathbf{B},i}(t)\triangleq b(t),\quad\sigma_{i}\triangleq\sigma and rewrite the dynamics: \displaystyle\dot{a}(t) \displaystyle=-f_{\beta}((a(t)b(t)-\sigma)b(t)), \displaystyle\dot{b}(t) \displaystyle=-f_{\beta}((a(t)b(t)-\sigma)a(t)). We first note an immediate consequence from the dynamics: a(t)\geq 0,b(t)\geq 0 always hold. If it were violated, then a and b must be both zero simultaneously, but it implies that the dynamics terminates. Another consequence is that a(t)b(t)\leq\sigma and non-decreasing from the update of a(t)b(t) itself: \frac{\mathrm{d}}{\mathrm{d}t}a(t)b(t)=-(a(t)b(t)-\sigma)\left(\frac{a(t)^{2}}{\sqrt{(a(t)b(t)-\sigma)^{2}a(t)^{2}+\beta}}+\frac{b(t)^{2}}{\sqrt{(a(t)b(t)-\sigma)^{2}b(t)^{2}+\beta}}\right). It is straightforward that a(t)b(t) is non-decreasing as long as a(t)b(t)\leq\sigma . Thus, there exists \sigma_{\infty}\in[0,\sigma] such that a(t)b(t)\to\sigma_{\infty} . Moreover, a(t) and b(t) are strictly increasing on t>0 as long as a(t)b(t)<\sigma ; the problematic case is t=0 , but, without loss of generality, if a(0)=0 then b(0)>0 thus \dot{a}(0)>0 . Hence, a(t),b(t)>0 for all t>0 , which yields strictly monotonic evolution. Therefore, both a(t) and b(t) are bounded and attain their respective forward limit, namely a_{\infty}>0 and b_{\infty}>0 . Then a_{\infty}b_{\infty}=\sigma_{\infty}>0 with \lim_{t\to\infty}\dot{a}(t)=-f_{\beta}((\sigma_{\infty}-\sigma)b_{\infty})=0 since f_{\beta}^{\prime\prime}(x) is bounded, which implies \sigma_{\infty}=\sigma . Albeit the global convergence is proved, a closed form for the solution is not generally obtained. As a roundabout, we show that if a(0),b(0)\leq\gamma then a(t)\approx b(t) and \dot{a}(t)\approx\dot{b}(t) as \gamma\to 0 . Let d(t)\triangleq a(t)-b(t) . Then \displaystyle\dot{d}(t) \displaystyle=f_{\beta}((a(t)b(t)-\sigma)a(t))-f_{\beta}((a(t)b(t)-\sigma)b(t)) \displaystyle\leq\frac{1}{\sqrt{\beta}}(a(t)b(t)-\sigma)(a(t)-b(t)) \displaystyle=\frac{1}{\sqrt{\beta}}(a(t)b(t)-\sigma)d(t). The inequality holds as f_{\beta} is \frac{1}{\sqrt{\beta}} -Lipschitz. Since a(t)b(t)\leq\sigma for all t , |d(t)|\leq|d(0)|\leq\gamma . We now bound v(t)\triangleq\dot{a}(t)-\dot{b}(t) . From above, |\dot{v}(t)|\leq\frac{1}{\sqrt{\beta}}|(a(t)b(t)-\sigma)d(t)|\leq\frac{\sigma}{\sqrt{\beta}}\gamma.\qed Appendix B Uniform Growth of Singular Values This section proves the uniform growth results stated in Theorem 5.7 . B.1 Setup and Notation \beta -regularized orthogonalization. Fix \beta>0 . For \mathbf{G}\in\mathbb{R}^{m\times r} define \displaystyle\mathcal{T}_{\mathrm{col},\beta}(\mathbf{G})\coloneqq\mathbf{G}(\mathbf{G}^{\top}\mathbf{G}+\beta\mathbf{I})^{-1/2}, and for \mathbf{G}\in\mathbb{R}^{r\times n} define \displaystyle\mathcal{T}_{\mathrm{row},\beta}(\mathbf{G})\coloneqq(\mathbf{G}\mathbf{G}^{\top}+\beta\mathbf{I})^{-1/2}\mathbf{G}. The case \beta=0 (unregularized SpecGF) can be handled by a limiting argument; we focus on \beta>0 throughout. Lemma B.1 (Operator norm bound) . For all \beta>0 and all \mathbf{G} , \displaystyle\lVert\mathcal{T}_{\mathrm{col},\beta}(\mathbf{G})\rVert_{2}\leq 1,\qquad\lVert\mathcal{T}_{\mathrm{row},\beta}(\mathbf{G})\rVert_{2}\leq 1. Proof. Let s be any singular value of \mathbf{G} . Then s becomes s/\sqrt{s^{2}+\beta}\leq 1 under either normalization, hence the operator norm is \leq 1 . ∎ Lemma B.2 (Invariance) . If \mathbf{A}(0)=\mathbf{0} , then \mathbf{A}(t)\in\mathrm{col}(\mathbf{U}_{r}) for all t\geq 0 . Consequently, \mathbf{A}(t)=\mathbf{U}_{r}\mathbf{X}(t) for some \mathbf{X}(t)\in\mathbb{R}^{r\times r} . Proof. The gradient with respect to \mathbf{A} is \nabla_{\mathbf{A}}\mathcal{L}=(\mathbf{A}\mathbf{B}-\mathbf{Y})\mathbf{B}^{\top} . At t=0 , we have \nabla_{\mathbf{A}}\mathcal{L}(0)=-\mathbf{Y}\mathbf{B}(0)^{\top}=-\mathbf{U}_{r}\bm{\Sigma}\mathbf{V}_{r}^{\top}\mathbf{B}(0)^{\top} , which lies in \mathrm{col}(\mathbf{U}_{r}) . Since \mathcal{T}_{\beta} (and \mathcal{T} ) preserves the column space, \dot{\mathbf{A}}(0)\in\mathrm{col}(\mathbf{U}_{r}) . By induction, suppose \mathbf{A}(t)\in\mathrm{col}(\mathbf{U}_{r}) , i.e., \mathbf{A}(t)=\mathbf{U}_{r}\mathbf{X}(t) for some \mathbf{X}(t)\in\mathbb{R}^{r\times r} . Then \displaystyle\nabla_{\mathbf{A}}\mathcal{L}(t) \displaystyle=(\mathbf{U}_{r}\mathbf{X}(t)\mathbf{B}(t)-\mathbf{U}_{r}\bm{\Sigma}\mathbf{V}_{r}^{\top})\mathbf{B}(t)^{\top}=\mathbf{U}_{r}(\mathbf{X}(t)\mathbf{B}(t)\mathbf{B}(t)^{\top}-\bm{\Sigma}\mathbf{V}_{r}^{\top}\mathbf{B}(t)^{\top}). This again lies in \mathrm{col}(\mathbf{U}_{r}) , so \dot{\mathbf{A}}(t)=-\mathcal{T}_{\beta}(\nabla_{\mathbf{A}}\mathcal{L}(t))\in\mathrm{col}(\mathbf{U}_{r}) . By continuity of the flow, \mathbf{A}(t)\in\mathrm{col}(\mathbf{U}_{r}) for all t\geq 0 . ∎ Core variables. Let \mathbf{Y}=\mathbf{U}_{r}\bm{\Sigma}\mathbf{V}_{r}^{\top} be the compact SVD with \bm{\Sigma}=\mathrm{Diag}(\sigma_{1},\ldots,\sigma_{r}) and \sigma_{1}>\cdots>\sigma_{r}>0 , where \mathbf{U}_{r}\in\mathbb{R}^{m\times r} and \mathbf{V}_{r}\in\mathbb{R}^{n\times r} . Let \mathbf{V}_{\perp}\in\mathbb{R}^{n\times(n-r)} denote the orthogonal complement of \mathbf{V}_{r} . We define the core variables: \displaystyle\mathbf{X}(t)\coloneqq\mathbf{U}_{r}^{\top}\mathbf{A}(t)\in\mathbb{R}^{r\times r},\quad\mathbf{Z}(t)\coloneqq\mathbf{B}(t)\mathbf{V}_{r}\in\mathbb{R}^{r\times r},\quad\mathbf{Z}_{\perp}(t)\coloneqq\mathbf{B}(t)\mathbf{V}_{\perp}\in\mathbb{R}^{r\times(n-r)}. The core product is \mathbf{G}(t)\coloneqq\mathbf{X}(t)\mathbf{Z}(t) . For a square matrix \mathbf{M} , we write \mathrm{Off}(\mathbf{M})\coloneqq\mathbf{M}-\mathrm{Diag}(\mathrm{diag}(\mathbf{M})) for its off-diagonal part. We introduce two tolerances: • Alignment tolerance \delta>0 : Time t is \delta -aligned if \left\lVert\mathrm{Off}(\mathbf{G}(t))\right\rVert_{\mathrm{F}}\leq\delta and \left\lVert\mathbf{X}(t)\mathbf{Z}_{\perp}(t)\right\rVert_{\mathrm{F}}\leq\delta . • Target tolerance \varepsilon>0 : We denote d_{i}(t)\coloneqq[\mathbf{G}(t)]_{ii} , e_{i}(t)\coloneqq d_{i}(t)-\sigma_{i} , and the active set \mathcal{I}_{\varepsilon}(t)\coloneqq\{i\in[r]:|e_{i}(t)|>\varepsilon\} . A mode i is active if i\in\mathcal{I}_{\varepsilon}(t) . Let T\coloneqq\inf\{t\geq\tau:\mathcal{I}_{\varepsilon}(t)=\emptyset\} be the termination time. B.2 Alignment from Small Initialization Proposition B.3 (High-probability alignment) . Let T be a termination time above. Under Gaussian initialization \mathbf{A}(0)=\mathbf{0} , \mathbf{B}(0)=\gamma\mathbf{N} with N_{ij}\stackrel{{\scriptstyle\mathrm{i.i.d.}}}{{\sim}}\mathcal{N}(0,1) , there exist constants C_{1},C_{2}<\infty (depending on \mathbf{Y},r,n,T ) such that, with probability at least 1-\exp(-c_{1}r^{2})-\exp(-c_{1}rn) for some constant c_{1}>0 , \displaystyle\sup_{t\leq T}\left\lVert\mathrm{Off}(\mathbf{G}(t))\right\rVert_{\mathrm{F}}\leq C_{1}\gamma,\qquad\sup_{t\leq T}\left\lVert\mathbf{X}(t)\mathbf{Z}_{\perp}(t)\right\rVert_{\mathrm{F}}\leq C_{2}\gamma. Consequently, the trajectory is O(\gamma) -aligned for all t\leq T . Proof. Both bounds follow from Lipschitz stability near the diagonal manifold \mathcal{M}\coloneqq\{(\mathbf{X},\mathbf{Z},\mathbf{Z}_{\perp}):\mathbf{X},\mathbf{Z}\text{ diagonal},\mathbf{Z}_{\perp}=0\} . Let \mathbf{S}(t)\coloneqq(\mathbf{X}(t),\mathbf{Z}(t),\mathbf{Z}_{\perp}(t)) and consider the orthogonal projection onto \mathcal{M} : \displaystyle\Pi(\mathbf{X},\mathbf{Z},\mathbf{Z}_{\perp})\coloneqq\big(\mathrm{Diag}(\mathrm{diag}(\mathbf{X})),\mathrm{Diag}(\mathrm{diag}(\mathbf{Z})),\mathbf{0}\big). Let \mathbf{S}^{\star}(t)\coloneqq(\mathbf{X}^{\star}(t),\mathbf{Z}^{\star}(t),\mathbf{Z}^{\star}_{\perp}(t)) be the trajectory of SpecGF with \mathcal{T}_{\beta} trajectory, started at \mathbf{S}^{\star}(0)\coloneqq\Pi(\mathbf{S}(0)) . By diagonal manifold invariance, \mathbf{S}^{\star}(t)\in\mathcal{M} for all t , hence \mathbf{Z}^{\star}_{\perp}(t)\equiv\mathbf{0} and, writing \mathbf{G}^{\star}(t)\coloneqq\mathbf{X}^{\star}(t)\mathbf{Z}^{\star}(t) , \mathrm{Off}(\mathbf{G}^{\star}(t))\equiv\mathbf{0} . By speed bounds and Gaussian concentration, both trajectories remain in a compact set \mathcal{K} for t\in[0,T] . Since SpecGF with \mathcal{T}_{\beta} defines a \mathcal{C}^{1} vector field, finite-horizon Lipschitz stability gives \displaystyle\left\lVert\mathbf{S}(t)-\mathbf{S}^{\star}(t)\right\rVert_{\mathrm{F}}\leq e^{LT}\left\lVert\mathbf{S}(0)-\mathbf{S}^{\star}(0)\right\rVert_{\mathrm{F}} for some finite L=L(\mathcal{K}) . Note that \left\lVert\mathbf{S}(0)-\mathbf{S}^{\star}(0)\right\rVert_{\mathrm{F}}\leq\left\lVert\mathrm{Off}(\mathbf{Z}(0))\right\rVert_{\mathrm{F}}+\left\lVert\mathbf{Z}_{\perp}(0)\right\rVert_{\mathrm{F}} because \mathbf{X}(0)=\mathbf{0} . Since \mathbf{Z}(0)=\gamma\mathbf{N}\mathbf{V}_{r} and \mathbf{Z}_{\perp}(0)=\gamma\mathbf{N}\mathbf{V}_{\perp} where \mathbf{N} has i.i.d. \mathcal{N}(0,1) entries, and \mathbf{V}_{r} , \mathbf{V}_{\perp} have orthonormal columns, the entries of \mathbf{N}\mathbf{V}_{r} and \mathbf{N}\mathbf{V}_{\perp} are also i.i.d. \mathcal{N}(0,1) . Thus \left\lVert\mathrm{Off}(\mathbf{N}\mathbf{V}_{r})\right\rVert_{\mathrm{F}}^{2}\sim\chi^{2}_{r(r-1)} and \left\lVert\mathbf{N}\mathbf{V}_{\perp}\right\rVert_{\mathrm{F}}^{2}\sim\chi^{2}_{r(n-r)} . By chi-squared concentration, \displaystyle\mathbb{P}\big(\left\lVert\mathrm{Off}(\mathbf{Z}(0))\right\rVert_{\mathrm{F}}+\left\lVert\mathbf{Z}_{\perp}(0)\right\rVert_{\mathrm{F}}>C\gamma(r+\sqrt{rn})\big)\leq\exp(-c_{1}r^{2})+\exp(-c_{1}rn) for some absolute constants C,c_{1}>0 . Consequently, \sup_{t\leq T}\left\lVert\mathbf{S}(t)-\mathbf{S}^{\star}(t)\right\rVert_{\mathrm{F}}\leq\kappa\gamma for some finite \kappa>0 with probability at least 1-\exp(-c_{1}r^{2})-\exp(-c_{1}rn) . Since \mathbf{Z}^{\star}_{\perp}(t)\equiv\mathbf{0} , it follows that \displaystyle\left\lVert\mathbf{Z}_{\perp}(t)\right\rVert_{\mathrm{F}}=\left\lVert\mathbf{Z}_{\perp}(t)-\mathbf{Z}^{\star}_{\perp}(t)\right\rVert_{\mathrm{F}}\leq\left\lVert\mathbf{S}(t)-\mathbf{S}^{\star}(t)\right\rVert_{\mathrm{F}}\leq\kappa\gamma. By speed bounds \lVert\mathbf{X}(t)\rVert_{2}\leq t\leq T , hence \sup_{t\leq T}\left\lVert\mathbf{X}(t)\mathbf{Z}_{\perp}(t)\right\rVert_{\mathrm{F}}\leq T\kappa\gamma\coloneqq C_{2}\gamma . For the off-diagonal bound, using \mathbf{G}=\mathbf{X}\mathbf{Z} and \mathbf{G}^{\star}=\mathbf{X}^{\star}\mathbf{Z}^{\star} , we have \displaystyle\mathbf{G}-\mathbf{G}^{\star}=\mathbf{X}\mathbf{Z}-\mathbf{X}^{\star}\mathbf{Z}^{\star}=(\mathbf{X}-\mathbf{X}^{\star})\mathbf{Z}+\mathbf{X}^{\star}(\mathbf{Z}-\mathbf{Z}^{\star}). Since both trajectories remain in the compact set \mathcal{K} for t\in[0,T] , there exists M=M(\mathcal{K})<\infty such that \|\mathbf{Z}(t)\|_{2},\|\mathbf{X}^{\star}(t)\|_{2}\leq M . Therefore, we have \displaystyle\left\lVert\mathbf{G}(t)-\mathbf{G}^{\star}(t)\right\rVert_{\mathrm{F}} \displaystyle\leq\left\lVert\mathbf{X}(t)-\mathbf{X}^{\star}(t)\right\rVert_{\mathrm{F}}\lVert\mathbf{Z}(t)\rVert_{2}+\lVert\mathbf{X}^{\star}(t)\rVert_{2}\left\lVert\mathbf{Z}(t)-\mathbf{Z}^{\star}(t)\right\rVert_{\mathrm{F}} \displaystyle\leq M\big(\left\lVert\mathbf{X}(t)-\mathbf{X}^{\star}(t)\right\rVert_{\mathrm{F}}+\left\lVert\mathbf{Z}(t)-\mathbf{Z}^{\star}(t)\right\rVert_{\mathrm{F}}\big) \displaystyle\leq M\sqrt{2}\left\lVert\mathbf{S}(t)-\mathbf{S}^{\star}(t)\right\rVert_{\mathrm{F}}. Since \mathrm{Off}(\mathbf{G}^{\star}(t))\equiv\mathbf{0} , we obtain \displaystyle\left\lVert\mathrm{Off}(\mathbf{G}(t))\right\rVert_{\mathrm{F}}=\left\lVert\mathrm{Off}(\mathbf{G}(t)-\mathbf{G}^{\star}(t))\right\rVert_{\mathrm{F}}\leq\left\lVert\mathbf{G}(t)-\mathbf{G}^{\star}(t)\right\rVert_{\mathrm{F}}\leq M\sqrt{2}\left\lVert\mathbf{S}(t)-\mathbf{S}^{\star}(t)\right\rVert_{\mathrm{F}}. Combining these bounds yields \sup_{t\leq T}\left\lVert\mathrm{Off}(\mathbf{G}(t))\right\rVert_{\mathrm{F}}\leq C_{1}\gamma . ∎ Lemma B.4 (Diagonal approximation) . Suppose \left\lVert\mathrm{Off}(\mathbf{G}(t))\right\rVert_{\mathrm{F}}\leq\delta and \left\lVert\mathbf{X}(t)\mathbf{Z}_{\perp}(t)\right\rVert_{\mathrm{F}}\leq\delta . Then there exists a permutation \pi(t) such that \displaystyle|\sigma_{\pi_{i}(t)}(\mathbf{A}(t)\mathbf{B}(t))-d_{i}(t)|=O(\delta) for all i\in[r] , where \pi_{i}(t) denotes the index satisfying d_{i}(t)\approx\sigma_{\pi_{i}(t)}(\mathbf{A}(t)\mathbf{B}(t)) . Proof. We have \displaystyle\mathbf{A}\mathbf{B}=\mathbf{U}_{r}\mathbf{X}\mathbf{Z}\mathbf{V}_{r}^{\top}+\mathbf{U}_{r}\mathbf{X}\mathbf{Z}_{\perp}\mathbf{V}_{\perp}^{\top}. Since \mathbf{U}_{r},\mathbf{V}_{r},\mathbf{V}_{\perp} have orthonormal columns, the singular values of \mathbf{A}\mathbf{B} equal those of [\mathbf{G}\mid\mathbf{X}\mathbf{Z}_{\perp}] . We write \mathbf{D}=\mathrm{Diag}(d_{1},\ldots,d_{r}) . The singular values of [\mathbf{D}\mid\mathbf{0}] are exactly \{|d_{1}|,\ldots,|d_{r}|\} , and \displaystyle\|[\mathbf{G}\mid\mathbf{X}\mathbf{Z}_{\perp}]-[\mathbf{D}\mid\mathbf{0}]\|_{F}\leq\|\mathrm{Off}(\mathbf{G})\|_{F}+\|\mathbf{X}\mathbf{Z}_{\perp}\|_{F}\leq 2\delta. By Weyl’s inequality (or Mirsky), there exists a permutation \pi of [r] such that \displaystyle|\sigma_{\pi(i)}([\mathbf{G}\mid\mathbf{X}\mathbf{Z}_{\perp}])-|d_{i}||\leq 2\delta\quad\text{for all }i\in[r]. Since d_{i}>0 on our trajectory, we conclude |\sigma_{\pi(i)}(\mathbf{A}\mathbf{B})-d_{i}|=O(\delta) . ∎ B.3 Initial Growth and Non-degeneracy Lemma B.5 (Initial growth) . Let \mathbf{Z}(0)=\gamma\mathbf{M} where \mathbf{M}\coloneqq\mathbf{N}\mathbf{V}_{r}\in\mathbb{R}^{r\times r} , with \mathbf{N} having i.i.d. \mathcal{N}(0,1) entries and \beta=O(\gamma^{2}) . Write \mathbf{M}=[\mathbf{c}_{1}\mid\cdots\mid\mathbf{c}_{r}] where \mathbf{c}_{i}\in\mathbb{R}^{r} denotes the i -th column. Define the high-probability event \displaystyle\mathcal{E}\coloneqq\left\{\|\mathbf{M}\|_{2}\leq C_{3}\sqrt{r},\quad\min_{i\in[r]}\|\mathbf{c}_{i}\|^{2}\geq c_{1}r\right\}. Then \mathbb{P}(\mathcal{E})\geq 1-e^{-c_{1}r} for some absolute constants C_{3},c_{1}>0 . Let \tau\asymp\frac{\sigma_{r}}{\sigma_{1}\sqrt{r}} . On the event \mathcal{E} , \displaystyle d_{\min}(\tau)\geq c_{2}\tau\gamma for some constant c_{2}>0 . Moreover, all modes are active at \tau : \mathcal{I}_{\varepsilon}(\tau)=[r] and d_{i}(\tau)<\sigma_{i} for all i . Proof. For an r\times r Gaussian matrix \mathbf{M} , standard concentration gives \|\mathbf{M}\|_{2}\leq 3\sqrt{r} with probability at least 1-2e^{-r/2} . Each column \mathbf{c}_{i}\sim\mathcal{N}(0,\mathbf{I}_{r}) satisfies \|\mathbf{c}_{i}\|^{2}\sim\chi^{2}_{r} , so \mathbb{P}(\|\mathbf{c}_{i}\|^{2}<\frac{r}{2})\leq e^{-c_{1}r} for some absolute constant c_{1}>0 . By a union bound, \mathbb{P}(\mathcal{E})\geq 1-e^{-c_{1}r} . We now bound [\dot{\mathbf{X}}(0)\mathbf{Z}(0)]_{ii} on \mathcal{E} ; note that \dot{\mathbf{Z}}(0)=\bm{0} . From the dynamics of SpecGF with \mathcal{T}_{\beta} at t=0 with \mathbf{A}(0)=\mathbf{0} and \mathbf{Z}(0)=\gamma\mathbf{M} : \displaystyle\dot{\mathbf{X}}(0)=\mathcal{T}_{\mathrm{col},\beta}(\gamma\bm{\Sigma}\mathbf{M}^{\top})=\gamma\bm{\Sigma}\mathbf{M}^{\top}(\gamma^{2}\mathbf{M}\bm{\Sigma}^{2}\mathbf{M}^{\top}+\beta\mathbf{I})^{-1/2}. Define the positive definite matrix \mathbf{Q}\coloneqq\gamma^{2}\mathbf{M}\bm{\Sigma}^{2}\mathbf{M}^{\top}+\beta\mathbf{I} . Then \displaystyle\dot{\mathbf{X}}(0)\mathbf{Z}(0)=\gamma^{2}\bm{\Sigma}\mathbf{M}^{\top}\mathbf{Q}^{-1/2}\mathbf{M}, whose i -th diagonal entry equals \gamma^{2}\sigma_{i}\cdot\mathbf{c}_{i}^{\top}\mathbf{Q}^{-1/2}\mathbf{c}_{i} . Since \mathbf{Q}^{-1/2} is positive definite with minimum eigenvalue \lambda_{\max}(\mathbf{Q})^{-1/2} , \displaystyle\mathbf{c}_{i}^{\top}\mathbf{Q}^{-1/2}\mathbf{c}_{i}\geq\frac{\|\mathbf{c}_{i}\|^{2}}{\sqrt{\lambda_{\max}(\mathbf{Q})}}. On \mathcal{E} , we have \lambda_{\max}(\mathbf{Q})\leq\gamma^{2}\sigma_{1}^{2}\|\mathbf{M}\|_{2}^{2}+\beta\leq C_{3}^{2}\gamma^{2}\sigma_{1}^{2}r+\beta . Assuming \beta\leq C\gamma^{2}\sigma_{1}^{2}r for some absolute constant C>0 , this gives \lambda_{\max}(\mathbf{Q})\leq C^{\prime}\gamma^{2}\sigma_{1}^{2}r , and hence \displaystyle\sqrt{\lambda_{\max}(\mathbf{Q})}\leq\sqrt{C^{\prime}}\cdot\gamma\sigma_{1}\sqrt{r}. Combined with \|\mathbf{c}_{i}\|^{2}\geq c_{1}r , we obtain \displaystyle[\dot{\mathbf{X}}(0)\mathbf{Z}(0)]_{ii}\geq\gamma^{2}\sigma_{r}\cdot\frac{c_{1}r}{\sqrt{C^{\prime}}\cdot\gamma\sigma_{1}\sqrt{r}}=\frac{c_{1}\gamma\sigma_{r}\sqrt{r}}{\sqrt{C^{\prime}}\sigma_{1}}. Finally, we apply Taylor expansion. For \beta>0 , the vector field is C^{1} , so for sufficiently small \tau : \displaystyle\mathbf{X}(\tau)=\tau\dot{\mathbf{X}}(0)+O(\tau^{2}),\qquad\mathbf{Z}(\tau)=\mathbf{Z}(0)+O(\tau^{2}), where the second equality uses \dot{\mathbf{Z}}(0)=\mathbf{0} (because \mathbf{X}(0)=\mathbf{0} ). Expanding \mathbf{G}(\tau)=\mathbf{X}(\tau)\mathbf{Z}(\tau) , \displaystyle d_{\min}(\tau)\geq\tau\cdot\min_{i}[\dot{\mathbf{X}}(0)\mathbf{Z}(0)]_{ii}-O(\tau^{2}\gamma)\asymp\tau\gamma. For the second claim, note that d_{i}(\tau)\asymp\tau\gamma\ll\sigma_{r}-\varepsilon for sufficiently small \gamma . Hence, it holds that e_{i}(\tau)=d_{i}(\tau)-\sigma_{i}<-\varepsilon for all i\in[r] , so all modes are active (i.e., early phase of training). ∎ B.4 Invertibility and Near-Diagonal Factors Lemma B.6 (Invertibility under alignment and non-degeneracy) . Suppose \left\lVert\mathrm{Off}(\mathbf{G}(t))\right\rVert_{\mathrm{F}}\leq\delta and d_{\min}(t)>0 . If \delta0 , the diagonal matrix \mathbf{D} is invertible. Write \mathbf{G}=\mathbf{D}(\mathbf{I}+\mathbf{D}^{-1}\mathrm{Off}(\mathbf{G})) . We have \displaystyle\|\mathbf{D}^{-1}\mathrm{Off}(\mathbf{G})\|_{2}\leq\|\mathbf{D}^{-1}\|_{2}\|\mathrm{Off}(\mathbf{G})\|_{2}\leq\frac{1}{d_{\min}}\cdot\delta=\frac{\delta}{d_{\min}}<1. By the Neumann series, \mathbf{I}+\mathbf{D}^{-1}\mathrm{Off}(\mathbf{G}) is invertible, hence \mathbf{G}=\mathbf{D}(\mathbf{I}+\mathbf{D}^{-1}\mathrm{Off}(\mathbf{G})) is invertible. Thus, \det(\mathbf{G})=\det(\mathbf{X})\det(\mathbf{Z})\neq 0 implies both factors are invertible. ∎ Lemma B.7 (Near-diagonal factors) . Under the hypotheses of Proposition B.3 , it follows that \displaystyle\sup_{t\leq T}\left\lVert\mathrm{Off}(\mathbf{X}(t))\right\rVert_{\mathrm{F}}\leq C_{4}\gamma,\qquad\sup_{t\leq T}\left\lVert\mathrm{Off}(\mathbf{Z}(t))\right\rVert_{\mathrm{F}}\leq C_{5}\gamma. Moreover, on any interval where d_{\min}(t)\geq d_{0}>0 , the diagonal entries satisfy \displaystyle|[\mathbf{X}(t)]_{ii}-\widetilde{s}_{i}(t)|\leq C_{6}\Big(\gamma+\frac{\gamma}{\sqrt{d_{0}}}\Big),\qquad|[\mathbf{Z}(t)]_{ii}-\widetilde{s}_{i}(t)|\leq C_{6}\Big(\gamma+\frac{\gamma}{\sqrt{d_{0}}}\Big), where \widetilde{s}_{i}(t)\coloneqq\mathrm{sgn}(\xi_{i})\sqrt{d_{i}(t)} is the signed square root with \xi_{i}\sim\mathcal{N}(0,1) , \kappa is the Lipschitz constant from Proposition B.3 , and C_{6} depends only on \kappa . Proof. By Lipschitz stability ( Proposition B.3 ), \|\mathbf{X}(t)-\mathbf{X}^{\star}(t)\|_{F}\leq\kappa\gamma where \mathbf{X}^{\star}(t)\in\mathcal{M} is diagonal. Since \mathrm{Off}(\mathbf{X}^{\star})=\mathbf{0} , \displaystyle\|\mathrm{Off}(\mathbf{X})\|_{F}=\|\mathrm{Off}(\mathbf{X}-\mathbf{X}^{\star})\|_{F}\leq\|\mathbf{X}-\mathbf{X}^{\star}\|_{F}\leq\kappa\gamma. The same bound holds for \mathbf{Z} similarly. For the diagonal entries, we let \displaystyle x_{i}\coloneqq[\mathbf{X}]_{ii},\quad z_{i}\coloneqq[\mathbf{Z}]_{ii},\quad x_{i}^{\star}\coloneqq[\mathbf{X}^{\star}]_{ii},\quad z_{i}^{\star}\coloneqq[\mathbf{Z}^{\star}]_{ii}. The reference trajectory starts from x_{i}^{\star}(0)=0 and z_{i}^{\star}(0)=\gamma\xi_{i} . Since \xi_{i}\sim\mathcal{N}(0,1) , Gaussian concentration gives |\xi_{i}|=O(1) with high probability, so it holds that |x_{i}^{\star}(0)-z_{i}^{\star}(0)|=O(\gamma) . On diagonal manifold \mathcal{M} , the SpecGF dynamics for mode i are \displaystyle\dot{x}_{i}^{\star}=-\frac{e_{i}^{\star}z_{i}^{\star}}{\sqrt{(e_{i}^{\star}z_{i}^{\star})^{2}+\beta}},\qquad\dot{z}_{i}^{\star}=-\frac{e_{i}^{\star}x_{i}^{\star}}{\sqrt{(e_{i}^{\star}x_{i}^{\star})^{2}+\beta}}, where e_{i}^{\star}(t)=x_{i}^{\star}(t)z_{i}^{\star}(t)-\sigma_{i} . Define the strictly increasing function g(t)\coloneqq\frac{t}{\sqrt{t^{2}+\beta}} . The imbalance \delta_{i}^{\star}\coloneqq x_{i}^{\star}-z_{i}^{\star} satisfies \displaystyle\dot{\delta}_{i}^{\star}=g(e_{i}^{\star}x_{i}^{\star})-g(e_{i}^{\star}z_{i}^{\star}). Since e_{i}^{\star}x_{i}^{\star}-e_{i}^{\star}z_{i}^{\star}=e_{i}^{\star}\delta_{i}^{\star} , for e_{i}^{\star}<0 , the sign of (e_{i}^{\star}x_{i}^{\star})-(e_{i}^{\star}z_{i}^{\star}) is opposite to that of \delta_{i}^{\star} . By the monotonicity of g , this implies \delta_{i}^{\star}\cdot\dot{\delta}_{i}^{\star}\leq 0 , hence |\delta_{i}^{\star}(t)| is non-increasing. We now show that e_{i}^{\star}(t)\leq 0 for all t\geq 0 . At t=0 , we have e_{i}^{\star}(0)=x_{i}^{\star}(0)z_{i}^{\star}(0)-\sigma_{i}=-\sigma_{i}<0 . If e_{i}^{\star}(t_{0})=0 for some t_{0}>0 , then \dot{x}_{i}^{\star}(t_{0})=\dot{z}_{i}^{\star}(t_{0})=0 . By ODE uniqueness (due to locally Lipschitz), (x_{i}^{\star},z_{i}^{\star}) remains constant for t\geq t_{0} , so e_{i}^{\star}(t)=0 for all t\geq t_{0} . Thus, e_{i}^{\star}(t) cannot become positive, and e_{i}^{\star}(t)\leq 0 for all t . Combining the above, \delta_{i}^{\star}\cdot\dot{\delta}_{i}^{\star}\leq 0 holds for all t , and thus \displaystyle|x_{i}^{\star}(t)-z_{i}^{\star}(t)|\leq|x_{i}^{\star}(0)-z_{i}^{\star}(0)|=O(\gamma)\quad\text{for all }t. Now, we define \displaystyle\widetilde{s}_{i}^{\star}\coloneqq\mathrm{sgn}(\xi_{i})\sqrt{d_{i}^{\star}},\qquad\widetilde{s}_{i}\coloneqq\mathrm{sgn}(\xi_{i})\sqrt{d_{i}}. We claim that x_{i}^{\star},z_{i}^{\star},\widetilde{s}_{i}^{\star} all share the same sign as \xi_{i} . To see this, note that z_{i}^{\star}(0)=\gamma\xi_{i} has the sign of \xi_{i} , while x_{i}^{\star}(0)=0 . For e_{i}^{\star}\leq 0 , the dynamics give \displaystyle\dot{x}_{i}^{\star}=-\frac{e_{i}^{\star}z_{i}^{\star}}{\sqrt{(e_{i}^{\star}z_{i}^{\star})^{2}+\beta}}, which has the same sign as z_{i}^{\star} . Thus, x_{i}^{\star} immediately enters the same sign as z_{i}^{\star} and cannot cross zero thereafter (since \dot{x}_{i}^{\star} would push it back). Similarly, \dot{z}_{i}^{\star} has the same sign as x_{i}^{\star} , so z_{i}^{\star} also maintains its sign. Since \|\mathbf{G}-\mathbf{G}^{\star}\|_{F}\leq\kappa\gamma , we have |d_{i}-d_{i}^{\star}|\leq\kappa\gamma . On intervals where d_{\min}(t)\geq d_{0} , \displaystyle|\widetilde{s}_{i}-\widetilde{s}_{i}^{\star}|=|\sqrt{d_{i}}-\sqrt{d_{i}^{\star}}|=\frac{|d_{i}-d_{i}^{\star}|}{\sqrt{d_{i}}+\sqrt{d_{i}^{\star}}}\leq\frac{\kappa\gamma}{\sqrt{d_{i}}}\leq\frac{\kappa\gamma}{\sqrt{d_{0}}}. Since x_{i}^{\star} and \widetilde{s}_{i}^{\star} have the same sign, we obtain \displaystyle|x_{i}^{\star}-\widetilde{s}_{i}^{\star}|=\Big||x_{i}^{\star}|-\sqrt{|x_{i}^{\star}||z_{i}^{\star}|}\Big|\leq|x_{i}^{\star}-z_{i}^{\star}|=O(\gamma). Combining with |x_{i}-x_{i}^{\star}|\leq\|\mathbf{X}-\mathbf{X}^{\star}\|_{F}\leq\kappa\gamma , the triangle inequality gives \displaystyle|x_{i}-\widetilde{s}_{i}|\leq|x_{i}-x_{i}^{\star}|+|x_{i}^{\star}-\widetilde{s}_{i}^{\star}|+|\widetilde{s}_{i}^{\star}-\widetilde{s}_{i}|\leq C_{6}\Big(\gamma+\frac{\gamma}{\sqrt{d_{0}}}\Big), where C_{6} depends only on \kappa . The same argument gives |z_{i}-\widetilde{s}_{i}|\leq C_{6}(\gamma+\gamma/\sqrt{d_{0}}) . ∎ B.5 Square-Root Dynamics Lemma B.8 (Square-root dynamics) . Suppose \delta -alignment holds with \delta=O(\gamma) , \gamma=\Theta(\varepsilon^{1/2}) , and \beta=O(\varepsilon^{3}) . Let [\tau^{\prime},T^{\prime}] be any subinterval of [\tau,T] on which \displaystyle d_{\min}(t)\coloneqq\min_{i\in[r]}d_{i}(t)\geq d_{0}=\Theta(\gamma). Then for each active mode i (i.e., e_{i}(t)\leq-\varepsilon ), the square-root coordinate s_{i}(t)\coloneqq\sqrt{d_{i}(t)} satisfies, for a.e. t\in[\tau^{\prime},T^{\prime}] , \dot{s}_{i}(t)=-\mathrm{sgn}(e_{i}(t))\,\alpha_{i}(t)+R_{i}(t),\qquad\alpha_{i}(t)\coloneqq\frac{|e_{i}(t)|\,s_{i}(t)}{\sqrt{e_{i}(t)^{2}s_{i}(t)^{2}+\beta}}\in(0,1]. (19) Moreover, the remainder obeys |R_{i}(t)|\leq C_{7}\varepsilon^{1/4}, (20) where C_{7}=C_{7}(\mathbf{Y},r) depends only on constants from Lemma B.7 (and earlier constants). In particular, for every active mode i (i.e., e_{i}(t)\leq-\varepsilon ), \displaystyle\dot{s}_{i}(t)=1+O(\varepsilon^{1/4}). Proof. Since d_{i}(t)=[\mathbf{X}(t)\mathbf{Z}(t)]_{ii} and s_{i}(t)=\sqrt{d_{i}(t)} , the chain rule gives, for a.e. t with d_{i}(t)>0 , \displaystyle\dot{d}_{i}(t)=[\dot{\mathbf{X}}(t)\mathbf{Z}(t)]_{ii}+[\mathbf{X}(t)\dot{\mathbf{Z}}(t)]_{ii},\qquad\dot{s}_{i}(t)=\frac{\dot{d}_{i}(t)}{2s_{i}(t)}. (21) Let \mathbf{E}(t)\coloneqq\mathbf{G}(t)-\bm{\Sigma} and \mathbf{G}_{\mathbf{X}}\coloneqq\mathbf{E}\mathbf{Z}^{\top}+\mathbf{X}\mathbf{Z}_{\perp}\mathbf{Z}_{\perp}^{\top}=\mathbf{U}_{r}^{\top}\nabla_{\mathbf{A}}\mathcal{L} . By Lemma B.7 , \mathbf{X} and \mathbf{Z} are nearly diagonal: \displaystyle\mathbf{X}=\mathrm{Diag}(x_{i})+\bm{\Delta}_{X},\qquad\mathbf{Z}=\mathrm{Diag}(z_{i})+\bm{\Delta}_{Z}, where x_{i},z_{i}=\mathrm{sgn}(\xi_{i})s_{i}+O(\gamma+\gamma/\sqrt{d_{0}}) and \|\bm{\Delta}_{X}\|_{F},\|\bm{\Delta}_{Z}\|_{F}=O(\gamma) . The SpecGF dynamics give \displaystyle\dot{\mathbf{X}}=-\mathcal{T}_{\mathrm{col},\beta}(\mathbf{G}_{\mathbf{X}}),\qquad\dot{\mathbf{Z}}=-\mathcal{T}_{\mathrm{row},\beta}(\nabla_{\mathbf{B}}\mathcal{L})\,\mathbf{V}_{r}. Since \|\mathbf{X}\mathbf{Z}_{\perp}\|_{F}=O(\delta) (by \delta -alignment), we have \mathbf{G}_{\mathbf{X}}=\mathbf{E}\mathbf{Z}^{\top}+O(\delta) . Combined with \mathbf{E}=\mathrm{Diag}(e_{i})+O(\delta) and z_{i}=\mathrm{sgn}(\xi_{i})s_{i}+O(\gamma) , this yields \displaystyle\mathbf{G}_{\mathbf{X}}=\mathrm{Diag}(e_{i}z_{i})+O(\delta+\gamma),\qquad\mathbf{G}_{\mathbf{X}}^{\top}\mathbf{G}_{\mathbf{X}}=\mathrm{Diag}(e_{i}^{2}s_{i}^{2})+O(\delta+\gamma). Since \mathcal{T}_{\mathrm{col},\beta}(\mathbf{G})=\mathbf{G}\,(\mathbf{G}^{\top}\mathbf{G}+\beta\mathbf{I})^{-1/2} , we obtain \displaystyle[\dot{\mathbf{X}}]_{ii}=-\frac{e_{i}z_{i}}{\sqrt{e_{i}^{2}s_{i}^{2}+\beta}}+O(\delta+\gamma)=-\frac{e_{i}\,\mathrm{sgn}(\xi_{i})s_{i}}{\sqrt{e_{i}^{2}s_{i}^{2}+\beta}}+O(\delta+\gamma). (22) For \dot{\mathbf{Z}} , write \nabla_{\mathbf{B}}\mathcal{L}=\mathbf{X}^{\top}\mathbf{E}\mathbf{V}_{r}^{\top}+\mathbf{X}^{\top}\mathbf{X}\mathbf{Z}_{\perp}\mathbf{V}_{\perp}^{\top} . Using \mathbf{V}_{r}^{\top}\mathbf{V}_{r}=\mathbf{I} and \mathbf{V}_{\perp}^{\top}\mathbf{V}_{r}=\mathbf{0} , we have \nabla_{\mathbf{B}}\mathcal{L}\,\mathbf{V}_{r}=\mathbf{X}^{\top}\mathbf{E} , so \displaystyle\dot{\mathbf{Z}}=\dot{\mathbf{B}}\mathbf{V}_{r}=-\mathcal{T}_{\mathrm{row},\beta}(\nabla_{\mathbf{B}}\mathcal{L})\,\mathbf{V}_{r}=-\bigl((\mathbf{X}^{\top}\mathbf{E})(\mathbf{X}^{\top}\mathbf{E})^{\top}+\beta\mathbf{I}\bigr)^{-1/2}\,\mathbf{X}^{\top}\mathbf{E}. Moreover, (\mathbf{X}^{\top}\mathbf{E})(\mathbf{X}^{\top}\mathbf{E})^{\top}=\mathbf{X}^{\top}\mathbf{E}\mathbf{E}^{\top}\mathbf{X}=\mathrm{Diag}(s_{i}^{2}e_{i}^{2})+O(\delta+\gamma) , so \displaystyle[\dot{\mathbf{Z}}]_{ii}=-\frac{x_{i}e_{i}}{\sqrt{x_{i}^{2}e_{i}^{2}+\beta}}+O(\delta+\gamma)=-\frac{\mathrm{sgn}(\xi_{i})s_{i}\,e_{i}}{\sqrt{e_{i}^{2}s_{i}^{2}+\beta}}+O(\delta+\gamma). (23) Using Lemma B.7 and ( 22 )–( 23 ), we compute the diagonal entries of \dot{\mathbf{X}}\mathbf{Z} and \mathbf{X}\dot{\mathbf{Z}} : \displaystyle[\dot{\mathbf{X}}\mathbf{Z}]_{ii} \displaystyle=[\dot{\mathbf{X}}]_{ii}\,[\mathbf{Z}]_{ii}+O(\|\dot{\mathbf{X}}\|_{F}\|\mathrm{Off}(\mathbf{Z})\|_{F})=-\frac{e_{i}s_{i}^{2}}{\sqrt{e_{i}^{2}s_{i}^{2}+\beta}}+O\!\Bigl(\delta+\gamma+\frac{\gamma}{\sqrt{d_{0}}}\Bigr), (24) \displaystyle[\mathbf{X}\dot{\mathbf{Z}}]_{ii} \displaystyle=[\mathbf{X}]_{ii}\,[\dot{\mathbf{Z}}]_{ii}+O(\|\mathbf{X}\|_{F}\|\mathrm{Off}(\dot{\mathbf{Z}})\|_{F})=-\frac{e_{i}s_{i}^{2}}{\sqrt{e_{i}^{2}s_{i}^{2}+\beta}}+O\!\Bigl(\delta+\gamma+\frac{\gamma}{\sqrt{d_{0}}}\Bigr). (25) Summing ( 24 )–( 25 ) and using ( 21 ), \displaystyle\dot{d}_{i}=-\frac{2e_{i}s_{i}^{2}}{\sqrt{e_{i}^{2}s_{i}^{2}+\beta}}+O\!\Bigl(\delta+\gamma+\frac{\gamma}{\sqrt{d_{0}}}\Bigr),\qquad\dot{s}_{i}=-\frac{e_{i}s_{i}}{\sqrt{e_{i}^{2}s_{i}^{2}+\beta}}+O\!\Bigl(\delta+\frac{\gamma}{\sqrt{d_{0}}}\Bigr), where we used s_{i}\geq\sqrt{d_{0}} on [\tau^{\prime},T^{\prime}] to absorb the O(\gamma) contribution into O(\gamma/\sqrt{d_{0}}) after dividing by 2s_{i} . Under the scaling \gamma=\Theta(\varepsilon^{1/2}) , d_{0}=\Theta(\gamma) , and \delta=O(\gamma) , the dominant error term is \gamma/\sqrt{d_{0}}=O(\varepsilon^{1/4}) . Defining \alpha_{i}(t)\coloneqq\frac{|e_{i}(t)|\,s_{i}(t)}{\sqrt{e_{i}(t)^{2}s_{i}(t)^{2}+\beta}}\in(0,1] and R_{i}(t)\coloneqq\dot{s}_{i}(t)+\mathrm{sgn}(e_{i}(t))\,\alpha_{i}(t) , we see that ( 19 ) holds and ( 20 ) follows. Finally, for active modes e_{i}(t)\leq-\varepsilon , we have \mathrm{sgn}(e_{i})=-1 and \displaystyle\alpha_{i}=\frac{|e_{i}|\,s_{i}}{\sqrt{e_{i}^{2}s_{i}^{2}+\beta}}=\Bigl(1+\frac{\beta}{e_{i}^{2}s_{i}^{2}}\Bigr)^{-1/2}\geq 1-\frac{\beta}{2\varepsilon^{2}d_{0}}=1-O(\varepsilon^{1/2}), using \beta=O(\varepsilon^{3}) and d_{0}=\Theta(\varepsilon^{1/2}) . Since \alpha_{i}\leq 1 trivially and |R_{i}|=O(\varepsilon^{1/4}) , we conclude \dot{s}_{i}=1+O(\varepsilon^{1/4}) for active modes.∎ B.6 Non-degeneracy Persistence Lemma B.9 (Non-degeneracy persistence) . Suppose d_{\min}(\tau)\geq d_{0}=\Theta(\gamma) with \gamma=\Theta(\varepsilon^{1/2}) and \beta=O(\varepsilon^{3}) . Let \underline{d}\coloneqq\min\{d_{0},\sigma_{r}-\varepsilon\} . Then for all t\in[\tau,T] , \displaystyle d_{\min}(t)\geq\underline{d}>0. Proof. We define the exit time T_{\mathrm{exit}}\coloneqq\inf\{t>\tau:d_{\min}(t)<\underline{d}\} . We show that T_{\mathrm{exit}}>T , which implies d_{\min}(t)\geq\underline{d} for all t\in[\tau,T] . On [\tau,T_{\mathrm{exit}}) , we have d_{\min}(t)\geq\underline{d}>0 by definition, so Lemma B.8 applies. For any active mode i (i.e., e_{i}(t)\leq-\varepsilon ), we have \dot{s}_{i}(t)=1+O(\varepsilon^{1/4})>0 for small \varepsilon . Hence, s_{i}(t) is strictly increasing, and so is d_{i}(t)=s_{i}(t)^{2} . For contradiction, suppose that some mode i reaches d_{i}(t)=\underline{d} from above. Then e_{i}(t)=d_{i}(t)-\sigma_{i}\leq\underline{d}-\sigma_{r}<-\varepsilon (since \underline{d}\leq\sigma_{r}-\varepsilon ), so i is active and \dot{d}_{i}(t)>0 . This contradicts the decrease after reaching \underline{d} . By continuity, d_{\min}(t) cannot reach \underline{d} on [\tau,T] . It follows that T_{\mathrm{exit}}>T , and hence d_{\min}(t)\geq\underline{d} for all t\in[\tau,T] . ∎ B.7 Main Theorem: Uniform Growth Theorem B.10 (Uniform growth) . Fix a small target tolerance \varepsilon>0 . Consider SpecGF with \mathcal{T}_{\beta} initialized at \mathbf{A}(0)=\mathbf{0} , \mathbf{B}(0)=\gamma\mathbf{N} . Let \tau\asymp\frac{\sigma_{r}}{\sigma_{1}\sqrt{r}} and let T be the termination time. For \gamma=\Theta(\varepsilon^{1/2}) and \beta=O(\varepsilon^{3}) , the following holds with high probability for all t\in[\tau,T] and all active modes i\in\mathcal{I}_{\varepsilon}(t) : (i) Approximate unit speed: \displaystyle\frac{\mathrm{d}}{\mathrm{d}t}\sqrt{d_{i}(t)}=1+O(\varepsilon^{1/4}) . (ii) Uniform growth: For any two active modes i,j\in\mathcal{I}_{\varepsilon}(t) , \displaystyle\left\lvert\frac{\mathrm{d}}{\mathrm{d}t}\sqrt{d_{i}(t)}-\frac{\mathrm{d}}{\mathrm{d}t}\sqrt{d_{j}(t)}\right\rvert=O(\varepsilon^{1/4}). Proof. By Lemma B.5 , d_{\min}(\tau)\geq d_{0} for d_{0}\asymp\tau\gamma=\Theta(\gamma)=\Theta(\varepsilon^{1/2}) with high probability. By Proposition B.3 , \delta=O(\gamma)=O(\varepsilon^{1/2}) . Applying Lemmas B.8 and B.9 , every active mode satisfies \dot{s}_{i}(t)=1+O(\varepsilon^{1/4}) . This proves (i). For (ii), for any active i,j , \displaystyle|\dot{s}_{i}(t)-\dot{s}_{j}(t)|\leq|\dot{s}_{i}(t)-1|+|\dot{s}_{j}(t)-1|=O(\varepsilon^{1/4}), which concludes the proof. ∎ Appendix C Extension to Underparameterized Case ( r\cdots>\sigma_{r^{\star}}>0 . The LoRA factorization has rank r\leq r^{\star} : \mathbf{A}\in\mathbb{R}^{m\times r} and \mathbf{B}\in\mathbb{R}^{r\times n} . We partition the target singular vectors as \mathbf{U}=[\mathbf{U}_{r}\mid\mathbf{U}_{>r}] and \mathbf{V}=[\mathbf{V}_{r}\mid\mathbf{V}_{>r}] , where \mathbf{U}_{r},\mathbf{V}_{r}\in\mathbb{R}^{\cdot\times r} correspond to the top r singular values and \mathbf{U}_{>r},\mathbf{V}_{>r} correspond to the remaining r^{\star}-r singular values. Let \mathbf{V}_{\perp}\in\mathbb{R}^{n\times(n-r^{\star})} denote the orthogonal complement of \mathbf{V} in \mathbb{R}^{n} . The extended core variables are: \displaystyle\mathbf{X}_{r} \displaystyle\coloneqq\mathbf{U}_{r}^{\top}\mathbf{A}\in\mathbb{R}^{r\times r}, \displaystyle\mathbf{X}_{>r} \displaystyle\coloneqq\mathbf{U}_{>r}^{\top}\mathbf{A}\in\mathbb{R}^{(r^{\star}-r)\times r}, \displaystyle\mathbf{Z}_{r} \displaystyle\coloneqq\mathbf{B}\mathbf{V}_{r}\in\mathbb{R}^{r\times r}, \displaystyle\mathbf{Z}_{>r} \displaystyle\coloneqq\mathbf{B}\mathbf{V}_{>r}\in\mathbb{R}^{r\times(r^{\star}-r)}, \displaystyle\mathbf{Z}_{\perp} \displaystyle\coloneqq\mathbf{B}\mathbf{V}_{\perp}\in\mathbb{R}^{r\times(n-r^{\star})}. In this setup, the core product is defined by \mathbf{G}_{r}\coloneqq\mathbf{X}_{r}\mathbf{Z}_{r}\in\mathbb{R}^{r\times r} . Similar to \delta -alignment, we define extended \delta -alignment by \displaystyle\left\lVert\mathrm{Off}(\mathbf{G}_{r}(t))\right\rVert_{\mathrm{F}}\leq\delta,\quad\left\lVert\mathbf{X}_{>r}(t)\right\rVert_{\mathrm{F}}\leq\delta,\quad\left\lVert\mathbf{Z}_{>r}(t)\right\rVert_{\mathrm{F}}\leq\delta,\quad\left\lVert\mathbf{X}_{r}(t)\mathbf{Z}_{\perp}(t)\right\rVert_{\mathrm{F}}\leq\delta. This generalizes the original \delta -alignment condition to the underparameterized setting by additionally controlling the components outside the top- r subspace. Note that, given a convergence to the best rank- r approximation (cf. Propositions D.11 and D.13 ), for any \delta>0 we can always find some finite t_{0}>0 such that extended \delta -alignment holds for all t\geq t_{0} . C.2 Column Space Invariance Lemma C.1 (Column space invariance) . If \mathbf{A}(0)=\bm{0} , then \mathbf{A}(t)\in\mathrm{col}(\mathbf{U}) for all t\geq 0 . Consequently, \mathbf{A}(t)=\mathbf{U}_{r}\mathbf{X}_{r}(t)+\mathbf{U}_{>r}\mathbf{X}_{>r}(t) . Proof. The gradient \nabla_{\mathbf{A}}\mathcal{L}=(\mathbf{A}\mathbf{B}-\mathbf{Y})\mathbf{B}^{\top} lies in \mathrm{col}(\mathbf{A})+\mathrm{col}(\mathbf{Y})\subseteq\mathrm{col}(\mathbf{U}) whenever \mathbf{A}\in\mathrm{col}(\mathbf{U}) . At t=0 , \nabla_{\mathbf{A}}\mathcal{L}(0)=-\mathbf{Y}\mathbf{B}(0)^{\top}\in\mathrm{col}(\mathbf{U}) . Since \mathcal{T}_{\beta} preserves column spaces, \dot{\mathbf{A}}(0)\in\mathrm{col}(\mathbf{U}) . By induction and continuity of the flow, \mathbf{A}(t)\in\mathrm{col}(\mathbf{U}) for all t\geq 0 . ∎ C.3 Invariant Manifold Lemma C.2 (Invariant manifold) . Under the initialization \mathbf{A}(0)=\bm{0} and \mathbf{B}(0)=\gamma\mathbf{N} with \mathbf{N} having i.i.d. \mathcal{N}(0,1) entries, the extended manifold \displaystyle\mathcal{M}_{\mathrm{ext}}\coloneqq\big\{(\mathbf{X}_{r},\mathbf{X}_{>r},\mathbf{Z}_{r},\mathbf{Z}_{>r},\mathbf{Z}_{\perp}):\mathbf{X}_{>r}=\bm{0},\;\mathbf{Z}_{>r}=\bm{0},\;\mathbf{Z}_{\perp}=\bm{0}\big\} is invariant under the SpecGF dynamics. Proof. Let \bm{\Sigma}_{r}=\mathrm{Diag}(\sigma_{1},\ldots,\sigma_{r}) and \bm{\Sigma}_{>r}=\mathrm{Diag}(\sigma_{r+1},\ldots,\sigma_{r^{\star}}) . On \mathcal{M}_{\mathrm{ext}} , we have \mathbf{A}=\mathbf{U}_{r}\mathbf{X}_{r} and \mathbf{B}=\mathbf{Z}_{r}\mathbf{V}_{r}^{\top} , so \displaystyle\mathbf{A}\mathbf{B}-\mathbf{Y} \displaystyle=\mathbf{U}_{r}\mathbf{X}_{r}\mathbf{Z}_{r}\mathbf{V}_{r}^{\top}-\mathbf{U}_{r}\bm{\Sigma}_{r}\mathbf{V}_{r}^{\top}-\mathbf{U}_{>r}\bm{\Sigma}_{>r}\mathbf{V}_{>r}^{\top} \displaystyle=\mathbf{U}_{r}(\mathbf{X}_{r}\mathbf{Z}_{r}-\bm{\Sigma}_{r})\mathbf{V}_{r}^{\top}-\mathbf{U}_{>r}\bm{\Sigma}_{>r}\mathbf{V}_{>r}^{\top}. The gradient \nabla_{\mathbf{A}}\mathcal{L}=(\mathbf{A}\mathbf{B}-\mathbf{Y})\mathbf{B}^{\top}=\mathbf{U}_{r}(\mathbf{X}_{r}\mathbf{Z}_{r}-\bm{\Sigma}_{r})\mathbf{Z}_{r}^{\top} lies entirely in \mathrm{col}(\mathbf{U}_{r}) , implying \dot{\mathbf{X}}_{>r}=\mathbf{U}_{>r}^{\top}\dot{\mathbf{A}}=\bm{0} . Similarly, \nabla_{\mathbf{B}}\mathcal{L}=\mathbf{X}_{r}^{\top}(\mathbf{X}_{r}\mathbf{Z}_{r}-\bm{\Sigma}_{r})\mathbf{V}_{r}^{\top}-\mathbf{X}_{>r}^{\top}\bm{\Sigma}_{>r}\mathbf{V}_{>r}^{\top} . When \mathbf{X}_{>r}=\bm{0} , this lies entirely in the row space of \mathbf{V}_{r}^{\top} , so \dot{\mathbf{Z}}_{>r}=\dot{\mathbf{B}}\mathbf{V}_{>r}=\bm{0} and \dot{\mathbf{Z}}_{\perp}=\dot{\mathbf{B}}\mathbf{V}_{\perp}=\bm{0} . ∎ C.4 Lipschitz Stability near the Invariant Manifold Proposition C.3 (Alignment in the underparameterized case) . Under the initialization \mathbf{A}(0)=\bm{0} and \mathbf{B}(0)=\gamma\mathbf{N} , there exist constants C_{>r},C_{\perp}<\infty such that, with probability at least 1-e^{-cr(r^{\star}-r)}-e^{-cr(n-r^{\star})} , \displaystyle\sup_{t\leq T}\|\mathbf{X}_{>r}(t)\|_{F}\leq C_{>r}\gamma,\qquad\sup_{t\leq T}\|\mathbf{Z}_{>r}(t)\|_{F}\leq C_{>r}\gamma,\qquad\sup_{t\leq T}\|\mathbf{Z}_{\perp}(t)\|_{F}\leq C_{\perp}\gamma. Proof. Let \mathbf{S}(t)\coloneqq(\mathbf{X}_{r}(t),\mathbf{X}_{>r}(t),\mathbf{Z}_{r}(t),\mathbf{Z}_{>r}(t),\mathbf{Z}_{\perp}(t)) denote the true trajectory. At initialization with \mathbf{A}(0)=\bm{0} and \mathbf{B}(0)=\gamma\mathbf{N} , the core variables satisfy \displaystyle\mathbf{X}_{r}(0)=\bm{0},\quad\mathbf{X}_{>r}(0)=\bm{0},\quad\mathbf{Z}_{r}(0)=\gamma\mathbf{N}\mathbf{V}_{r},\quad\mathbf{Z}_{>r}(0)=\gamma\mathbf{N}\mathbf{V}_{>r},\quad\mathbf{Z}_{\perp}(0)=\gamma\mathbf{N}\mathbf{V}_{\perp}. We define the projection onto \mathcal{M}_{\mathrm{ext}} by \Pi(\mathbf{X}_{r},\mathbf{X}_{>r},\mathbf{Z}_{r},\mathbf{Z}_{>r},\mathbf{Z}_{\perp})\coloneqq(\mathbf{X}_{r},\bm{0},\mathbf{Z}_{r},\bm{0},\bm{0}) , and consider the reference trajectory \mathbf{S}^{\star}(t) initialized at \mathbf{S}^{\star}(0)=\Pi(\mathbf{S}(0))=(\bm{0},\bm{0},\gamma\mathbf{N}\mathbf{V}_{r},\bm{0},\bm{0}) . By Lemma C.2 , \mathbf{S}^{\star}(t)\in\mathcal{M}_{\mathrm{ext}} for all t\geq 0 . The initial discrepancy is \displaystyle\|\mathbf{S}(0)-\mathbf{S}^{\star}(0)\|_{F}=\|\mathbf{Z}_{>r}(0)\|_{F}+\|\mathbf{Z}_{\perp}(0)\|_{F}=O(\gamma\sqrt{r(n-r)}) with high probability by Gaussian concentration. The SpecGF vector field with \mathcal{T}_{\beta} is \mathcal{C}^{1} , so finite-horizon Lipschitz stability gives \displaystyle\|\mathbf{S}(t)-\mathbf{S}^{\star}(t)\|_{F}\leq e^{LH}\|\mathbf{S}(0)-\mathbf{S}^{\star}(0)\|_{F}\leq\kappa\gamma for some finite \kappa=\kappa(\mathbf{Y},r,T) . Since \mathbf{X}^{\star}_{>r}(t)\equiv\bm{0} , \mathbf{Z}^{\star}_{>r}(t)\equiv\bm{0} , and \mathbf{Z}^{\star}_{\perp}(t)\equiv\bm{0} , the bounds naturally hold. ∎ C.5 Reduction to the Square Case On the invariant manifold \mathcal{M}_{\mathrm{ext}} , the dynamics reduce to the r\times r system: \displaystyle\dot{\mathbf{X}}_{r} \displaystyle=-\mathcal{T}_{\mathrm{col},\beta}\big((\mathbf{X}_{r}\mathbf{Z}_{r}-\bm{\Sigma}_{r})\mathbf{Z}_{r}^{\top}\big), \displaystyle\dot{\mathbf{Z}}_{r} \displaystyle=-\mathcal{T}_{\mathrm{row},\beta}\big(\mathbf{X}_{r}^{\top}(\mathbf{X}_{r}\mathbf{Z}_{r}-\bm{\Sigma}_{r})\big). This is equivalent to the original rank- r^{\star} dynamics (in case of r=r^{\star} ) with target \bm{\Sigma}_{r}=\mathrm{Diag}(\sigma_{1},\ldots,\sigma_{r}) . By Proposition C.3 , perturbations from \mathcal{M}_{\mathrm{ext}} are controlled: \|\mathbf{X}_{>r}\|_{F},\|\mathbf{Z}_{>r}\|_{F},\|\mathbf{Z}_{\perp}\|_{F}=O(\gamma) . These contribute only at most O(\gamma) errors to the effective dynamics of (\mathbf{X}_{r},\mathbf{Z}_{r}) . C.6 Extended Uniform Growth Theorem Theorem C.4 (Uniform growth for rr} , \mathbf{Z}_{>r} , \mathbf{Z}_{\perp} contribute only at most O(\gamma) errors, which are absorbed into the existing O(\gamma) terms. The proof then follows Theorem B.10 . ∎ Appendix D Convergence to Global Minima of SpecGF D.1 Analyticity of SpecGF We first state that the inverse square root of symmetric positive definite (SPD) matrices is analytic. This directly follows from 7 . Lemma D.1 . At any real SPD \mathbf{S} , the function g(\mathbf{S})=\mathbf{S}^{-\frac{1}{2}} is analytic. That is, g is analytic on the (open) set of the symmetric positive-definite matrices. Proof. Any SPD matrix \mathbf{S} is closed, invertible, (densely) defined linear operator on a Banach space (also a Hilbert space for our case), denoted as X , and has real eigenvalues. Throughout this proof, we abuse the notation \sigma(\mathbf{S}) to denote the spectrum of \mathbf{S} . We first show \mathbf{S} is sectorial of any \omega\in[0,\pi) ; fix any \omega . \sigma(\mathbf{S}) is exactly the set of eigenvalues of \mathbf{S} , and therefore is contained in \overline{S_{\omega}} . Next, for each \omega^{\prime}\in(\omega,\pi) , take any \lambda\not\in\overline{S_{\omega^{\prime}}} . Then \left\lvert\mathrm{Arg}(\lambda)\right\rvert>\omega^{\prime} and thus, for any eigenvalue s of \mathbf{S} , |s-\lambda|\geq\mathrm{dist}(\lambda,(0,+\infty))=\begin{cases}|\lambda|\left\lvert\sin(\mathrm{Arg}(\lambda))\right\rvert\geq|\lambda|\sin\omega^{\prime}&\text{if }\left\lvert\mathrm{Arg}(\lambda)\right\rvert\leq\frac{\pi}{2},\\ |\lambda|\geq|\lambda|\sin\omega^{\prime}&\text{if }\left\lvert\mathrm{Arg}(\lambda)\right\rvert>\frac{\pi}{2}.\end{cases} Hence, \lVert\lambda(\mathbf{S}-\lambda)^{-1}\rVert=\max_{s\in\sigma(\mathbf{S})}\frac{|\lambda|}{|s-\lambda|}\leq\frac{1}{\sin\omega^{\prime}} is uniformly bounded. On Hilbert space, R-boundedness is equivalent to boundedness ( 7 ) . Thus, \mathbf{S} is R-sectorial of any \omega\in[0,\pi) . We also see that \mathbf{S} admits a bounded \mathcal{H}^{\infty}(S_{\omega}) for any \omega\in[0,\pi) , because for f\in\mathcal{H}^{\infty}(S_{\omega})\setminus\{0\} , \lVert f(\mathbf{S})\rVert_{L(X)}=\max_{s\in\sigma(\mathbf{S})}|f(s)|\leq\sup_{f}\;\lVert f\rVert_{\mathcal{H}^{\infty}(S_{\omega})} As \mathbf{S} is SPD, there exists a closed centered ball not in \sigma(\mathbf{S}) . Now take r=\frac{1}{2} in the statement of 7 . Then g(\mathbf{S}) is holomorphic, thus real-analytic restricted to the set of real matrices. As the dimension is finite, the fractional domain spaces \dot{X}_{p} coincides with X (viewed as sets) and thus L(\dot{X}_{p},\dot{X}_{q})=L(X) for any real p and q (so are L(\dot{X}_{0},\dot{X}_{1-r},\dot{X}_{r}) ). ∎ Corollary D.2 . Consider a matrix \mathbf{X}\in\mathbb{R}^{m\times r} with m\geq r . (a) \mathbf{X}\mapsto(\mathbf{X}^{\top}\mathbf{X})^{-\frac{1}{2}} is analytic on the (open) set of full-rank matrices. (b) For every \beta>0 , \mathbf{X}\mapsto(\mathbf{X}^{\top}\mathbf{X}+\beta\mathbf{I}_{r})^{-\frac{1}{2}} is analytic everywhere. The above statements analogously hold for \mathbf{X}\in\mathbb{R}^{r\times n} with n\geq r by switching \mathbf{X}^{\top}\mathbf{X} to \mathbf{X}\mathbf{X}^{\top} . Proof. This is apparent from Lemma D.1 , the fact that the polynomials are analytic, and the composition of analytic functions is again analytic. ∎ Recall that, for a matrix \mathbf{M}\in\mathbb{R}^{m\times n} , \displaystyle\mathcal{T}(\mathbf{M}) \displaystyle=\left(\left(\mathbf{M}\mathbf{M}^{\top}\right)^{\dagger}\right)^{1/2}\!\mathbf{M}, \displaystyle\mathcal{T}_{\beta}(\mathbf{M}) \displaystyle=\left(\mathbf{M}\mathbf{M}^{\top}+\beta\mathbf{I}_{m}\right)^{-1/2}\!\mathbf{M}, Let the compact SVD of \mathbf{X}\in\mathbb{R}^{m\times n} be \mathbf{U}_{r}\mathbf{S}\mathbf{V}_{r}^{\top} with \mathbf{S}\triangleq\mathrm{Diag}\{\sigma_{1},\ldots,\sigma_{r}\} . \sigma_{i}>0 are the positive singular values of \mathbf{X} , 1\leq i\leq r\leq\min\{m,n\} . Then, \displaystyle\mathcal{T}(\mathbf{X}) \displaystyle=\mathbf{U}_{r}\mathbf{V}_{r}^{\top}, \displaystyle\mathcal{T}_{\beta}(\mathbf{X}) \displaystyle=\mathbf{U}_{r}\mathbf{S}_{\beta}\mathbf{V}_{r}^{\top}, where \mathbf{S}_{\beta}\triangleq\mathrm{Diag}\left\{\frac{\sigma_{i}}{\sqrt{\sigma_{i}^{2}+\beta}}\right\}_{i\in[r]}. Proposition D.3 (Analyticity of \mathcal{T} and \mathcal{T}_{\beta} ) . SpecGF with \mathcal{T}_{\beta} is analytic everywhere. SpecGF with \mathcal{T} is analytic on the set of full-rank matrices. Moreover, the solution uniquely exists for all t\geq 0 . Proof. Analyticity is straightforward from Corollary D.2 , so we prove the existence and uniqueness part. For both \mathcal{T} and \mathcal{T}_{\beta} , \|\dot{\mathbf{A}}\|_{\mathrm{F}}^{2} and \|\dot{\mathbf{B}}\|_{\mathrm{F}}^{2} are both bounded by r . Hence, the trajectory should satisfy \left\lVert(\mathbf{A}(t),\mathbf{B}(t))\right\rVert_{\mathrm{F}}\leq\left\lVert(\mathbf{A}(0),\mathbf{B}(0))\right\rVert_{\mathrm{F}}+\sqrt{2r}t. If the maximal solution does not exist for all t\geq 0 , then the iterate should leave every compact set of the domain. Therefore, the solution must exist for all t\geq 0 . Uniqueness follows from the local Lipschitzness in (\mathbf{A},\mathbf{G}) of SpecGF (by Picard-Lindelöf theorem). ∎ D.2 Convergence Analysis of SpecGF We restate the setup of SpecGF. • The target matrix \mathbf{Y}\in\mathbb{R}^{m\times n} yields a compact SVD \mathbf{Y}=\mathbf{U}_{r^{\star}}\bm{\Sigma}\mathbf{V}_{r^{\star}}^{\top} where \bm{\Sigma}=\mathrm{Diag}(\sigma_{1},\sigma_{2},\cdots,\sigma_{r^{\star}}) with \sigma_{1}\geq\cdots\geq\sigma_{r^{\star}}>0 . • The loss function \mathcal{L}:\mathbb{R}^{m\times r}\times\mathbb{R}^{r\times n}\to\mathbb{R} is defined as \mathcal{L}(\mathbf{A},\mathbf{B})\triangleq\frac{1}{2}\left\lVert\mathbf{A}\mathbf{B}-\mathbf{Y}\right\rVert_{\mathrm{F}}^{2}. (26) We may abuse the notation and simply state \mathcal{L}(t) . • The time derivative of \mathbf{A} and \mathbf{B} is governed by either \mathcal{T} or \mathcal{T}_{\beta} applied to the gradient: \dot{\mathbf{A}}(t)=-\mathcal{T}\left(\nabla_{\mathbf{A}}\mathcal{L}(t)\right),\quad\dot{\mathbf{B}}(t)=-\mathcal{T}\left(\nabla_{\mathbf{B}}\mathcal{L}(t)\right). Lemma D.4 . On SpecGF with \mathcal{T} , the loss is monotonically non-increasing: \displaystyle\frac{d\mathcal{L}(t)}{\mathrm{d}t}\leq 0. The equality holds if and only if \nabla_{\mathbf{A}}\mathcal{L}(t)=\nabla_{\mathbf{B}}\mathcal{L}(t)=0 holds. Proof. We first show that for any matrix \mathbf{X} , \left\langle\mathcal{T}(\mathbf{X}),\mathbf{X}\right\rangle=\lVert\mathbf{X}\rVert_{*} . Let the compact SVD of \mathbf{X} be \mathbf{U}^{\prime}\mathbf{S}^{\prime}\mathbf{V}^{\prime\top} . \displaystyle\left\langle\mathcal{T}(\mathbf{X}),\mathbf{X}\right\rangle=\mathrm{tr}\left((\mathbf{U}^{\prime}\mathbf{V}^{\prime\top})^{\top}\mathbf{U}^{\prime}\mathbf{S}^{\prime}\mathbf{V}^{\prime\top}\right)=\mathrm{tr}\left(\mathbf{V}^{\prime}\mathbf{S}^{\prime}\mathbf{V}^{\prime\top}\right)=\mathrm{tr}\left(\mathbf{S}^{\prime}\right)=\lVert\mathbf{X}\rVert_{*}. We drop the argument t for simplicity. \displaystyle\frac{d\mathcal{L}}{\mathrm{d}t} \displaystyle=\left\langle\nabla_{\mathbf{A}}\mathcal{L},\dot{\mathbf{A}}\right\rangle+\left\langle\nabla_{\mathbf{B}}\mathcal{L},\dot{\mathbf{B}}\right\rangle \displaystyle=\left\langle\nabla_{\mathbf{A}}\mathcal{L},-\mathcal{T}\left(\nabla_{\mathbf{A}}\mathcal{L}\right)\right\rangle+\left\langle\nabla_{\mathbf{B}}\mathcal{L},-\mathcal{T}\left(\nabla_{\mathbf{B}}\mathcal{L}\right)\right\rangle \displaystyle=-\|\nabla_{\mathbf{A}}\mathcal{L}\|_{*}-\|\nabla_{\mathbf{B}}\mathcal{L}\|_{*}\leq 0.\qed Lemma D.5 . On SpecGF with \mathcal{T}_{\beta} , the loss is monotonically non-increasing: \displaystyle\frac{d\mathcal{L}(t)}{\mathrm{d}t}\leq 0. The equality holds if and only if \nabla_{\mathbf{A}}\mathcal{L}(t)=\nabla_{\mathbf{B}}\mathcal{L}(t)=0 holds. Proof. We first simplify \left\langle\mathcal{T}_{\beta}(\mathbf{X}),\mathbf{X}\right\rangle for a matrix \mathbf{X} . Let the SVD of \mathbf{X} be \mathbf{U}^{\prime}\mathbf{S}^{\prime}\mathbf{V}^{\prime\top} with singular values \sigma_{1},\ldots,\sigma_{r}\geq 0 . \displaystyle\left\langle\mathcal{T}_{\beta}(\mathbf{X}),\mathbf{X}\right\rangle=\mathrm{tr}\left((\mathbf{U}^{\prime}\mathbf{S}^{\prime}_{\beta}\mathbf{V}^{\prime\top})^{\top}\mathbf{U}^{\prime}\mathbf{S}^{\prime}\mathbf{V}^{\prime\top}\right)=\mathrm{tr}\left(\mathbf{V}^{\prime}\mathbf{S}^{\prime}_{\beta}\mathbf{S}^{\prime}\mathbf{V}^{\prime\top}\right)=\mathrm{tr}\left(\mathbf{S}^{\prime}_{\beta}\mathbf{S}^{\prime}\right)=\sum_{i\in[r]}\frac{\sigma_{i}^{2}}{\sqrt{\sigma_{i}^{2}+\beta}}. We drop the argument t for simplicity. Let \sigma_{i}(\nabla_{\mathbf{A}}\mathcal{L}),\sigma_{i}(\nabla_{\mathbf{B}}\mathcal{L}) be the singular values of \nabla_{\mathbf{A}}\mathcal{L} and \nabla_{\mathbf{B}}\mathcal{L} , respectively. Then: \displaystyle\frac{d\mathcal{L}}{\mathrm{d}t} \displaystyle=\left\langle\nabla_{\mathbf{A}}\mathcal{L},\dot{\mathbf{A}}\right\rangle+\left\langle\nabla_{\mathbf{B}}\mathcal{L},\dot{\mathbf{B}}\right\rangle \displaystyle=\left\langle\nabla_{\mathbf{A}}\mathcal{L},-\mathcal{T}_{\beta}\left(\nabla_{\mathbf{A}}\mathcal{L}\right)\right\rangle+\left\langle\nabla_{\mathbf{B}}\mathcal{L},-\mathcal{T}_{\beta}\left(\nabla_{\mathbf{B}}\mathcal{L}\right)\right\rangle \displaystyle=-\sum_{i\in[r]}\frac{\sigma_{i}(\nabla_{\mathbf{A}}\mathcal{L})^{2}}{\sqrt{\sigma_{i}(\nabla_{\mathbf{A}}\mathcal{L})^{2}+\beta}}-\sum_{i\in[r]}\frac{\sigma_{i}(\nabla_{\mathbf{B}}\mathcal{L})^{2}}{\sqrt{\sigma_{i}(\nabla_{\mathbf{B}}\mathcal{L})^{2}+\beta}}\leq 0.\qed Thus, \mathcal{L}(t) is monotonically non-increasing and lower bounded, hence \mathcal{L}(\infty)\leq\mathcal{L}(0) should exist. Moreover, \displaystyle\left\lVert\mathbf{A}(t)\mathbf{B}(t)\right\rVert_{\mathrm{F}}\leq\left\lVert\mathbf{A}(t)\mathbf{B}(t)-\mathbf{Y}\right\rVert_{\mathrm{F}}+\left\lVert\mathbf{Y}\right\rVert_{\mathrm{F}}=2\sqrt{\mathcal{L}(t)}+\left\lVert\mathbf{Y}\right\rVert_{\mathrm{F}}\leq 2\sqrt{\mathcal{L}(0)}+\left\lVert\mathbf{Y}\right\rVert_{\mathrm{F}}. (27) We now show that SpecGF iterate should either converge or diverge to infinity. Lemma D.6 (Łojasiewicz) . Let F:U\to\mathbb{R} be a real analytic function on an open set U\subset\mathbb{R}^{d} . Then, for every critical point \mathbf{p}\in U of F , there exists a (possibly small) neighborhood W of \mathbf{p} , constants C>0 and b\in[\frac{1}{2},1) such that for all \mathbf{x}\in W , |F(\mathbf{x})-F(\mathbf{p})|^{b}\leq C\|\nabla F(\mathbf{x})\|. (28) Proposition D.7 . Let \mathbf{x}(t) be a differentiable and absolutely continuous curve on \mathbb{R}^{d} and F be a real analytic function on an open set of \mathbb{R}^{d} . Assume that F(\mathbf{x}(t)) is well-defined for all t\geq 0 and the condition \frac{\mathrm{d}F(\mathbf{x}(t))}{\mathrm{d}t}=0\quad\Rightarrow\quad\dot{\mathbf{x}}(t)=0. (29) If there exists a constant c>0 such that \frac{\mathrm{d}F(\mathbf{x}(t))}{\mathrm{d}t}\leq-c\|\nabla F(\mathbf{x}(t))\|\|\dot{\mathbf{x}}(t)\| (30) for all t\geq 0 , then either \mathbf{x}(t) converges in \mathbb{R}^{d} or it diverges to infinity (in norm). Proof. Assume that \|\mathbf{x}(t)\|\not\to\infty as t\to\infty . Then \mathbf{x}(t) has an accumulation point in \mathbb{R}^{n} ; it suffices to prove that the accumulation point is the limit of \mathbf{x}(t) . Denote the accumulation point as \bar{\mathbf{x}} . By continuity of F and monotonicity of F from Equation 30 , F(\mathbf{x}(t))\downarrow F(\bar{\mathbf{x}}) . If there exists a finite time t^{\prime}\geq 0 such that F(\mathbf{x}(t^{\prime}))=F(\bar{\mathbf{x}}) , then F(\mathbf{x}(t))=F(\bar{\mathbf{x}}) for t\geq t^{\prime} , hence \mathbf{x}(t)=\bar{\mathbf{x}} by Equation 29 . From here we assume that F(\mathbf{x}(t))>F(\bar{\mathbf{x}}) for all t . From Lemma D.6 and Equation 30 , we see that \frac{\mathrm{d}F(\mathbf{x}(t))}{\mathrm{d}t}\leq-c\|\nabla F(\mathbf{x}(t))\|\|\dot{\mathbf{x}}(t)\|\leq-\frac{c}{C}|F(\mathbf{x}(t))-F(\bar{\mathbf{x}})|^{b}\|\dot{\mathbf{x}}(t)\|=-\frac{c}{C}\left(F(\mathbf{x}(t))-F(\bar{\mathbf{x}})\right)^{b}\|\dot{\mathbf{x}}(t)\| on some neighborhood W of \bar{\mathbf{x}} with C>0 and b\in[\frac{1}{2},1) . Hence, \displaystyle-\|\dot{\mathbf{x}}(t)\|\geq\frac{C}{c}\frac{\mathrm{d}F(\mathbf{x}(t))}{\mathrm{d}t}\left(F(\mathbf{x}(t))-F(\bar{\mathbf{x}})\right)^{-b}=\frac{C}{c(1-b)}\frac{\mathrm{d}}{\mathrm{d}t}\left(F(\mathbf{x}(t))-F(\bar{\mathbf{x}})\right)^{1-b}. For brevity, let G(t)\triangleq F(\mathbf{x}(t))-F(\bar{\mathbf{x}})>0 . Then G is absolutely continuous and G(t)\downarrow 0 . We also see that -\|\dot{\mathbf{x}}(t)\|\geq\frac{C}{c(1-b)}\frac{\mathrm{d}}{\mathrm{d}t}\left(G(t)^{1-b}\right). (31) We now show that \mathbf{x}(t) eventually moves to the vicinity of \bar{\mathbf{x}} and remains there forever. As W is open, there exists r>0 where B(r)\triangleq\{z\in\mathbb{R}^{n}:\|z-\bar{\mathbf{x}}\|0 . Because \bar{\mathbf{x}} is the accumulation point of \mathbf{x}(t) and G continuously decreases to zero, there exists a time t_{1}\geq 0 such that \|x(t_{1})-\bar{\mathbf{x}}\|<\frac{r}{2},\quad G(t_{1})<\left(\frac{rc(1-b)}{2C}\right)^{\frac{1}{1-b}}. Notice the former implies x(t_{1})\in B(r) and the latter implies \tfrac{C}{c(1-b)}G(t_{1})^{1-b}<\tfrac{r}{2} . Suppose the claim is false, so there exists t_{2}>t_{1} where x(t_{2})\not\in B(r) ; choose the smallest such t_{2} . Then \mathbf{x}(t)\in B(r) for t\in[t_{1},t_{2}) , hence \displaystyle\|x(t_{2})-x(t_{1})\| \displaystyle\leq\int_{t_{1}}^{t_{2}}\|\dot{\mathbf{x}}(t)\|\mathrm{d}t \displaystyle\leq\int_{t_{1}}^{t_{2}}-\frac{C}{c(1-b)}\frac{\mathrm{d}}{\mathrm{d}t}\left(G(t)^{1-b}\right)\mathrm{d}t \displaystyle=\frac{C}{c(1-b)}\left[G(t_{1})^{1-b}-G(t_{2})^{1-b}\right]<\frac{r}{2}. The first inequality and the equality follow from the absolute continuity of \mathbf{x}(t) and G(t) on [t_{1},t_{2}] , respectively. The second inequality stems from Equation 31 . The above yields a contradiction: \|x(t_{2})-\bar{\mathbf{x}}\|\leq\|x(t_{2})-x(t_{1})\|+\|x(t_{1})-\bar{\mathbf{x}}\|\mathbb{R}\times\mathbb{R} . Proof. Note that, as \mathbb{R} is a field, \mathbb{R}[a][b]\cong\mathbb{R}[a,b]\cong\mathbb{R}[b][a] . In addition, \mathbb{R}[x] is a unique factorization domain (UFD) for an indeterminate x . Let P\in\mathbb{R}[a,b] be a polynomial (of finite degree) with real coefficients. Then \partial_{a}P(a(t),b(t))\dot{a}(t)+\partial_{b}P(a(t),b(t))\dot{b}(t)=0. Substitute \dot{a}(t)=-\mathcal{T}_{\beta}(\partial_{a}\mathcal{L}) and \dot{b}(t)=-\mathcal{T}_{\beta}(\partial_{b}\mathcal{L}) . Rearranging the equations gives that, for any a and b , \partial_{a}P(a,b)\times b(ab-\sigma)\sqrt{a^{2}(ab-\sigma)^{2}+\beta}+\partial_{b}P(a,b)\times a(ab-\sigma)\sqrt{b^{2}(ab-\sigma)^{2}+\beta}=0. For simplicity, denote a^{2}(ab-\sigma)^{2}+\beta and b^{2}(ab-\sigma)^{2}+\beta as Q_{(a)} and Q_{(b)} in \mathbb{R}[a,b] , respectively. Then \left(\partial_{a}P(a,b)\right)^{2}\times b^{2}Q_{(a)}=\left(\partial_{b}P(a,b)\right)^{2}\times a^{2}Q_{(b)}. Plugging a=0 gives \left(\partial_{a}P(0,b)\right)^{2}\times b^{2}\beta=0 , which implies that a\>|\>\partial_{a}P(a,b) . Plugging b=0 implies b\>|\>\partial_{b}P(a,b) by analogous reasoning. We see that there exist some polynomials P_{(a)}^{\prime} and P_{(b)}^{\prime} that \partial_{a}P(a,b)=aP_{(a)}^{\prime},\quad\partial_{b}P(a,b)=bP_{(b)}^{\prime}. Thus, for any a and b , P_{(a)}^{\prime 2}Q_{(a)}=P_{(b)}^{\prime 2}Q_{(b)}. As \beta>0 , the discriminant of Q_{(a)} , viewed as a quadratic in b , is (-2a^{3}\sigma)^{2}-4(a^{4})(a^{2}\sigma^{2}+\beta)=-4a^{4}\beta, which is not a perfect square in \mathbb{R}[a] . For a quadratic with a non-perfect square discriminant, it has no roots in \mathbb{R}[a] . \mathbb{R}[a] being UFD allows us to use Gauss’ lemma, hence Q_{(a)} is (irreducible, thus) a prime in \mathbb{R}[a,b] . Similarly, Q_{(b)} is also a prime in \mathbb{R}[a,b] . We now show that Q_{(a)} and Q_{(b)} are coprime. Indeed, for every common factor D of Q_{(a)} and Q_{(b)} , D should also divide b^{2}Q_{(a)}-a^{2}Q_{(b)}=\beta(b^{2}-a^{2})=\beta(b-a)(b+a). Since b-a and b+a are coprime, either D\>|\>(b-a) or D\>|\>(b+a) . However, Q_{(a)}(a,a)=a^{2}(a^{2}-\sigma)^{2}+\beta\not\equiv 0 and Q_{(a)}(a,-a)=a^{2}(-a^{2}-\sigma)^{2}+\beta\not\equiv 0 ; so D=1 . Therefore, we have \displaystyle Q_{(a)}\>|\>P_{(b)}^{\prime 2}\quad \displaystyle\Rightarrow\quad Q_{(a)}\>|\>P_{(b)}^{\prime}, \displaystyle Q_{(b)}\>|\>P_{(a)}^{\prime 2}\quad \displaystyle\Rightarrow\quad Q_{(b)}\>|\>P_{(a)}^{\prime}. Then, there exist some polynomials P_{(a)}^{\prime\prime} and P_{(b)}^{\prime\prime} that P_{(a)}^{\prime}=Q_{(b)}P_{(a)}^{\prime\prime},\quad P_{(b)}^{\prime}=Q_{(a)}P_{(b)}^{\prime\prime}. Thus, P_{(a)}^{\prime\prime 2}Q_{(b)}=P_{(b)}^{\prime\prime 2}Q_{(a)}. Then, with similar argument, there exist some polynomials P_{(a)}^{\prime\prime\prime} and P_{(b)}^{\prime\prime\prime} that P_{(a)}^{\prime\prime}=Q_{(a)}P_{(a)}^{\prime\prime\prime},\quad P_{(b)}^{\prime\prime}=Q_{(b)}P_{(b)}^{\prime\prime\prime}. Thus, P_{(a)}^{\prime\prime\prime 2}Q_{(a)}=P_{(b)}^{\prime\prime\prime 2}Q_{(b)}. The above reasoning implies that P_{(a)}^{\prime} is divisible by Q_{(a)}^{k} for all positive integers k\geq 1 , which can only happen when P_{(a)}^{\prime}\equiv 0 ; we also have P_{(b)}^{\prime}\equiv 0 . Then \partial_{a}P(a,b)\equiv 0 and \partial_{b}P(a,b)\equiv 0 . Therefore, such an invariant P has to be constant. ∎ The similar result is obtained for \mathcal{T} on the set where SpecGF with \mathcal{T} is analytic. Proposition D.9 (Absence of polynomial invariant for \mathcal{T} ) . For SpecGF with \mathcal{T} , there does not exist a non-constant invariant polynomial with real coefficients in a and b on O\triangleq\left\{(a,b):a\neq 0,b\neq 0,ab-\sigma\neq 0\right\}. Proof. Note that on \mathbb{R} , \mathcal{T}(x)=\mathrm{sgn}(x)\quad(x\neq 0), where \mathrm{sgn} is the sign function. Let P be a polynomial (of finite degree) with real coefficients. Then \partial_{a}P(a(t),b(t))\dot{a}(t)+\partial_{b}P(a(t),b(t))\dot{b}(t)=0. Substitute \dot{a}(t)=-\mathcal{T}(\partial_{a}\mathcal{L}) and \dot{b}(t)=-\mathcal{T}(\partial_{b}\mathcal{L}) . Rearranging the equations gives that, for any a and b in O , \partial_{a}P(a,b)\times\mathrm{sgn}\left(b(ab-\sigma)\right)+\partial_{b}P(a,b)\times\mathrm{sgn}\left(a(ab-\sigma)\right)=0. If \mathrm{sgn}(a)=\mathrm{sgn}(b) , we have \partial_{a}P(a,b)+\partial_{b}P(a,b)=0. As the equality holds on the open subset, we actually have \partial_{a}P(a,b)+\partial_{b}P(a,b)\equiv 0 on all of \mathbb{R}\times\mathbb{R} . Analogously, if \mathrm{sgn}(a)=-\mathrm{sgn}(b) , we have \partial_{a}P(a,b)-\partial_{b}P(a,b)=0, so in fact, on all of \mathbb{R}\times\mathbb{R} , \partial_{a}P(a,b)-\partial_{b}P(a,b)\equiv 0. Therefore \partial_{a}P(a,b) and \partial_{b}P(a,b) is identically zero everywhere, implying that such an invariant P has to be constant. ∎ In addition, we show that \mathbf{A}(t)^{\top}\mathbf{A}(t)-\mathbf{B}(t)\mathbf{B}(t)^{\top} is not an invariant for SpecGF with both \mathcal{T} and \mathcal{T}_{\beta} . Proposition D.10 . For both SpecGF with \mathcal{T}_{\beta} and \mathcal{T} , there exists \mathbf{Y} such that \mathbf{A}(t)^{\top}\mathbf{A}(t)-\mathbf{B}(t)\mathbf{B}(t)^{\top} cannot be an invariant. Proof. Fix \sigma>0 and a,b\neq 0 with ab\neq\sigma , |a|\neq|b| . We show that with \mathbf{Y}=\begin{bmatrix}\sigma\mathbf{I}_{r^{\star}}&\bm{0}\\ \bm{0}&\bm{0}\end{bmatrix} for any r^{\star}\geq r , SpecGF with \mathcal{T}_{\beta} cannot have \mathbf{A}(t)^{\top}\mathbf{A}(t)-\mathbf{B}(t)\mathbf{B}(t)^{\top} as its invariant, due to its non-zero time derivative at (\mathbf{A},\mathbf{B})=\left(\begin{bmatrix}a\mathbf{I}_{r}\\ \bm{0}\end{bmatrix},\begin{bmatrix}b\mathbf{I}_{r}&\bm{0}\\ \end{bmatrix}\right) Compute the gradient at (\mathbf{A},\mathbf{B}) : \displaystyle\nabla_{\mathbf{A}}\mathcal{L}(\mathbf{A},\mathbf{B}) \displaystyle=\begin{bmatrix}b(ab-\sigma)\mathbf{I}_{r}\\ \bm{0}\end{bmatrix}\triangleq\begin{bmatrix}a^{\prime}\mathbf{I}_{r}\\ \bm{0}\end{bmatrix} \displaystyle\nabla_{\mathbf{B}}\mathcal{L}(\mathbf{A},\mathbf{B}) \displaystyle=\begin{bmatrix}a(ab-\sigma)\mathbf{I}_{r}&\bm{0}\end{bmatrix}\triangleq\begin{bmatrix}b^{\prime}\mathbf{I}_{r}&\bm{0}\end{bmatrix}. Then, for SpecGF with \mathcal{T}_{\beta} \displaystyle\dot{\mathbf{A}}=-\mathcal{T}_{\beta}\left(\nabla_{\mathbf{A}}\mathcal{L}(\mathbf{A},\mathbf{B})\right) \displaystyle=\begin{bmatrix}-\frac{a^{\prime}}{\sqrt{a^{\prime 2}+\beta}}\mathbf{I}_{r}\\ \bm{0}\end{bmatrix} \displaystyle\dot{\mathbf{B}}=-\mathcal{T}_{\beta}\left(\nabla_{\mathbf{B}}\mathcal{L}(\mathbf{A},\mathbf{B})\right) \displaystyle=\begin{bmatrix}-\frac{b^{\prime}}{\sqrt{b^{\prime 2}+\beta}}\mathbf{I}_{r}&\bm{0}\end{bmatrix}. Hence, the time derivative of \mathbf{A} and \mathbf{B} translate to that of a and b , respectively. We obtain \displaystyle\frac{\mathrm{d}}{\mathrm{d}t}(\mathbf{A}^{\top}\mathbf{A}-\mathbf{B}\mathbf{B}^{\top}) \displaystyle=\frac{\mathrm{d}}{\mathrm{d}t}(a^{2}-b^{2})\mathbf{I}_{r} \displaystyle=-2\left(a\frac{a^{\prime}}{\sqrt{a^{\prime 2}+\beta}}-b\frac{b^{\prime}}{\sqrt{b^{\prime 2}+\beta}}\right)\mathbf{I}_{r} \displaystyle=-2ab(ab-\sigma)\left(\frac{1}{\sqrt{b^{2}(ab-\sigma)^{2}+\beta}}-\frac{1}{\sqrt{a^{2}(ab-\sigma)^{2}+\beta}}\right)\mathbf{I}_{r}\neq\bm{0}. The same setting results in the same result for SpecGF with \mathcal{T} . For SpecGF with \mathcal{T} \displaystyle\dot{\mathbf{A}}=-\mathcal{T}\left(\nabla_{\mathbf{A}}\mathcal{L}(\mathbf{A},\mathbf{B})\right) \displaystyle=\begin{bmatrix}-\mathrm{sgn}(a^{\prime})\mathbf{I}_{r}\\ \bm{0}\end{bmatrix} \displaystyle\dot{\mathbf{B}}=-\mathcal{T}\left(\nabla_{\mathbf{B}}\mathcal{L}(\mathbf{A},\mathbf{B})\right) \displaystyle=\begin{bmatrix}-\mathrm{sgn}(b^{\prime})\mathbf{I}_{r}&\bm{0}\end{bmatrix}. Again, the time derivative of \mathbf{A} and \mathbf{B} translate to that of a and b , respectively. We obtain \displaystyle\frac{\mathrm{d}}{\mathrm{d}t}(\mathbf{A}^{\top}\mathbf{A}-\mathbf{B}\mathbf{B}^{\top}) \displaystyle=\frac{\mathrm{d}}{\mathrm{d}t}(a^{2}-b^{2})\mathbf{I}_{r} \displaystyle=-2\left(a\cdot\mathrm{sgn}(a^{\prime})-b\cdot\mathrm{sgn}(b^{\prime})\right)\mathbf{I}_{r} \displaystyle=-2\mathrm{sgn}(ab-\sigma)\left(a\cdot\mathrm{sgn}(b)-b\cdot\mathrm{sgn}(a)\right)\mathbf{I}_{r}\neq\bm{0}.\qed D.3 Saddle Point Avoidance 6 characterize the critical points of the loss L^{N}(\mathbf{W}_{1},\ldots,\mathbf{W}_{N})=\frac{1}{2}\left\lVert\mathbf{Y}-\mathbf{W}_{N}\cdots\mathbf{W}_{1}\mathbf{X}\right\rVert_{\mathrm{F}}^{2}, (32) where \mathbf{W}_{j}\in\mathbb{R}^{d_{j}\times d_{j-1}} for d_{0}=d_{x} , d_{N}=d_{y} and \mathbf{X}\in\mathbb{R}^{d_{x}\times m} and \mathbf{Y}\in\mathbb{R}^{d_{y}\times m} . They analyzed whether the critical points are the minima or strict saddle points through a closely related loss L^{1}(\mathbf{W})=\frac{1}{2}\left\lVert\mathbf{Y}-\mathbf{W}\mathbf{X}\right\rVert_{\mathrm{F}}^{2}, where \mathbf{W}\in\mathbb{R}^{d_{y}\times d_{x}} . We summarize their result, especially for N=2 case. Proposition D.11 ( ( 6 ) , Proposition 6.6, 6.9, 6.11) . Let r\triangleq\min\{d_{0},d_{1},d_{2}\} . Assume that \mathbf{X}\mathbf{X}^{\top} has full rank d_{0} , \mathbf{Y} is of rank q , and \mathbf{W}_{k} is the global minimum of L^{1} restricted to a set of matrices whose rank is at most k . Let K be the rank of \mathbf{W}_{2}\mathbf{W}_{1} . 1. \mathbf{W}_{k} is the best rank- k approximation of \mathbf{Y} . 2. If K=\min\{r,q\} , every critical point (\mathbf{W}_{1},\mathbf{W}_{2}) that satisfies \mathbf{W}_{2}\mathbf{W}_{1}=\mathbf{W}_{K} is the global minimum of L^{2} . If \mathbf{W}_{2}\mathbf{W}_{1}\neq\mathbf{W}_{K} , then (\mathbf{W}_{1},\mathbf{W}_{2}) is a strict saddle point. 3. Any other critical point (\mathbf{W}_{1},\mathbf{W}_{2}) , including the origin, is a strict saddle point. We say that a critical point x_{c} of a twice continuously differentiable function f is a strict saddle point if the Hessian of f at x_{c} has a negative eigenvalue. Proposition D.11 implies that every critical point of L^{2} is exactly either a global minimum or a strict saddle point. Notice that if \mathbf{X} is the identity matrix and N=2 , then the setting of 6 exactly matches with ours. They also proved that the set of initial points such that the corresponding plain gradient flow for Equation 32 converges to a strict saddle point has measure zero ( 6 ) [Theorem 6.3]. We also show that this holds for SpecGF with \mathcal{T}_{\beta} . Then, if SpecGF converges, it almost surely converges to the global minima, or the best low-rank approximation of \mathbf{Y} . To this end, we employ a proposition from 11 that claims the similar result. Proposition D.12 ( ( 11 ) , Proposition 2.5) . Let f:\mathbb{R}^{d}\to\mathbb{R}^{d} . Suppose that there exist open sets V\subseteq U\subseteq\mathbb{R}^{d} such that the complement of V have Lebesgue measure zero and f is continuously differentiable on U with a locally Lipschitz continuous Jacobian, which is non-degenerate on V . Let \mathcal{S}\subseteq\{x\in U:f(x)=x\} and assume for all x\in\mathcal{S} that f^{\prime}(x) is symmetric and has an eigenvalue of absolute value strictly greater than 1. Then, the set \{x\in\mathbb{R}^{d}:\lim_{k\to\infty}f^{k}(x)\in\mathcal{S}\} has Lebesgue measure 0. There, f^{k}(x) for k\in\mathbb{N}_{0} is inductively defined as f^{0}\equiv\mathrm{id} and f^{k+1}=f\circ f^{k} . Theorem D.13 . SpecGF with \mathcal{T}_{\beta} almost surely avoids strict saddle points. Specifically, if r\leq\mathrm{rank}(\mathbf{Y}) , then SpecGF with \mathcal{T}_{\beta} almost surely converges to the best rank- r approximation of \mathbf{Y} if convergence is guaranteed. Proof. We first verify that the conditions of Proposition D.12 are met for SpecGF with \mathcal{T}_{\beta} and then extend the statement from the discrete to the continuous version. For simplicity, let \mathbf{x}(t)\triangleq(\mathbf{A}(t),\mathbf{B}(t)) and f_{\beta}(\mathbf{x})\triangleq\left(-\mathcal{T}_{\beta}(\nabla_{\mathbf{A}}\mathcal{L}(\mathbf{x})),-\mathcal{T}_{\beta}(\nabla_{\mathbf{B}}\mathcal{L}(\mathbf{x}))\right) . Denote the solution of \dot{\mathbf{x}}(t)=f_{\beta}(\mathbf{x}),\quad\mathbf{x}(0)\triangleq\mathbf{x}_{0}. as \phi(t) ; when the initial point matters, we use \varphi_{t}(\mathbf{x}_{0})=\phi(t). We note that \varphi_{t}(\mathbf{x}_{0}) , and also \phi(t) , satisfy the semigroup property: \varphi_{t^{\prime}+t}(\mathbf{x}_{0})=\varphi_{t^{\prime}}(\varphi_{t}(\mathbf{x}_{0})). The Jacobian matrix of \varphi_{t}(\mathbf{x}_{0}) , D\varphi_{t}(\mathbf{x}_{0}) , satisfies the following with Jacobian matrix of f_{\beta} , Df_{\beta} : \frac{\mathrm{d}}{\mathrm{d}t}D\varphi_{t}(\mathbf{x}_{0})=Df_{\beta}\left(\varphi_{t}(\mathbf{x}_{0})\right)D\varphi_{t}(\mathbf{x}_{0}),\quad D\varphi_{0}(\mathbf{x}_{0})=\mathbf{I}. Liouville’s theorem yields: \left\lvert D\varphi_{t}(\mathbf{x}_{0})\right\rvert=\left\lvert D\varphi_{0}(\mathbf{x}_{0})\right\rvert\exp\left(\int_{0}^{t}\mathrm{tr}Df_{\beta}\left(\varphi_{\tau}(\mathbf{x}_{0})\right)\mathrm{d}\tau\right). Analyticity of f_{\beta} makes \left\lvert D\varphi_{t}(\mathbf{x}_{0})\right\rvert nonzero for all \mathbf{x}_{0} and all t\geq 0 , hence the conditions meet with V=U=\mathbb{R}^{d} . Now take \mathcal{S} as the set of strict saddle points. It is straightforward to see that \mathcal{T}_{\beta} is (Fréchet) differentiable at 0 with D\mathcal{T}_{\beta}(0)[\bm{\Delta}]=\beta^{-1/2}\bm{\Delta}, (33) for all matrix \bm{\Delta} . It suffices to prove that \lim_{h\to 0}\frac{\left\lVert\mathcal{T}_{\beta}(h\bm{\Delta})-\mathcal{T}_{\beta}(0)-h\beta^{-\frac{1}{2}}\bm{\Delta}\right\rVert_{\mathrm{F}}}{|h|}=0, or \lim_{h\to 0}\left\lVert\frac{\mathcal{T}_{\beta}(h\bm{\Delta})}{h}-\beta^{-\frac{1}{2}}\bm{\Delta}\right\rVert_{\mathrm{F}}=0. Notice that, for s(x)=x^{-\frac{1}{2}} on [\beta,\infty) and c>0 , |s(x+c)-s(x)|=|s^{\prime}(x+rc)|c\leq\frac{c}{2\beta^{\frac{3}{2}}} (34) for some r\in(0,1) . Applying Equation 34 to the eigenvalues of \bm{\Delta}^{\top}\bm{\Delta} yields \displaystyle\left\lVert\frac{\mathcal{T}_{\beta}(h\bm{\Delta})}{h}-\beta^{-\frac{1}{2}}\bm{\Delta}\right\rVert_{\mathrm{F}} \displaystyle=\left\lVert\bm{\Delta}\left(h^{2}\bm{\Delta}^{\top}\bm{\Delta}+\beta\mathbf{I}\right)^{-\frac{1}{2}}-\beta^{-\frac{1}{2}}\bm{\Delta}\right\rVert_{\mathrm{F}} \displaystyle\leq\left\lVert\bm{\Delta}\right\rVert_{\mathrm{F}}\lVert\left(h^{2}\bm{\Delta}^{\top}\bm{\Delta}+\beta\mathbf{I}\right)^{-\frac{1}{2}}-\beta^{-\frac{1}{2}}\mathbf{I}\rVert_{2} \displaystyle\leq\left\lVert\bm{\Delta}\right\rVert_{\mathrm{F}}\frac{h^{2}\lVert\bm{\Delta}\rVert_{2}^{2}}{2\beta^{\frac{3}{2}}}. Equation 33 directly yields that Df_{\beta}(\bar{\mathbf{x}})=-\beta^{-1/2}\nabla^{2}\mathcal{L}(\bar{\mathbf{x}}) (35) and it is symmetric for all \bar{\mathbf{x}}\in\mathcal{S} (in fact, this holds at any critical point). As \bar{\mathbf{x}} is an equilibrium point, \frac{\mathrm{d}}{\mathrm{d}t}D\varphi_{t}(\bar{\mathbf{x}})=Df_{\beta}\left(\varphi_{t}(\bar{\mathbf{x}})\right)D\varphi_{t}(\bar{\mathbf{x}})=Df_{\beta}\left(\bar{\mathbf{x}}\right)D\varphi_{t}(\bar{\mathbf{x}}). Thus D\varphi_{t}(\bar{\mathbf{x}})=e^{tDf_{\beta}\left(\bar{\mathbf{x}}\right)}. Since \bar{\mathbf{x}} is a strict saddle, the Hessian of \mathcal{L} at \bar{\mathbf{x}} has a negative eigenvalue. By Equation 35 , Df_{\beta}\left(\bar{\mathbf{x}}\right) has a positive eigenvalue, and hence D\varphi_{t}(\bar{\mathbf{x}}) has an eigenvalue whose (absolute) value is strictly greater than 1 at any t\geq 0 . The remaining part, the extension from discrete to continuous, is simple. Fix any time T>0 . By the semigroup property, \displaystyle\left\{\mathbf{x}_{0}:\lim_{t\to\infty}\varphi_{t}(\mathbf{x}_{0})\in\mathcal{S}\right\} \displaystyle=\bigcup_{\bar{\mathbf{x}}\in\mathcal{S}}\left\{\mathbf{x}_{0}:\lim_{t\to\infty}\varphi_{t}(\mathbf{x}_{0})=\bar{\mathbf{x}}\right\} \displaystyle\subseteq\bigcup_{\bar{\mathbf{x}}\in\mathcal{S}}\left\{\mathbf{x}_{0}:\lim_{k\to\infty}\varphi_{kT}(\mathbf{x}_{0})=\bar{\mathbf{x}}\right\} \displaystyle=\left\{\mathbf{x}_{0}:\lim_{k\to\infty}(\varphi_{T})^{k}(\mathbf{x}_{0})\in\mathcal{S}\right\}. The proof is concluded since the subset of a Lebesgue measure zero set is again of Lebesgue measure zero. ∎ D.4 Stability Analysis of SpecGF Recall that, from Section D.1 , \displaystyle\mathcal{T}_{\beta}(\mathbf{X}) \displaystyle=\mathbf{U}\mathbf{S}_{\beta}\mathbf{V}^{\top},\quad\mathbf{S}_{\beta}\triangleq\mathrm{Diag}\left\{\frac{\sigma_{i}}{\sqrt{\sigma_{i}^{2}+\beta}}\right\}_{i\in[r]}. Thus, \left\langle\mathbf{X},\mathcal{T}_{\beta}(\mathbf{X})\right\rangle\geq\sqrt{\beta}\left\lVert\mathcal{T}_{\beta}(\mathbf{X})\right\rVert_{\mathrm{F}}^{2}\quad\Rightarrow\quad\left\lVert\mathcal{T}_{\beta}(\mathbf{X})\right\rVert_{\mathrm{F}}\leq\beta^{-\frac{1}{2}}\left\lVert\mathbf{X}\right\rVert_{\mathrm{F}}. (36) Proposition D.14 ( ( 9 ) , Theorem 5.2) . Consider the system of differential equations \dot{\mathbf{x}}(t)=f(\mathbf{x}(t)) (37) with f:\mathbb{R}^{d}\to\mathbb{R}^{d} that yields a unique \mathcal{C}^{1} solution for every initial condition \mathbf{x}(0) . Let \mathbf{x}\in\mathbb{R}^{d} , and suppose there exists a continuous function V:\mathcal{V}\to\mathbb{R} defined on an open neighborhood \mathcal{V} of \mathbf{x} , and c>0 such that c\dot{V}(\mathbf{z})+\lVert f(\mathbf{z})\rVert\leq 0 (38) is satisfied for every \mathbf{z}\in\mathcal{V} . If \mathbf{x} is a local minimizer of V, then \mathbf{x} is a Lyapunov stable equilibrium. We now prove that every (global) minimum of the loss is Lyapunov stable. Theorem D.15 (Lyapunov stability of minima) . Every global minimum of \mathcal{L}(\mathbf{A},\mathbf{B}) is Lyapunov stable for SpecGF with \mathcal{T}_{\beta} . Proof. Recall Lemma D.6 : at each global minima \mathbf{x}_{*} of \mathcal{L} , there exist a neighborhood W of \mathbf{x}_{*} , constants C>0 and b\in[\frac{1}{2},1) such that for all \mathbf{x}\in W , |\mathcal{L}(\mathbf{x})-\mathcal{L}(\mathbf{x}_{*})|^{b}\leq C\|\nabla\mathcal{L}(\mathbf{x})\|. Note that \mathcal{L}(\mathbf{x}_{*})\triangleq\mathcal{L}_{*} for any global minima \mathbf{x}_{*} . Without loss of generalization, take W as an open ball centered at \mathbf{x}_{*} and of (finite) positive radius. On W , define V(\mathbf{x})\triangleq\left(\mathcal{L}(\mathbf{x})-\mathcal{L}_{*}\right)^{1-b}. (39) Then V is continuous, nonnegative, minimized at \mathbf{x}_{*} and \dot{V}(\mathbf{x})=(1-b)\left(\mathcal{L}(\mathbf{x})-\mathcal{L}_{*}\right)^{-b}\dot{\mathcal{L}}(\mathbf{x}). On W , \nabla_{\mathbf{A}}\mathcal{L} and \nabla_{\mathbf{B}}\mathcal{L} are both bounded; take the bound as C_{W} so that \left\lVert\nabla_{\mathbf{A}}\mathcal{L}\right\rVert_{\mathrm{F}}\leq C_{W},\quad\left\lVert\nabla_{\mathbf{B}}\mathcal{L}\right\rVert_{\mathrm{F}}\leq C_{W}. Observe \displaystyle-\dot{\mathcal{L}} \displaystyle=\left\langle\nabla_{\mathbf{A}}\mathcal{L},\mathcal{T}_{\beta}(\nabla_{\mathbf{A}}\mathcal{L})\right\rangle+\left\langle\nabla_{\mathbf{B}}\mathcal{L},\mathcal{T}_{\beta}(\nabla_{\mathbf{B}}\mathcal{L})\right\rangle \displaystyle=\sum_{i}\frac{\sigma_{i}^{2}(\nabla_{\mathbf{A}}\mathcal{L})}{\sqrt{\sigma_{i}^{2}(\nabla_{\mathbf{A}}\mathcal{L})+\beta}}+\sum_{j}\frac{\sigma_{j}^{2}(\nabla_{\mathbf{B}}\mathcal{L})}{\sqrt{\sigma_{j}^{2}(\nabla_{\mathbf{B}}\mathcal{L})+\beta}} \displaystyle\geq\sum_{i}\frac{\sigma_{i}^{2}(\nabla_{\mathbf{A}}\mathcal{L})}{\sqrt{C_{W}^{2}+\beta}}+\sum_{j}\frac{\sigma_{j}^{2}(\nabla_{\mathbf{B}}\mathcal{L})}{\sqrt{C_{W}^{2}+\beta}} \displaystyle=\frac{1}{\sqrt{C_{W}^{2}+\beta}}\left\lVert\nabla\mathcal{L}\right\rVert_{\mathrm{F}}^{2}. In addition, for the same f_{\beta} in the proof of Theorem D.13 , \left\lVert f_{\beta}(\mathbf{x})\right\rVert_{\mathrm{F}}=\sqrt{\left\lVert\mathcal{T}_{\beta}(\nabla_{\mathbf{A}}\mathcal{L}(\mathbf{x}))\right\rVert_{\mathrm{F}}^{2}+\left\lVert\mathcal{T}_{\beta}(\nabla_{\mathbf{B}}\mathcal{L}(\mathbf{x}))\right\rVert_{\mathrm{F}}^{2}}\leq\sqrt{\beta^{-1}\left(\left\lVert\nabla_{\mathbf{A}}\mathcal{L}(\mathbf{x})\right\rVert_{\mathrm{F}}^{2}+\left\lVert\nabla_{\mathbf{B}}\mathcal{L}(\mathbf{x})\right\rVert_{\mathrm{F}}^{2}\right)}=\beta^{-\frac{1}{2}}\left\lVert\nabla\mathcal{L}(\mathbf{x})\right\rVert_{\mathrm{F}}. Hence, on W , \displaystyle-\dot{V}(\mathbf{x}) \displaystyle=-(1-b)\left(\mathcal{L}(\mathbf{x})-\mathcal{L}_{*}\right)^{-b}\dot{\mathcal{L}}(\mathbf{x}) \displaystyle\geq(1-b)\left(\mathcal{L}(\mathbf{x})-\mathcal{L}_{*}\right)^{-b}\frac{1}{\sqrt{C_{W}^{2}+\beta}}\left\lVert\nabla\mathcal{L}(\mathbf{x})\right\rVert_{\mathrm{F}}^{2} \displaystyle\geq\frac{1-b}{C\sqrt{C_{W}^{2}+\beta}}\left\lVert\nabla\mathcal{L}(\mathbf{x})\right\rVert_{\mathrm{F}} \displaystyle\geq\frac{1-b}{C\sqrt{C_{W}^{2}+\beta}}\beta^{\frac{1}{2}}\left\lVert f_{\beta}(\mathbf{x})\right\rVert_{\mathrm{F}}. c=\frac{C\sqrt{C_{W}^{2}+\beta}}{(1-b)\sqrt{\beta}} gives the desired inequality. ∎ Therefore, the a.s. local convergence of SpecGF with \mathcal{T}_{\beta} towards global minima is guaranteed. Conjecture: The basin of attraction is large. For now, let \rho_{W} be the radius of W . Fix \epsilon>0 and take \rho=\min\{\rho_{W},\epsilon\}>0 . We will find \delta>0 for the Lyapunov stability of each minimizer \mathbf{x}_{*}=(\mathbf{A}_{*},\mathbf{B}_{*}) . Let \mathbf{z}=(\mathbf{A},\mathbf{B}) and assume \left\lVert\mathbf{z}-\mathbf{x}_{*}\right\rVert_{\mathrm{F}}\leq\delta . We have \left\lVert\Delta\mathbf{A}\right\rVert_{\mathrm{F}}\triangleq\left\lVert\mathbf{A}-\mathbf{A}_{*}\right\rVert_{\mathrm{F}}\leq\delta,\quad\left\lVert\Delta\mathbf{B}\right\rVert_{\mathrm{F}}\triangleq\left\lVert\mathbf{B}-\mathbf{B}_{*}\right\rVert_{\mathrm{F}}\leq\delta. Then, for some constants c^{\prime}>0 (may depend on \mathbf{x}_{*} ) \displaystyle\left\lVert\mathbf{A}\mathbf{B}-\mathbf{Y}\right\rVert_{\mathrm{F}} \displaystyle=\left\lVert\Delta\mathbf{A}\mathbf{B}_{*}+\mathbf{A}_{*}\Delta\mathbf{B}+\Delta\mathbf{A}\Delta\mathbf{B}\right\rVert_{\mathrm{F}} \displaystyle\leq\left\lVert\Delta\mathbf{A}\mathbf{B}_{*}\right\rVert_{\mathrm{F}}+\left\lVert\mathbf{A}_{*}\Delta\mathbf{B}\right\rVert_{\mathrm{F}}+\left\lVert\Delta\mathbf{A}\Delta\mathbf{B}\right\rVert_{\mathrm{F}} \displaystyle\leq\delta\left(\lVert\mathbf{A}_{*}\rVert_{2}+\lVert\mathbf{B}_{*}\rVert_{2}\right)+\delta^{2} \displaystyle\leq c^{\prime}\max\{\delta,\delta^{2}\}. (e.g., take c^{\prime}=\lVert\mathbf{A}_{*}\rVert_{2}+\lVert\mathbf{B}_{*}\rVert_{2}+1 .) Hence, \mathcal{L}(\mathbf{z})=\frac{1}{2}\left\lVert\mathbf{A}\mathbf{B}-\mathbf{Y}\right\rVert_{\mathrm{F}}^{2}\leq\frac{1}{2}c^{\prime 2}\max\{\delta^{2},\delta^{4}\}\quad\Rightarrow\quad V(\mathbf{z})\leq\left(\frac{1}{2}c^{\prime 2}\max\{\delta^{2},\delta^{4}\}-\mathcal{L}_{*}\right)^{1-b}. In the proof of 9 , two conditions are required: \left\lVert\mathbf{z}-\mathbf{x}_{*}\right\rVert_{\mathrm{F}}\leq\frac{\rho}{4},\quad|V(\mathbf{z})-V(\mathbf{x}_{*})|\leq\frac{\rho}{4c}, where c=\frac{C\sqrt{C_{W}^{2}+\beta}}{(1-b)\sqrt{\beta}} . It suffices that \delta\leq\frac{\rho}{4},\quad\left(\frac{1}{2}c^{\prime 2}\max\{\delta^{2},\delta^{4}\}-\mathcal{L}_{*}\right)^{1-b}\leq\frac{\rho}{4c}, which translates to \delta(\epsilon,\mathbf{x}_{*})=\min\left\{\frac{\rho}{4},\sqrt{\frac{2}{c^{\prime 2}}\left(\left(\frac{\rho}{4c}\right)^{\frac{1}{1-b}}+\mathcal{L}_{*}\right)},\sqrt[4]{\frac{2}{c^{\prime 2}}\left(\left(\frac{\rho}{4c}\right)^{\frac{1}{1-b}}+\mathcal{L}_{*}\right)}\right\},\quad\rho=\min\{\rho_{W},\epsilon\}. (40) We conjecture that \rho_{W} is sufficiently large for each and every minimizer \mathbf{x}_{*} . If then, \delta increases as \epsilon increases, and there is a possibility that initial points near (but not exactly) the origin fall in some basin of attraction. We empirically checked that the basin of attraction is sufficiently large. Figure 5 : Lyapunov stability of global minima. Left : Loss trajectory. Right : Decrement of the distance from global minima. The attraction of the basin of some global minima is sufficiently large that, empirically, the origin lies in there. We set \mathbf{Y}\in\mathbb{R}^{9\times 9} with r=4 and \bm{\Sigma}=\mathrm{Diag}(1,0.5,0.2,0.05) , \mathbf{A}\in\mathbb{R}^{9\times 4} , and \mathbf{B}\in\mathbb{R}^{4\times 9} . We first sample 20 random initial points \mathbf{B}_{i}(0) with i.i.d. Gaussian entries of scale \gamma=5\times 10^{-4} ; \mathbf{A}_{i}(0)=\bm{0} . We use a fixed learning rate of \eta=10^{-4} . We ran SpecGD with \mathcal{T}_{\beta} starting from (\mathbf{A}_{i}(0),\mathbf{B}_{i}(0)) until convergence. For all i\in[20] , SpecGD converged to global minima; see the left plot of Figure 6 . Name those global minima as (\mathbf{A}_{i}^{*},\mathbf{B}_{i}^{*}) . For each i , we sample 50 random directions (\Delta\mathbf{A}_{ij},\Delta\mathbf{B}_{ij}) from i.i.d. Gaussian entries, with magnitude equal to the Euclidean distance between (\mathbf{A}_{i}(0),\mathbf{B}_{i}(0)) and (\mathbf{A}_{i}^{*},\mathbf{B}_{i}^{*}) . Denote the initial points as (\mathbf{A}_{ij}(0),\mathbf{B}_{ij}(0))=(\mathbf{A}_{i}^{*},\mathbf{B}_{i}^{*})+(\Delta\mathbf{A}_{ij},\Delta\mathbf{B}_{ij}) . We ran 20\times 50=1000 SpecGD experiments and recorded (i) loss trajectory \mathcal{L}(t) and (ii) decrement of the distance from global minima, \mathrm{dist}((\mathbf{A}_{ij}(0),\mathbf{B}_{ij}(0)),(\mathbf{A}_{i}^{*},\mathbf{B}_{i}^{*}))-\mathrm{dist}((\mathbf{A}_{ij}(t),\mathbf{B}_{ij}(t)),(\mathbf{A}_{i}^{*},\mathbf{B}_{i}^{*})) ; \mathrm{dist}(\mathbf{x},\mathbf{y}) stands for the Euclidean distance between \mathbf{x} and \mathbf{y} . The recorded values are plotted in Figure 5 . Figure 6 : Loss trajectories started at (\mathbf{A}_{i}(0),\mathbf{B}_{i}(0)) . We see that 1) all SpecGF simulation initialized at (\mathbf{A}_{ij}(0),\mathbf{B}_{ij}(0)) converged to the global minima and 2) the decrement is positive, supporting the conjecture that the near-origin points lies in a basin of attraction of some global minima. D.5 Convergence Rate of SpecGF Here, we study the convergence rate of SpecGF with \mathcal{T}_{\beta} , assuming the convergence itself. This analysis closely follows that of Proposition D.7 ; SpecGF iterate will get close to some critical point of \mathcal{L} after a finite time, where the Łojasiewicz gradient inequality holds. On the vicinity of that critical point, the rate of convergence is explicitly derived. Let (\mathbf{A}(t),\mathbf{B}(t))\to(\bar{\mathbf{A}},\bar{\mathbf{B}}) . Then \mathcal{T}_{\beta}(\nabla_{\mathbf{A}}\mathcal{L}(t)),\mathcal{T}_{\beta}(\nabla_{\mathbf{A}}\mathcal{L}(t))\to 0 , thus \nabla_{\mathbf{A}}\mathcal{L}(t),\nabla_{\mathbf{B}}\mathcal{L}(t)\to 0 . This implies that there exists some time t^{\prime}>0 such that for all t\geq t^{\prime} , \lVert\nabla_{\mathbf{A}}\mathcal{L}(t)\rVert_{2}\leq\sqrt{\beta},\quad\lVert\nabla_{\mathbf{B}}\mathcal{L}(t)\rVert_{2}\leq\sqrt{\beta}. In addition, let W be the neighborhood of (\bar{\mathbf{A}},\bar{\mathbf{B}}) where the Łojasiewicz gradient inequality hold: there also exist constants C>0 and b\in[\frac{1}{2},1) such that, for all (\mathbf{A},\mathbf{B})\in W , |\mathcal{L}(\mathbf{A},\mathbf{B})-\mathcal{L}(\bar{\mathbf{A}},\bar{\mathbf{B}})|^{b}\leq C\|\nabla\mathcal{L}(\mathbf{A},\mathbf{B})\|. We know that there exists some time t^{\prime\prime}>0 such that for all t\geq t^{\prime\prime} , (\mathbf{A}(t),\mathbf{B}(t))\in W . Take T\triangleq\max\{t^{\prime},t^{\prime\prime}\}>0 . We briefly state the helper lemma for deriving the convergence rate. Lemma D.16 . For all matrix \mathbf{M} such that \lVert\mathbf{M}\rVert_{2}\leq\sqrt{\beta} , \left\langle\mathbf{M},\mathcal{T}_{\beta}(\mathbf{M})\right\rangle\geq\frac{1}{\sqrt{2\beta}}\left\lVert\mathbf{M}\right\rVert_{\mathrm{F}}^{2}. Proof. \left\langle\mathbf{M},\mathcal{T}_{\beta}(\mathbf{M})\right\rangle=\sum_{i}\frac{\sigma_{i}(\mathbf{M})^{2}}{\sqrt{\sigma_{i}(\mathbf{M})^{2}+\beta}}\geq\sum_{i}\frac{\sigma_{i}(\mathbf{M})^{2}}{\sqrt{2\beta}}=\frac{1}{\sqrt{2\beta}}\left\lVert\mathbf{M}\right\rVert_{\mathrm{F}}^{2}. ∎ Observe that, for t\geq T , \displaystyle\frac{d}{\mathrm{d}t}\mathcal{L}(t) \displaystyle=\left\langle\nabla_{\mathbf{A}}\mathcal{L}(t),-\mathcal{T}_{\beta}\left(\nabla_{\mathbf{A}}\mathcal{L}(t)\right)\right\rangle+\left\langle\nabla_{\mathbf{B}}\mathcal{L}(t),-\mathcal{T}_{\beta}\left(\nabla_{\mathbf{B}}\mathcal{L}(t)\right)\right\rangle \displaystyle\leq-\frac{1}{\sqrt{2\beta}}\left(\left\lVert\nabla_{\mathbf{A}}\mathcal{L}(t)\right\rVert_{\mathrm{F}}^{2}+\left\lVert\nabla_{\mathbf{B}}\mathcal{L}(t)\right\rVert_{\mathrm{F}}^{2}\right)=-\frac{1}{\sqrt{2\beta}}\left\lVert\nabla\mathcal{L}(t)\right\rVert_{\mathrm{F}}^{2} \displaystyle\leq-\frac{1}{C^{2}\sqrt{2\beta}}(\mathcal{L}(t)-\mathcal{L}(\bar{\mathbf{A}},\bar{\mathbf{B}}))^{2b}. Let \mathcal{E}(\mathbf{A},\mathbf{B})\triangleq\mathcal{L}(\mathbf{A},\mathbf{B})-\mathcal{L}(\bar{\mathbf{A}},\bar{\mathbf{B}})\geq 0 ; we use simplifed notation \mathcal{E}(t)\triangleq\mathcal{L}(t)-\mathcal{L}(\bar{\mathbf{A}},\bar{\mathbf{B}})\geq 0 . Then for t\geq T , \dot{\mathcal{E}}(t)\leq-C(\beta)\mathcal{E}(t)^{2b} , where C(\beta)=\frac{1}{C^{2}\sqrt{2\beta}} . The convergence rate of \mathcal{E} differs by b . In our case, we established the almost sure convergence towards global minima (under assuming convergence), hence assume (\bar{\mathbf{A}},\bar{\mathbf{B}}) is the global minimum. Then, \mathcal{E} is the Morse-Bott function, and thus the (optimal) exponent b is exactly \frac{1}{2} ( 13 ) . Then, \dot{\mathcal{E}}(t)\leq-C(\beta)\mathcal{E}(t), or \mathcal{E}(t)\leq\mathcal{E}(T)e^{-C(\beta)(t-T)}. Therefore, \mathcal{E}(t) exponentially decreases to 0. It remains to show that \mathcal{E}(\mathbf{A},\mathbf{B}) is the Morse-Bott function. Assumption D.17 . The nonzero singular values of \mathbf{Y} are mutually distinct. Recall that \bar{\mathbf{A}}\bar{\mathbf{B}} is the best rank- r approximation of \mathbf{Y} ( 6 ) . We denote \bar{\mathbf{Y}}_{r}\triangleq\bar{\mathbf{A}}\bar{\mathbf{B}} for simplicity. Furthermore, denote \bar{\mathbf{Y}}_{r}=\mathbf{U}_{r}\bm{\Sigma}_{r}\mathbf{V}_{r}^{\top},\quad\mathbf{Y}-\bar{\mathbf{Y}}_{r}\triangleq\mathbf{U}_{\perp}\bm{\Sigma}_{\perp}\mathbf{V}_{\perp}^{\top}. [\mathbf{U}_{r}|\mathbf{U}_{\perp}] forms a m\times m orthogonal matrix; similarly, [\mathbf{V}_{r}|\mathbf{V}_{\perp}] forms a n\times n orthogonal matrix. Note that zero singular values may appear in \bm{\Sigma}_{\perp} , but never appear in \bm{\Sigma}_{r} . The solution set S\triangleq\left\{\mathbf{M}\in\mathbb{R}^{m\times r},\mathbf{N}\in\mathbb{R}^{r\times n}:\mathbf{M}\mathbf{N}=\bar{\mathbf{Y}}_{r}\right\} is easily identified with (\bar{\mathbf{A}},\bar{\mathbf{B}}) : S=\left\{(\bar{\mathbf{A}}\mathbf{Q},\mathbf{Q}^{-1}\bar{\mathbf{B}}):\mathbf{Q}\in\mathsf{GL}(r)\right\}. Indeed, for (\mathbf{M},\mathbf{N})\in S , if the left inverse of \bar{\mathbf{A}} is \mathbf{L}_{\bar{\mathbf{A}}} , then \mathbf{Q}=\mathbf{L}_{\bar{\mathbf{A}}}\mathbf{M} gives \bar{\mathbf{A}}\mathbf{Q}=\bar{\mathbf{A}}\mathbf{L}_{\bar{\mathbf{A}}}\mathbf{M}=\mathbf{M} and \mathbf{Q}\mathbf{N}=\mathbf{L}_{\bar{\mathbf{A}}}\bar{\mathbf{Y}}_{r}=\bar{\mathbf{B}} ; definitely \mathrm{rank}(\mathbf{Q})=r from \bar{\mathbf{A}}\mathbf{Q}=\mathbf{M} . The mapping \Phi:\mathsf{GL}(r)\to\mathbb{R}^{m\times r}\times\mathbb{R}^{r\times n},\quad\Phi(\mathbf{Q})=(\bar{\mathbf{A}}\mathbf{Q},\mathbf{Q}^{-1}\bar{\mathbf{B}}) is real-analytic and satisfies \Phi^{-1}(\mathbf{M},\mathbf{N})=\mathbf{L}_{\bar{\mathbf{A}}}\mathbf{M} for a left inverse \mathbf{L}_{\bar{\mathbf{A}}} of \bar{\mathbf{A}} . Thus \Phi is a real-analytic diffeomorphism from an open subset (hence an analytic submanifold) \mathsf{GL}(r) to \Phi(\mathsf{GL}(r))=S , and S is an analytic submanifold. As \mathsf{GL}(r) has two connected components—matrices with positive and negative determinants—so does S . This issue is resolved by shrinking the domain of S to an open neighborhood of (\bar{\mathbf{A}},\bar{\mathbf{B}}) ; then S is connected. It remains to show that T_{(\bar{\mathbf{A}},\bar{\mathbf{B}})}S=\mathrm{ker}\nabla^{2}\mathcal{E}(\bar{\mathbf{A}},\bar{\mathbf{B}}) ( 13 , Definition 1.5) ; Other conditions for being a Morse-Bott function are automatically satisfied ( 13 , Remark 1.6) ; \nabla^{2}\mathcal{E}:(\mathbb{R}^{m\times r}\times\mathbb{R}^{r\times n})\to(\mathbb{R}^{m\times r}\times\mathbb{R}^{r\times n}) is a linear map, thus a Fredholm operator index 0, hence the range condition is automatically satisfied. Let \mathbf{R}(\mathbf{A},\mathbf{B})\triangleq\mathbf{A}\mathbf{B}-\bar{\mathbf{Y}}_{r} be the residual matrix. The differential of \mathbf{R} at (\bar{\mathbf{A}},\bar{\mathbf{B}}) is D\mathbf{R}(\bar{\mathbf{A}},\bar{\mathbf{B}})[\Delta\mathbf{A},\Delta\mathbf{B}]=\bar{\mathbf{A}}\Delta\mathbf{B}+\Delta\mathbf{A}\bar{\mathbf{B}}. It is straightforward to see that \mathrm{ker}D\mathbf{R}(\bar{\mathbf{A}},\bar{\mathbf{B}})=\left\{(\bar{\mathbf{A}}\mathbf{K},-\mathbf{K}\bar{\mathbf{B}}):\mathbf{K}\in\mathbb{R}^{r\times r}\right\}. Indeed, if \bar{\mathbf{A}}\Delta\mathbf{B}+\Delta\mathbf{A}\bar{\mathbf{B}}=\bm{0} , the right inverse of \bar{\mathbf{B}} , \mathbf{R}_{\bar{\mathbf{B}}} , gives \bar{\mathbf{A}}\Delta\mathbf{B}\mathbf{R}_{\bar{\mathbf{B}}}+\Delta\mathbf{A}=\bm{0} , which implies \Delta\mathbf{A}=-\bar{\mathbf{A}}\Delta\mathbf{B}\mathbf{R}_{\bar{\mathbf{B}}} and \bar{\mathbf{A}}(\Delta\mathbf{B}-\Delta\mathbf{B}\mathbf{R}_{\bar{\mathbf{B}}}\bar{\mathbf{B}})=\bm{0}\Rightarrow\Delta\mathbf{B}=\Delta\mathbf{B}\mathbf{R}_{\bar{\mathbf{B}}}\bar{\mathbf{B}} . T_{(\bar{\mathbf{A}},\bar{\mathbf{B}})}S\subset\mathrm{ker}D\mathbf{R}(\bar{\mathbf{A}},\bar{\mathbf{B}}) is trivial; we show the opposite inclusion. For (\Delta\mathbf{A},\Delta\mathbf{B})\in\mathrm{ker}D\mathbf{R}(\bar{\mathbf{A}},\bar{\mathbf{B}}) , we have (\Delta\mathbf{A},\Delta\mathbf{B})=(\bar{\mathbf{A}}\mathbf{K},-\mathbf{K}\bar{\mathbf{B}}) for some \mathbf{K} . Define a smooth curve \gamma(t)\triangleq(\bar{\mathbf{A}}(\mathbf{I}_{r}+t\mathbf{K}),(\mathbf{I}_{r}+t\mathbf{K})^{-1}\bar{\mathbf{B}}). For all t sufficiently close to 0, \mathbf{I}_{r}+t\mathbf{K} is invertible and \gamma(t)\in S . Moreover, \gamma^{\prime}(0)=(\bar{\mathbf{A}}\mathbf{K},-\mathbf{K}\bar{\mathbf{B}}). Hence, every (\Delta\mathbf{A},\Delta\mathbf{B})\in\mathrm{ker}D\mathbf{R}(\bar{\mathbf{A}},\bar{\mathbf{B}}) is realized as the velocity of a curve in S , or (\Delta\mathbf{A},\Delta\mathbf{B})\in T_{(\bar{\mathbf{A}},\bar{\mathbf{B}})}S . We obtain T_{(\bar{\mathbf{A}},\bar{\mathbf{B}})}S=\mathrm{ker}D\mathbf{R}(\bar{\mathbf{A}},\bar{\mathbf{B}}). The Hessian of \mathcal{E} at (\bar{\mathbf{A}},\bar{\mathbf{B}}) is computed: \displaystyle\nabla^{2}\mathcal{E}(\bar{\mathbf{A}},\bar{\mathbf{B}})[(\Delta\mathbf{A}_{0},\Delta\mathbf{B}_{0}),(\Delta\mathbf{A}_{1},\Delta\mathbf{B}_{1})] \displaystyle=\left\langle\Delta\mathbf{A}_{0}\bar{\mathbf{B}}+\bar{\mathbf{A}}\Delta\mathbf{B}_{0},\Delta\mathbf{A}_{1}\bar{\mathbf{B}}+\bar{\mathbf{A}}\Delta\mathbf{B}_{1}\right\rangle+\left\langle\bar{\mathbf{A}}\bar{\mathbf{B}}-\mathbf{Y},\Delta\mathbf{A}_{0}\Delta\mathbf{B}_{1}+\Delta\mathbf{A}_{1}\Delta\mathbf{B}_{0}\right\rangle. Without loss of generality, take \bar{\mathbf{A}}=\mathbf{U}_{r}\bm{\Sigma}_{r}^{1/2} and \bar{\mathbf{B}}=\bm{\Sigma}_{r}^{1/2}\mathbf{V}_{r}^{\top} . Note that any other (\hat{\mathbf{A}},\hat{\mathbf{B}})\in S must satisfy (\hat{\mathbf{A}},\hat{\mathbf{B}})=(\bar{\mathbf{A}}\mathbf{Q},\mathbf{Q}^{-1}\bar{\mathbf{B}})\triangleq\phi_{\mathbf{Q}}(\bar{\mathbf{A}},\bar{\mathbf{B}}) for \mathbf{Q}\in\mathsf{GL}(r) . From \bar{\mathbf{A}}\Delta\mathbf{B}+\Delta\mathbf{A}\bar{\mathbf{B}}=\bm{0} , we can check that (\Delta\mathbf{A},\Delta\mathbf{B})\in T_{(\bar{\mathbf{A}},\bar{\mathbf{B}})}S\quad\Leftrightarrow\quad\phi_{\mathbf{Q}}(\Delta\mathbf{A},\Delta\mathbf{B})\in T_{\phi_{\mathbf{Q}}(\bar{\mathbf{A}},\bar{\mathbf{B}})}S. This implies \nabla^{2}\mathcal{E}(\bar{\mathbf{A}},\bar{\mathbf{B}})[(\Delta\mathbf{A}_{0},\Delta\mathbf{B}_{0}),(\Delta\mathbf{A}_{1},\Delta\mathbf{B}_{1})]=\nabla^{2}\mathcal{E}(\phi_{\mathbf{Q}}(\bar{\mathbf{A}},\bar{\mathbf{B}}))[\phi_{\mathbf{Q}}(\Delta\mathbf{A}_{0},\Delta\mathbf{B}_{0}),\phi_{\mathbf{Q}}(\Delta\mathbf{A}_{1},\Delta\mathbf{B}_{1})], and thus (\Delta\mathbf{A},\Delta\mathbf{B})\in\mathrm{ker}\nabla^{2}\mathcal{E}(\bar{\mathbf{A}},\bar{\mathbf{B}})\quad\Leftrightarrow\quad\phi_{\mathbf{Q}}(\Delta\mathbf{A},\Delta\mathbf{B})\in\mathrm{ker}\nabla^{2}\mathcal{E}(\hat{\mathbf{A}},\hat{\mathbf{B}}). Hence, it suffices to show \mathrm{ker}\nabla^{2}\mathcal{E}(\bar{\mathbf{A}},\bar{\mathbf{B}})=T_{(\bar{\mathbf{A}},\bar{\mathbf{B}})}S at a single point (\bar{\mathbf{A}},\bar{\mathbf{B}})\in S to show that \mathrm{ker}\nabla^{2}\mathcal{E}(\mathbf{M},\mathbf{N})=T_{(\mathbf{M},\mathbf{N})}S for any (\mathbf{M},\mathbf{N})\in S . Moreover, as the critical set S is an analytic manifold of the analytic \mathcal{L} , we have T_{(\mathbf{M},\mathbf{N})}S\subset\mathrm{ker}\nabla^{2}\mathcal{E}(\bar{\mathbf{A}},\bar{\mathbf{B}}) ( 13 , Remark 2.9) , hence what remains is to show \mathrm{ker}\nabla^{2}\mathcal{E}(\bar{\mathbf{A}},\bar{\mathbf{B}})\subset T_{(\mathbf{M},\mathbf{N})}S . Now return to the specific point \bar{\mathbf{A}}=\mathbf{U}_{r}\bm{\Sigma}_{r}^{1/2} and \bar{\mathbf{B}}=\bm{\Sigma}_{r}^{1/2}\mathbf{V}_{r}^{\top} . If (\Delta\mathbf{A}_{0},\Delta\mathbf{B}_{0})\in\mathrm{ker}\nabla^{2}\mathcal{E}(\bar{\mathbf{A}},\bar{\mathbf{B}}), then for all (\Delta\mathbf{A}_{1},\Delta\mathbf{B}_{1}) , \nabla^{2}\mathcal{E}(\bar{\mathbf{A}},\bar{\mathbf{B}})[(\Delta\mathbf{A}_{0},\Delta\mathbf{B}_{0}),(\Delta\mathbf{A}_{1},\Delta\mathbf{B}_{1})]=\bm{0} . Specifically, \displaystyle\Delta\mathbf{A}_{1}=\bm{0} \displaystyle\Rightarrow\bar{\mathbf{A}}^{\top}(\Delta\mathbf{A}_{0}\bar{\mathbf{B}}+\bar{\mathbf{A}}\Delta\mathbf{B}_{0})+\Delta\mathbf{A}_{0}^{\top}(\bar{\mathbf{A}}\bar{\mathbf{B}}-\mathbf{Y})=\bm{0}, (41) \displaystyle\Delta\mathbf{B}_{1}=\bm{0} \displaystyle\Rightarrow(\Delta\mathbf{A}_{0}\bar{\mathbf{B}}+\bar{\mathbf{A}}\Delta\mathbf{B}_{0})\bar{\mathbf{B}}^{\top}+(\bar{\mathbf{A}}\bar{\mathbf{B}}-\mathbf{Y})\Delta\mathbf{B}_{0}^{\top}=\bm{0}. (42) There exists \mathbf{C}_{r},\mathbf{D}_{r}\in\mathbb{R}^{r\times r} , \mathbf{C}_{\perp}\in\mathbb{R}^{(m-r)\times r} , and \mathbf{D}_{\perp}\in\mathbb{R}^{r\times(n-r)} such that \Delta\mathbf{A}_{0}=\mathbf{U}_{r}\mathbf{C}_{r}+\mathbf{U}_{\perp}\mathbf{C}_{\perp},\quad\Delta\mathbf{B}_{0}=\mathbf{D}_{r}\mathbf{V}_{r}^{\top}+\mathbf{D}_{\perp}\mathbf{V}_{\perp}^{\top}. Plugging into Equations 41 and 42 , we obtain \displaystyle\bm{\Sigma}_{r}^{1/2}(\mathbf{C}_{r}\bm{\Sigma}_{r}^{1/2}+\bm{\Sigma}_{r}^{1/2}\mathbf{D}_{r})\mathbf{V}_{r}^{\top}+\bm{\Sigma}_{r}\mathbf{D}_{\perp}\mathbf{V}_{\perp}^{\top}-\mathbf{C}_{\perp}^{\top}\bm{\Sigma}_{\perp}\mathbf{V}_{\perp}^{\top}=\bm{0}, \displaystyle\mathbf{U}_{r}(\mathbf{C}_{r}\bm{\Sigma}_{r}^{1/2}+\bm{\Sigma}_{r}^{1/2}\mathbf{D}_{r})\bm{\Sigma}_{r}^{1/2}+\mathbf{U}_{\perp}\mathbf{C}_{\perp}\bm{\Sigma}_{r}-\mathbf{U}_{\perp}\bm{\Sigma}_{\perp}\mathbf{D}_{\perp}^{\top}=\bm{0}, which reduces to \displaystyle\mathbf{C}_{r}\bm{\Sigma}_{r}^{1/2}+\bm{\Sigma}_{r}^{1/2}\mathbf{D}_{r} \displaystyle=\bm{0}, (43) \displaystyle\bm{\Sigma}_{r}\mathbf{D}_{\perp}-\mathbf{C}_{\perp}^{\top}\bm{\Sigma}_{\perp} \displaystyle=\bm{0}, (44) \displaystyle\mathbf{C}_{\perp}\bm{\Sigma}_{r}-\bm{\Sigma}_{\perp}\mathbf{D}_{\perp}^{\top} \displaystyle=\bm{0}. (45) Since no singular values of \bm{\Sigma}_{r} equals with that of \bm{\Sigma}_{\perp} , Equations 44 and 45 implies that \mathbf{C}_{\perp}\bm{\Sigma}_{r}^{2}=\bm{\Sigma}_{\perp}^{2}\mathbf{C}_{\perp} thus \mathbf{C}_{\perp}=\bm{0} ; \mathbf{D}_{\perp}=\bm{0} also holds. Moreover, take \mathbf{K}=\bm{\Sigma}_{r}^{-1/2}\mathbf{C}_{r} ; we see that \displaystyle\bar{\mathbf{A}}\mathbf{K} \displaystyle=\mathbf{U}_{r}\mathbf{C}_{r}=\Delta\mathbf{A}_{0}, \displaystyle\mathbf{K}\bar{\mathbf{B}} \displaystyle=\bm{\Sigma}_{r}^{-1/2}\mathbf{C}_{r}\bm{\Sigma}_{r}^{1/2}\mathbf{V}_{r}^{\top}=\bm{\Sigma}_{r}^{-1/2}(-\bm{\Sigma}_{r}^{1/2}\mathbf{D}_{r})\mathbf{V}_{r}^{\top}=-\Delta\mathbf{B}_{0}. Therefore, \mathrm{ker}\nabla^{2}\mathcal{E}(\bar{\mathbf{A}},\bar{\mathbf{B}}) is a subset of T_{(\bar{\mathbf{A}},\bar{\mathbf{B}})}S . This argument analogously holds at any zero of \mathbf{R} , or any (\mathbf{M},\mathbf{N})\in S , so \mathcal{E} is Morse-Bott. Finite-time convergence of \mathcal{T} . We further show that if SpecGF with \mathcal{T} is well-defined—the gradients of \mathcal{L} with respect to both \mathbf{A} and \mathbf{B} are never rank-deficient—and converges, then the convergence should happen at a finite time. Proposition D.18 . Assume that SpecGF with \mathcal{T} is analytic for all time t\geq 0 . If SpecGF with \mathcal{T} converges, then it converges finitely to a critical point. Proof. Let \Delta(t)\triangleq\mathcal{L}(\mathbf{A}(t),\mathbf{B}(t))-\mathcal{L}(\mathbf{A}(\infty),\mathbf{B}(\infty))\triangleq\mathcal{L}(\mathbf{A}(t),\mathbf{B}(t))-\mathcal{L}_{\infty}\geq 0. For each (open) neighborhood U of (\mathbf{A}(\infty),\mathbf{B}(\infty)) , there exists a time t_{U}\geq 0 such that for all t\geq t_{U} , (\mathbf{A}(t),\mathbf{B}(t))\in U . Apply Lemma D.6 : there are a neighborhood W of (\mathbf{A}(\infty),\mathbf{B}(\infty)) , constants C>0 and b\in[\frac{1}{2},1) such that for all x\in W , |\mathcal{L}(\mathbf{A}(t),\mathbf{B}(t))-\mathcal{L}_{\infty}|^{b}\leq C\|\nabla\mathcal{L}(\mathbf{A}(t),\mathbf{B}(t))\|. Observe that \displaystyle\Delta^{\prime}(t) \displaystyle=\frac{\mathrm{d}}{\mathrm{d}t}\mathcal{L}(\mathbf{A}(t),\mathbf{B}(t))=-\left(\|\nabla_{\mathbf{A}}\mathcal{L}\|_{*}+\|\nabla_{\mathbf{B}}\mathcal{L}\|_{*}\right)\leq-\left(\|\nabla_{\mathbf{A}}\mathcal{L}\|_{\mathrm{F}}+\|\nabla_{\mathbf{B}}\mathcal{L}\|_{\mathrm{F}}\right) \displaystyle\leq-\|\nabla\mathcal{L}(\mathbf{A}(t),\mathbf{B}(t))\|_{\mathrm{F}} \displaystyle\leq-\frac{1}{C}\left(\mathcal{L}(\mathbf{A}(t),\mathbf{B}(t))-\mathcal{L}_{\infty}\right)^{b}=-\frac{1}{C}\Delta(t)^{b}. At any time t\geq t_{W} , if \Delta(t)=0 , done. Otherwise, \displaystyle\frac{\mathrm{d}}{\mathrm{d}t}\Delta(t)^{1-b}=(1-b)\Delta(t)^{-b}\Delta^{\prime}(t)\leq-\frac{1-b}{C} \displaystyle\Rightarrow \displaystyle\Delta(t)^{1-b}-\Delta(t_{W})^{1-b}\leq-\frac{1-b}{C}(t-t_{W}). Taking T=t_{W}+\frac{C}{1-b}\Delta(t_{W})^{1-b} gives \Delta(T)\leq 0 , hence \Delta(T)=0 . ∎ D.6 SpecGF with \ell_{2} Regularization With such regularization, the global convergence of SpecGF immediately follows. The setup of regularized SpecGF becomes • The target matrix \mathbf{Y}\in\mathbb{R}^{m\times n} yields a compact SVD \mathbf{Y}=\mathbf{U}\bm{\Sigma}\mathbf{V}^{\top} where \bm{\Sigma}=\mathrm{Diag}(\sigma_{1},\sigma_{2},\cdots,\sigma_{r^{\star}}) with \sigma_{1}\geq\cdots\geq\sigma_{r^{\star}}>0 . • The loss function \mathcal{L}_{\lambda}:\mathbb{R}^{m\times r}\times\mathbb{R}^{r\times n}\to\mathbb{R} for a fixed \lambda>0 is defined as \mathcal{L}_{\lambda}(\mathbf{A},\mathbf{B})\triangleq\frac{1}{2}\left\lVert\mathbf{A}\mathbf{B}-\mathbf{Y}\right\rVert_{\mathrm{F}}^{2}+\frac{\lambda}{2}\left\lVert\mathbf{A}\right\rVert_{\mathrm{F}}^{2}+\frac{\lambda}{2}\left\lVert\mathbf{B}\right\rVert_{\mathrm{F}}^{2}. (46) We may abuse the notation and simply state \mathcal{L}_{\lambda}(t) . \mathcal{L}_{\lambda} is also analytic in \mathbf{A} and \mathbf{B} . • The time derivative of \mathbf{A} and \mathbf{B} is governed by either \mathcal{T} or \mathcal{T}_{\beta} applied to the gradient: \dot{\mathbf{A}}(t)=-\mathcal{T}\left(\nabla_{\mathbf{A}}\mathcal{L}_{\lambda}(t)\right),\quad\dot{\mathbf{B}}(t)=-\mathcal{T}\left(\nabla_{\mathbf{B}}\mathcal{L}_{\lambda}(t)\right). The following statements are true, because the corresponding statements do not utilize the structure of the loss; only the analyticity of \mathcal{T}_{\beta} and monotonicity of the loss matter, if any of those two is required. 1. Lemmas D.4 and D.5 readily follows for the regularized SpecGF with \mathcal{T}_{\beta} . Moreover, \left\lVert\mathbf{A}(t)\right\rVert_{\mathrm{F}}^{2}+\left\lVert\mathbf{B}(t)\right\rVert_{\mathrm{F}}^{2}\leq\frac{2}{\lambda}\mathcal{L}_{\lambda}(t)\leq\frac{2}{\lambda}\mathcal{L}_{\lambda}(0)=\frac{1}{\lambda}\left\lVert\mathbf{Y}\right\rVert_{\mathrm{F}}^{2}+\left\lVert\mathbf{A}(0)\right\rVert_{\mathrm{F}}^{2}+\left\lVert\mathbf{B}(0)\right\rVert_{\mathrm{F}}^{2}. Thus, \mathbf{A}(t) and \mathbf{B}(t) are both bounded for all t\geq 0 . 2. Equation 30 in Proposition D.7 also holds for \mathcal{L}_{\lambda} (with the same constant), implying that regularized SpecGF with \mathcal{T}_{\beta} should either converge or diverge to infinity. However, the latter cannot happen; hence, regularized SpecGF with \mathcal{T}_{\beta} always converges. 3. Proposition D.12 is applicable for the regularized SpecGF with \mathcal{T}_{\beta} . That is, regularized SpecGF with \mathcal{T}_{\beta} avoids strict saddle points almost surely as SpecGF with \mathcal{T}_{\beta} does. 4. Theorem D.15 readily follows for the regularized SpecGF with \mathcal{T}_{\beta} . That is, every global minimum of regularized SpecGF with \mathcal{T}_{\beta} is Lyapunov stable. It remains to characterize the critical points of regularized SpecGF with \mathcal{T}_{\beta} , especially when LoRA rank r is smaller than the target rank r^{\star} . Although it is specified in 35 , we reproduce their result. Proposition D.19 ( ( 35 ) ) . Set r\leq r^{\star} . Assume that all singular values of \mathbf{Y} are distinct and \lambda is strictly smaller than the smallest singular value of \mathbf{Y} . (i) Let \Omega\subset[r^{\star}] with |\Omega|\leq r . Every stationary point of Equation 46 is of the form \mathbf{A}=\mathbf{U}_{\Omega}\left(\bm{\Sigma}_{\Omega}-\frac{\lambda}{2}\mathbf{I}\right)^{\frac{1}{2}},\quad\mathbf{B}=\left(\bm{\Sigma}_{\Omega}-\frac{\lambda}{2}\mathbf{I}\right)^{\frac{1}{2}}\mathbf{V}_{\Omega}^{\top}. up to orthogonal transformation by \mathbf{R}\in\mathsf{O}(r) . We denote the submatrix of \mathbf{U} with columns indexed by \Omega as \mathbf{U}_{\Omega} , and similarly for \bm{\Sigma} and \mathbf{V} . (ii) Every critical point except global minima has a descent direction. Item D.19.(ii) implies that every local minimum is a balanced global minimum, and every saddle point is a strict saddle point. Therefore, the a.s. global convergence of regularized SpecGF with \mathcal{T}_{\beta} towards global minima is guaranteed. Proof of Proposition D.19 . We omit the proof of Item D.19.(i) ; readers may find the proof in 35 . Though, we note that at every critical point, \mathbf{A}^{\top}\mathbf{A}=\mathbf{B}\mathbf{B}^{\top} should hold from the first-order optimality condition. Every critical point, indexed by \Omega , should be of the form \mathbf{A}\mathbf{B}=\sum_{i\in\Omega}d_{i}\mathbf{u}_{i}\mathbf{v}_{i}^{\top}, where d_{i}=\sigma_{i}-\lambda>0 , \mathbf{u}_{i} and \mathbf{v}_{i} are the i -th column of \mathbf{U} and \mathbf{V} , respectively. We use the representative element from each orbit described above, so each column of \mathbf{A} is \mathbf{u}_{i}\sqrt{d_{i}} and each row of \mathbf{B} is \sqrt{d_{i}}\mathbf{v}_{i}^{\top} . If a critical point is not the global minimum, then \sigma_{j}>\sigma_{i} for some i\in\Omega and j\not\in\Omega . With small \epsilon>0 , form \tilde{\mathbf{A}} by replacing the column \mathbf{u}_{i}\sqrt{d_{i}} by (\mathbf{u}_{i}+\epsilon\mathbf{u}_{j})\sqrt{d_{i}} ; form \tilde{\mathbf{B}} by replacing the column \sqrt{d_{i}}\mathbf{v}_{i}^{\top} by \sqrt{d_{i}}(\mathbf{v}_{i}+\epsilon\mathbf{v}_{j})^{\top} . One can directly compute \lambda\left[\left(\|\tilde{\mathbf{A}}\|_{\mathrm{F}}^{2}+\|\tilde{\mathbf{B}}\|_{\mathrm{F}}^{2}\right)-\left(\left\lVert\mathbf{A}\right\rVert_{\mathrm{F}}^{2}+\left\lVert\mathbf{B}\right\rVert_{\mathrm{F}}^{2}\right)\right]=2\lambda d_{i}\epsilon^{2} and \left\lVert\tilde{\mathbf{A}}\tilde{\mathbf{B}}-\mathbf{Y}\right\rVert_{\mathrm{F}}^{2}-\left\lVert\mathbf{A}\mathbf{B}-\mathbf{Y}\right\rVert_{\mathrm{F}}^{2}=2\epsilon^{2}d_{i}(d_{i}-\sigma_{j})+\epsilon^{4}d_{i}^{2}. Hence, the net change in the objective is 2\lambda d_{i}\epsilon^{2}+2\epsilon^{2}d_{i}(d_{i}-\sigma_{j})+\epsilon^{4}d_{i}^{2}=2\epsilon^{2}d_{i}(\sigma_{i}-\sigma_{j})+\epsilon^{4}d_{i}^{2}, which can be made (strictly) negative for small \epsilon . ∎ Appendix E Experiments E.1 Matrix Factorization Experiments Under Various Settings In this section, we present additional experiments that vary the training setting of Figure 2 . In Figure 7 , we vary the momentum parameter \mu in Equation 1 , where \mu=0 corresponds to SpecGD, while applying exact orthogonalization via \mathcal{T} at every update. We observe that the singular values of the product matrix exhibit consistent uniform growth across all choices of \mu , indicating that this behavior is robust to the strength of momentum under exact orthogonalization. In Figure 8 , we consider the same range of momentum parameters but replace the exact orthogonalization operator with Newton–Schulz iterations; this setting is identical to the practical Muon optimizer used in real training scenarios. Despite the use of approximate orthogonalization, we again observe uniform growth of singular values, demonstrating that the phenomenon persists under practical implementations of Muon. In Figure 9 , we vary the LoRA rank from 1 to 4 , which is strictly smaller than the target rank 5=\mathrm{rank}(\bm{\Sigma}) . In all cases, the SpecGD iteration converges, reaching plateaus at the top- r singular values of \bm{\Sigma} , and the resulting solution corresponds to the best rank- r approximation of \mathbf{Y} . Finally, we increase the number of LoRA factors (depth), by reparameterizing the weight \mathbf{W} as \mathbf{W}_{L}\mathbf{W}_{L-1}\cdots\mathbf{W}_{1} . All intermediate factors \mathbf{W}_{l} lie in \mathbb{R}^{r\times r} , with the first factor \mathbf{W}_{L} in \mathbb{R}^{m\times r} , and the last factor \mathbf{W}_{1} in \mathbb{R}^{r\times n} . Only the first factor \mathbf{W}_{L} is initialized at zero, while the remaining factors are initialized as i.i.d. Gaussian matrices scaled by \gamma . In Figure 10 , we observe that the uniform growth of singular values persists as depth increases, and Figure 11 further reveals that the growth rate scales as \Theta(t^{L}) for depth L . Figure 7 : Comparison of singular value evolutions across \mu\in\{0,0.3,0.5,0.9\} from exact orthogonalization \mathcal{T} . Notice the consistent uniform growth. Figure 8 : Comparison of singular value evolutions across \mu\in\{0,0.3,0.5,0.9\} from Newton-Schulz iterations. Notice the consistent uniform growth. Figure 9 : Comparison of singular value evolutions across LoRA rank r\in\{1,2,3,4\} . Notice that SpecGF converges toward best rank- r approximation. Figure 10 : Comparison of singular value evolutions across the depth L\in\{2,3,4,5\} . Notice the consistent uniform growth. Figure 11 : Comparison of the L -th root of the singular value evolutions across the depth L\in\{2,3,4,5\} . Notice the consistent uniform linear growth. E.2 LLM Experiments E.2.1 Experimental Details We fine-tuned a pre-trained RoBERTa-base model (125M) ( 27 ) on the SST-2 dataset ( 37 ) using LoRA adapters applied to the query and key matrices. We set the LoRA rank to r=8 and the scaling factor to \alpha=16 . We trained the model for 10 epochs with a batch size of 16 using AdamW and Muon optimizers, with a weight decay of 0.01. The learning rate was set to 10^{-4} and scheduled using a cosine decay. For larger scale experiments, we train LLaMA-3.2-1B ( 15 ) on Alpaca dataset ( 34 ) for supervised fine-tuning. As in RoBERTa-base, we also use the same LoRA configuration, i.e., rank r=8 and the scaling factor \alpha=16 . We train 3 epochs with a batch size 128 using AdamW and Muon optimizers, with weight decay 0.01. We use cosine learning rate scheduling with 4\% warmup steps of total training iterations. E.2.2 Singular Value Dynamics We provide the full training dynamics of the singular values of query Q and value V matrices where LoRA is applied. Each line in each subplot stands for a single singular value. Figures 12 and 13 show the singular value evolutions for all LoRA \mathbf{A}\mathbf{B} layers in RoBERTa-Base trained under Muon and AdamW, respectively; Figures 14 and 15 show those in LLaMA-3.2-1B trained under Muon and AdamW, respectively. The singular values of all LoRA \mathbf{A}\mathbf{B} layers trained under Muon ( Figures 14 and 12 ) consistently and nearly uniformly evolve. On the other hand, those of all LoRA \mathbf{A}\mathbf{B} layers trained under AdamW ( Figures 15 and 13 ) vary from matrix to matrix. (a) Q matrix (b) V matrix Figure 12 : Singular values for all LoRA \mathbf{A}\mathbf{B} layers in RoBERTa-Base trained with Muon. (a) Q matrix (b) V matrix Figure 13 : Singular values for all LoRA \mathbf{A}\mathbf{B} layers in RoBERTa-Base trained with AdamW. (a) Q matrix (b) V matrix Figure 14 : Singular values for all LoRA \mathbf{A}\mathbf{B} layers in LLaMA-3.2-1B trained with Muon. (a) Q matrix (b) V matrix Figure 15 : Singular values for all LoRA \mathbf{A}\mathbf{B} layers in LLaMA-3.2-1B trained with AdamW. E.2.3 Effective Rank We provide the evolution of the effective rank of query Q and value V matrices where LoRA is applied. Blue and red lines are the effective rank of the matrix trained under Muon and Adam W, respectively. Figure 16 shows the effective rank evolutions for all LoRA \mathbf{A}\mathbf{B} layers in LLaMA-3.2-1B; Figure 17 shows those in RoBERTa-Base. The effective rank of matrices trained under Muon maintains a stable value, close to the LoRA rank r=8 . That of matrices trained under AdamW exhibits non-trivial evolution. (a) Q matrix (b) V matrix Figure 16 : Effective rank of the LoRA \mathbf{A}\mathbf{B} layers for the Q and V matrices in RoBERTa-Base. (a) Q matrix (b) V matrix Figure 17 : Effective rank of the LoRA \mathbf{A}\mathbf{B} matrices for the Q and V in LLaMA-3.2-1B.