FIRE: Frobenius-Isometry Reinitialization for Balancing the Stability–Plasticity Tradeoff Isaac Han Sangyeon Park Seungwon Oh Donghu Kim, Hojoon Lee* Kyung-Joong Kim* Abstract Deep neural networks trained on nonstationary data must balance stability (i.e., retaining prior knowledge) and plasticity (i.e., adapting to new tasks). Standard reinitialization methods, which reinitialize weights toward their original values, are widely used but difficult to tune: conservative reinitializations fail to restore plasticity, while aggressive ones erase useful knowledge. We propose FIRE, a principled reinitialization method that explicitly balances the stability–plasticity tradeoff. FIRE quantifies stability through Squared Frobenius Error (SFE), measuring proximity to past weights, and plasticity through Deviation from Isometry (DfI), reflecting weight isotropy. The reinitialization point is obtained by solving a constrained optimization problem, minimizing SFE subject to DfI being zero, which is efficiently approximated by Newton–Schulz iteration. FIRE is evaluated on continual visual learning (CIFAR-10 with ResNet-18), language modeling (OpenWebText with GPT-0.1B), and reinforcement learning (HumanoidBench with SAC and Atari games with DQN). Across all domains, FIRE consistently outperforms both naive training without intervention and standard reinitialization methods, demonstrating effective balancing of the stability–plasticity tradeoff. Explore codes and videos at project page: https://isaac7778.github.io/fire/ . 1 Introduction Figure 1: Illustration of FIRE. Solving a constrained optimization problem, FIRE places weights at the intersection of high-stability and high-plasticity manifolds. Deep neural networks are typically trained under a fixed, stationary data distribution ( 7 ; 53 ) . However, many real-world applications require models to adapt continually as new data and shifting distributions emerge. In computer vision, autonomous driving systems must recognize unseen traffic signs, road layouts, or weather conditions that were absent during training ( 63 ) . Large language models, trained once and deployed with a fixed knowledge cutoff date, quickly become outdated unless continually updated ( 25 ) . Likewise, robots deployed in dynamic physical environments must adjust their perception and control policies as the environment changes ( 65 ) . In all of these domains, a central challenge is reliable adaptation to nonstationary data while preserving prior knowledge. This challenge is often framed as a balance between two competing properties: stability , the retention of learned knowledge, and plasticity , the ability to incorporate new information ( 47 ) . Different research communities emphasize these properties to varying degrees. Conventional continual learning assumes limited access to past data, prioritizing stability to mitigate catastrophic forgetting ( 28 ; 54 ; 56 ) . In contrast, most foundation models and robotic agents are trained on expanding datasets where past data remain accessible ( 1 ; 62 ) , making plasticity loss a central challenge ( 43 ; 5 ) . In this regime, stability is less about preserving past knowledge and more about accelerating adaptation to new tasks by leveraging prior representations. Motivated by these real world scenarios, we study the stability-plasticity tradeoff under the assumption of access to past data during continual learning. Existing approaches to mitigating plasticity loss fall broadly into two categories: regularization-based and reinitialization-based. Regularization-based methods constrain parameters or features near their initialization ( 32 ; 40 ) , or enforce weight orthogonality ( 9 ) . While these methods can preserve a favorable geometry for future learning, overly strong constraints slow convergence, and overly weak ones fail to prevent plasticity degradation. Reinitialization-based methods instead reset weights to earlier checkpoints when new data arrive ( 3 ; 50 ; 33 ; 60 ) . Their advantage lies in avoiding interference with current optimization, often yielding faster adaptation with lower overhead. However, they also suffer from a tuning dilemma: aggressive resets erase useful knowledge, while conservative ones provide little plasticity gain. We aim to resolve this dilemma by treating reinitialization as a principled constrained optimization problem. Our approach relies on two complementary measures that capture the core dimensions of the stability–plasticity tradeoff. First, we define stability using the Squared Frobenius Error (SFE) between current and past weights, which measures the sum of squared differences across all weight entries. A smaller SFE indicates greater similarity, meaning the model remains closer to its previous representations. For plasticity, prior work has linked plasticity loss to sharp loss curvature ( 43 ) , dormant neurons ( 61 ) , and low rank features ( 29 ) , but these metrics depend on incoming data and are non differentiable, limiting their use for optimization. We instead propose the Deviation from Isometry (DfI) ( 52 ; 66 ) , which measures how close weight matrices are to orthonormal. We show that reducing DfI simultaneously decreases curvature, prevents neuron dormancy, increases feature rank, and remains differentiable, making it a practical measure of plasticity to optimize. A formal proof is provided in Section 3 . We propose FIRE (Frobenius–Isometry REinitialization), which minimizes the SFE subject to the DfI being 0 . As illustrated in Figure 1 , FIRE avoids the pitfalls of either overly conservative or aggressive reinitialization by projecting weights onto the isotropy manifold while remaining close to their previous subspace. While directly solving this constrained optimization is costly, it can be implemented efficiently with the Newton–Schulz iteration, adding less than 1\% to training time. We evaluate FIRE on continual learning benchmarks in vision, language, and reinforcement learning, assuming access to past data. For vision, we split CIFAR-10, CIFAR-100, and Tiny-ImageNet into chunks under random or class-incremental protocols using ResNet ( 20 ) and Vision Transformer ( 11 ) . For language, we use a warm-start setup where GPT-0.1B ( 24 ) is pretrained on WikiText-103 and then continually trained on a mixture of OpenWebText and WikiText-103. For reinforcement learning, we test continuous control with SAC ( 18 ) on HumanoidBench ( 58 ) and discrete control with DQN ( 48 ) on Atari ( 4 ) . For vision and language tasks, reinitialization is applied whenever new data arrive, while in reinforcement learning it is applied once at the midpoint of training. Across all domains, FIRE consistently outperforms naive training and standard reinitialization, showing its effectiveness as a unified solution to the stability plasticity tradeoff. 2 Related Work 2.1 Stability-Plasticity tradeoff The stability–plasticity tradeoff ( 47 ; 26 ) is a fundamental challenge in continual learning. Stability refers to the ability of a model to preserve previously acquired knowledge and avoid catastrophic forgetting when exposed to new data. Plasticity refers to the ability of a model to adapt flexibly and effectively to novel tasks. These two properties often conflict with each other since strong stability can make the model rigid and resistant to new learning while excessive plasticity can lead to the loss of past knowledge. Research in continual learning has therefore focused on methods that balance these competing requirements such as constraining parameter updates through regularization ( 28 ) , revisiting earlier data through replay ( 54 ; 55 ; 38 ; 8 ; 2 ) , or designing architectures that separate parameters across tasks ( 56 ; 46 ; 45 ; 69 ; 64 ) . Their aim is to develop models that can maintain previously learned skills while remaining adaptive to new experiences. 2.2 Loss of plasticity Deep learning has traditionally been studied under stationary datasets ( 17 ; 19 ) , yet real-world applications often involve non-stationary streams ( 59 ; 31 ) . Training in such environments leads to a loss of plasticity ( 43 ; 10 ; 32 ) , where models fail to adapt to new distributions. Prior work has identified potential indicators of this phenomenon, including dormant neurons ( 61 ; 67 ) , shifts in pre-activations ( 42 ) , feature rank collapse ( 29 ) , and diverging weight magnitudes ( 42 ) . Loss of plasticity hinders not only the ability to fit the training data, but also the ability to generalize to unseen data. Models trained incrementally often generalize worse than those trained from scratch ( 3 ; 5 ; 41 ) , due to factors such as diminished gradient norms ( 3 ) , weak feature changes ( 41 ) , and the compounding effects of small pretraining datasets or noisy labels ( 33 ) . To counteract plasticity loss, reinitialization-based strategies such as S&P ( 3 ) , DASH ( 60 ) reinitialize weights into an intermediate checkpoint, weight regularizers constrain parameters to initialization or specific subspaces ( 32 ; 13 ; 36 ) , and spectral or rank-based approaches explicitly maintain representation quality ( 29 ; 30 ; 21 ) . Another approach proposed reinitializing at the neuronal level, based on the utility of each neuron ( 61 ; 10 ; 14 ) . Recent work further leverages the fact that linear networks do not suffer from plasticity loss ( 10 ; 37 ; 51 ) . 3 Method In this section we explain how we frame the stability-plasticity tradeoff as a constrained optimization problem. To do this we first need two metrics: one for stability and one for plasticity loss. With these two pieces in place the optimization problem naturally emerges, and from this formulation we introduce our method FIRE, which provides an efficient approximation to the solution. 3.1 Measure for stability To measure stability, we propose a simple yet effective metric, the Squared Frobenius Error (SFE) . SFE provides a natural way to quantify the preservation of learned information by comparing an original weight matrix W with its modified counterpart \widetilde{W} . Formally, \text{SFE}(W,\widetilde{W})=\|W-\widetilde{W}\|_{F}^{2}, (1) which captures the element-wise squared deviation between the two weight configurations. However, it remains unclear whether SFE can be used as a metric that can meaningfully capture similarity of feature representations. To clarify this point, we establish a theoretical link between SFE and the normalized feature covariance, a metric widely used in prior work to measure representation similarity ( 41 ; 68 ) . In particular, we show that SFE provides an upper bound on the discrepancy between the normalized feature covariances of two distinct neural networks’ output features (Theorem 1 ). Theorem 1 (SFE bounds output feature covariance between two deep neural networks) . Let \Theta=\{W^{1},\dots,W^{L}\} and \widetilde{\Theta}=\{\widetilde{W}^{1},\dots,\widetilde{W}^{L}\} be the parameters of two depth- L feedforward networks with elementwise activations \sigma_{\ell} (Lipschitz constants L_{\sigma_{\ell}} ). For an input batch Z\in\mathbb{R}^{n\times d_{0}} , we denote the layer outputs recursively by H_{\Theta}^{\,0}(Z)=Z and H_{\Theta}^{\,\ell}(Z)=\sigma_{\ell}(H_{\Theta}^{\,\ell-1}(Z)W^{\ell}) . Let B_{\ell}=\max\{\|W^{\ell}\|_{2},\|\widetilde{W}^{\ell}\|_{2}\} be the maximum spectral norm of the weights in layer \ell and B_{\Pi}^{\ell}=\prod_{k=1}^{\ell}B_{k} the product across all layers. We further define m_{\ell}=\min\{\|H_{\Theta}^{\,\ell}(Z)\|_{F},\|H_{\widetilde{\Theta}}^{\,\ell}(Z)\|_{F}\}>0 . The normalized feature covariances of the two networks are given by C_{\Theta}^{\,\ell}(Z)=H_{\Theta}^{\,\ell}(Z)H_{\Theta}^{\,\ell}(Z)^{\top}/\|H_{\Theta}^{\,\ell}(Z)\|_{F}^{2},\quad C_{\widetilde{\Theta}}^{\,\ell}(Z)=H_{\widetilde{\Theta}}^{\,\ell}(Z)H_{\widetilde{\Theta}}^{\,\ell}(Z)^{\top}/\|H_{\widetilde{\Theta}}^{\,\ell}(Z)\|_{F}^{2}. Then the difference between the output feature covariances of the two networks is bounded as follows: \|C_{\Theta}^{\,\ell}(Z)-C_{\widetilde{\Theta}}^{\,\ell}(Z)\|_{F}^{2}\leq\frac{16\,\|Z\|_{F}^{2}}{m_{\ell}^{2}}\Big(\prod_{k=1}^{\ell}L_{\sigma_{k}}\Big)^{\!2}B_{\Pi}^{2}\Bigg(\sum_{j=1}^{\ell}\tfrac{1}{B_{j}^{2}}\Bigg)\,\mathrm{SFE}(\Theta,\widetilde{\Theta}). In particular, if each activation is 1 -Lipschitz (L_{\sigma_{k}}\leq 1) and each spectral norm is bounded (B_{j}\leq S) , then \|C_{\Theta}^{\,\ell}(Z)-C_{\widetilde{\Theta}}^{\,\ell}(Z)\|_{F}\;\leq\;\frac{4\,\|Z\|_{F}}{m_{\ell}}\,\sqrt{\ell}\,S^{\ell-1}\,\sqrt{\mathrm{SFE}(\Theta,\widetilde{\Theta})}. Theorem 1 shows that the discrepancy between normalized feature covariances is bounded by four factors: the input norm ( \|Z\|_{F}^{2} ), the Lipschitz constants of activation functions ( L_{\sigma_{\ell}} ), the spectral norms of the weight matrices ( B_{\Pi} and B_{j} ), and SFE. In practice, inputs are typically normalized and have a fixed upper bound, and commonly used activation functions have small Lipschitz constants (e.g., 1 for ReLU and Tanh). Thus, the contributions from \|Z\|_{F}^{2} and L_{\sigma_{\ell}} can be ignored. The remaining dominant factors are the spectral norms of the weight matrices ( B_{\Pi} and B_{j} ) and SFE. This has two implications. First, for any fixed architecture and weight scale, minimizing SFE monotonically tightens the upper bound, and thus is an effective way to preserve feature similarity between two networks. Second, the slope of the bound with respect to SFE is proportional to the spectral norms: larger spectral norms make the worst-case discrepancy more sensitive to changes in SFE. In such regimes, reducing SFE becomes even more important for stability, since even a small increase in SFE can, in principle, lead to a substantial change in the resulting feature representations. 3.2 Measure for plasticity loss In this work, we view plasticity loss as the situation where weights learned from previous data fail to serve as a favorable initialization point for new data. This perspective is particularly useful when designing reinitialization strategies that partially reinitialize weights before the arrival of new data. Prior analyses of plasticity loss suggest that one of its main causes is the increasing sharpness of the loss landscape curvature with respect to new data during training on the current task, which destabilizes optimization ( 43 ) . In addition, an increase in dormant neurons ( 61 ) and a collapse in effective rank ( 29 ) have also been known to indicate plasticity loss. However, as these measures are data-dependent and non-differentiable, they are inappropriate for the optimization objective. Instead, we propose Deviation from Isometry (DfI) ( 52 ; 66 ) , an optimizable metric that closely connected to previous plasticity measure. \text{DfI}(W)=\|W^{\top}W-I\|_{F}^{2}. (2) Our theoretical analysis reveals that minimizing DfI results in a smoother loss landscape curvature (Theorem 2 ), a smaller number of dormant neurons (Theorem 4 ), and a higher effective rank (Theorem 3 ). This supports the use of DfI as a suitable measure for plasticity measure. In addition, minimizing DfI also makes the weights close to isotropy, which is known as a key property of favorable neural network initializations, termed as dynamical isometry ( 66 ) . Theorem 2 (Hessian spectral norm bounded by layerwise DfIs) . We assume that • the inputs are whitened, so that the empirical covariance \Sigma_{Z}=\tfrac{1}{n}Z^{\top}Z is approximately the identity: \Sigma_{Z}:=\frac{1}{n}Z^{\top}Z\approx I • for every sample i and relevant parameter vector u , the Hessian norm satisfies |\nabla^{2}_{u}\ell_{i}(u)|_{2}\leq\beta , while the gradient norm is bounded by |\nabla_{u}\ell_{i}(u)|_{2}\leq\gamma . • finally, we focus on a fixed ReLU activation pattern at the point of interest, so that the network is piecewise linear in that region. In particular, each diagonal gating matrix arising from the ReLU has operator norm \leq 1 . Let \nu_{k}=1+\sqrt{\mathrm{DfI}(W_{k})} , then the Hessian spectral norm can be bounded as follows: \big\|\nabla^{2}_{\theta}\mathcal{L}(W_{1:L})\big\|_{2}\;\leq\;\beta\,\sum_{k=1}^{L}\prod_{j\neq k}\nu_{j}\;+\;2\gamma\,\sum_{1\leq k<\ell\leq L}\prod_{j\notin\{k,\ell\}}\nu_{j}. Theorem 3 (DfI controls effective rank) . Let Z\in\mathbb{R}^{n\times a} and W\in\mathbb{R}^{a\times b} be an input and weight matrix, respectively, \Phi=ZW be a feature matrix. Let \Sigma_{Z}=\tfrac{1}{n}Z^{\top}Z be the empirical covariance of the inputs and let W=QS be the right polar decomposition of W , where Q\in\mathbb{R}^{a\times b} has orthonormal columns. Then \eta_{1}\geq\cdots\geq\eta_{d}>0 denote the positive eigenvalues of Q^{\top}\Sigma_{Z}Q , with d=\operatorname{rank}(Q^{\top}\Sigma_{Z}Q) , and \mathrm{srank}_{\delta}(\Phi) be defined from the nonzero singular values \{\sigma_{i}(\Phi)\}_{i=1}^{d} by \mathrm{srank}_{\delta}(\Phi)=\min\Bigl\{k:\;({\sum_{i=1}^{k}\sigma_{i}(\Phi)}/{\sum_{i=1}^{d}\sigma_{i}(\Phi)})\geq 1-\delta\Bigr\}. Then \varepsilon=\sqrt{\mathrm{DfI}(W)}<1 gives a lower bound on the srank as left inequality below. If additionally \Sigma_{Z}=I , the bound simplifies to the right inequality below. \mathrm{srank}_{\delta}(\Phi)\;\;\geq\;\;\left\lceil\frac{(1-\delta)\,d}{\delta\,\sqrt{\tfrac{1+\varepsilon}{1-\varepsilon}}\;\sqrt{\tfrac{\eta_{1}}{\eta_{d}}}+(1-\delta)}\right\rceil,\qquad\mathrm{srank}_{\delta}(\Phi)\;\;\geq\;\;\left\lceil\frac{(1-\delta)\,d}{\delta\,\sqrt{\tfrac{1+\varepsilon}{1-\varepsilon}}+(1-\delta)}\right\rceil. (3) Theorem 4 (Minimizing DfI increases neuron activity score) . Let W\in\mathbb{R}^{a\times b} and let \sigma be positive-homogeneous ( \sigma(\alpha t)=\alpha\sigma(t) for \alpha\geq 0 ). Assume the input vector z\sim\mathcal{N}(0,I_{a}) (isotropic Gaussian). For neuron j with column w_{j} of W , define activity score of neuron j as s_{j}=\frac{\mathbb{E}_{z}[|\sigma(\langle z,w_{j}\rangle)|]}{\frac{1}{b}\sum_{k=1}^{b}\mathbb{E}_{z}[|\sigma(\langle z,w_{k}\rangle)|]}. If \varepsilon:=\sqrt{\mathrm{DfI}(W)}<1 , then for all j , \sqrt{\frac{1-\varepsilon}{\,1+\varepsilon\,}}\;\leq\;s_{j}\;\leq\;\sqrt{\frac{1+\varepsilon}{\,1-\varepsilon\,}}. 61 classified neurons with s_{j}<\tau as dormant neurons. Hence, reducing the number of dormant neurons requires increasing the activity scores s_{j} . Theorem 4 states that minimizing DfI increases the lower bound \sqrt{\frac{1-\epsilon}{1+\epsilon}}0 , \left\|\frac{F}{\|F\|}-\frac{\widetilde{F}}{\|\widetilde{F}\|}\right\|\;\leq\;\frac{2}{m}\,\|F-\widetilde{F}\|. If arbitrary two matrix U and V satisfies \|U\|=\|V\|=1 then \|UU^{\top}-VV^{\top}\|\;\leq\;2\,\|U-V\|. Combining these two facts with U:=F/\|F\| and V:=\widetilde{F}/\|\widetilde{F}\| yields \left\|\frac{FF^{\top}}{\|F\|^{2}}-\frac{\widetilde{F}\,\widetilde{F}^{\top}}{\|\widetilde{F}\|^{2}}\right\|\;\leq\;\frac{4}{m}\,\|F-\widetilde{F}\|. (6) Applying equation 6 with F=H_{\Theta}^{\,\ell}(Z) and \widetilde{F}=H_{\widetilde{\Theta}}^{\,\ell}(Z) , and m_{\ell}:=\min\{\|H_{\Theta}^{\,\ell}(Z)\|,\|H_{\widetilde{\Theta}}^{\,\ell}(Z)\|\} gives \|C_{\Theta}^{\,\ell}(Z)-C_{\widetilde{\Theta}}^{\,\ell}(Z)\|\;\leq\;\frac{4}{m_{\ell}}\,\|H_{\Theta}^{\,\ell}(Z)-H_{\widetilde{\Theta}}^{\,\ell}(Z)\|. (7) Introduce the hybrid outputs H_{(\leq j)}^{\,\ell}(Z) : the first j layers use \Theta and the remaining j+1,\dots,\ell layers use \widetilde{\Theta} . Note that H_{(\leq\ell)}^{\,\ell}(Z)=H_{\Theta}^{\,\ell}(Z) and H_{(\leq 0)}^{\,\ell}(Z)=H_{\widetilde{\Theta}}^{\,\ell}(Z) . Then H_{\Theta}^{\,\ell}(Z)-H_{\widetilde{\Theta}}^{\,\ell}(Z)=\sum_{j=1}^{\ell}\Big(H_{(\leq j)}^{\,\ell}(Z)-H_{(\leq j-1)}^{\,\ell}(Z)\Big), so by the triangle inequality, \|H_{\Theta}^{\,\ell}(Z)-H_{\widetilde{\Theta}}^{\,\ell}(Z)\|\;\leq\;\sum_{j=1}^{\ell}\|H_{(\leq j)}^{\,\ell}(Z)-H_{(\leq j-1)}^{\,\ell}(Z)\|. (8) Each summand differs in only the j -th layer weights . Let X_{j-1}:=H_{\Theta}^{\,j-1}(Z) denote the shared input fed to layer j in both hybrids. Consider the backend subnetwork \mathcal{T}^{(\widetilde{\Theta})}_{j\to\ell}(Y):=\sigma_{\ell}\big(\cdots\sigma_{j+1}(Y\,\widetilde{W}^{\,j+1})\cdots\,\widetilde{W}^{\,\ell}\big). For arbitrary Y_{1} and Y_{2} , its input-Lipschitz constant is \big\|\mathcal{T}^{(\widetilde{\Theta})}_{j\to\ell}(Y_{1})-\mathcal{T}^{(\widetilde{\Theta})}_{j\to\ell}(Y_{2})\big\|\;\leq\;\Big(\prod_{k=j+1}^{\ell}L_{\sigma_{k}}\,\|\widetilde{W}^{\,k}\|_{2}\Big)\,\|Y_{1}-Y_{2}\|. (9) Hence \displaystyle\|H_{(\leq j)}^{\,\ell}(Z)-H_{(\leq j-1)}^{\,\ell}(Z)\| \displaystyle\leq\Big(\prod_{k=j+1}^{\ell}L_{\sigma_{k}}\,\|\widetilde{W}^{\,k}\|_{2}\Big)\,L_{\sigma_{j}}\,\|X_{j-1}\|\,\|W^{j}-\widetilde{W}^{\,j}\|. (10) Meanwhile, \|X_{j-1}\|=\|H_{\Theta}^{\,j-1}(Z)\|\;\leq\;\|Z\|\,\prod_{k=1}^{j-1}L_{\sigma_{k}}\,\|W^{k}\|_{2}. (11) Combining equation 8 , equation 10 , and equation 11 yields \|H_{\Theta}^{\,\ell}(Z)-H_{\widetilde{\Theta}}^{\,\ell}(Z)\|\;\leq\;\|Z\|\Big(\prod_{k=1}^{\ell}L_{\sigma_{k}}\Big)\,\sum_{j=1}^{\ell}\Big(\prod_{k\neq j}B_{k}\Big)\,\|W^{j}-\widetilde{W}^{\,j}\|. By Cauchy–Schwarz, \sum_{j=1}^{\ell}\Big(\prod_{k\neq j}B_{k}\Big)\,\|W^{j}-\widetilde{W}^{\,j}\|\;\leq\;B_{\Pi}^{\ell}\Bigg(\sum_{j=1}^{\ell}\frac{1}{B_{j}^{2}}\Bigg)^{\!1/2}\sqrt{\mathrm{SFE}(\Theta,\widetilde{\Theta})}. Therefore, \|H_{\Theta}^{\,\ell}(Z)-H_{\widetilde{\Theta}}^{\,\ell}(Z)\|\;\leq\;\|Z\|\Big(\prod_{k=1}^{\ell}L_{\sigma_{k}}\Big)\,B_{\Pi}^{\ell}\Bigg(\sum_{j=1}^{\ell}\frac{1}{B_{j}^{2}}\Bigg)^{\!1/2}\sqrt{\mathrm{SFE}(\Theta,\widetilde{\Theta})}. (12) Substitute equation 12 into equation 7 : \|C_{\Theta}^{\,\ell}(Z)-C_{\widetilde{\Theta}}^{\,\ell}(Z)\|\;\leq\;\frac{4\,\|Z\|}{m_{\ell}}\,\Big(\prod_{k=1}^{\ell}L_{\sigma_{k}}\Big)\,B_{\Pi}^{\ell}\Bigg(\sum_{j=1}^{\ell}\frac{1}{B_{j}^{2}}\Bigg)^{\!1/2}\sqrt{\mathrm{SFE}(\Theta,\widetilde{\Theta})}. Squaring both sides gives \|C_{\Theta}^{\,\ell}-C_{\widetilde{\Theta}}^{\,\ell}\|^{2}\;\leq\;\frac{16\,\|Z\|^{2}}{m_{\ell}^{2}}\Big(\prod_{k=1}^{\ell}L_{\sigma_{k}}\Big)^{\!2}(B_{\Pi}^{\ell})^{2}\Bigg(\sum_{j=1}^{\ell}\frac{1}{B_{j}^{2}}\Bigg)\,\mathrm{SFE}(\Theta,\widetilde{\Theta}). If L_{\sigma_{k}}\leq 1 and B_{j}\leq S for all j , then (B_{\Pi}^{\ell})^{2}\sum_{j=1}^{\ell}B_{j}^{-2}\leq\ell\,S^{2\ell-2}. Therefore \|C_{\Theta}^{\,\ell}-C_{\widetilde{\Theta}}^{\,\ell}\|\;\leq\;\frac{4\,\|Z\|}{m_{\ell}}\,\sqrt{\ell}\,S^{\ell-1}\,\sqrt{\mathrm{SFE}(\Theta,\widetilde{\Theta})}. ∎ Network, Loss, and Notation. Let Z\in\mathbb{R}^{n\times d_{0}} be the input matrix and W_{k}\in\mathbb{R}^{d_{k-1}\times d_{k}} (k=1,\dots,L) be the weight matrices. Define A_{k}=H_{k-1}W_{k}\in\mathbb{R}^{n\times d_{k}},\qquad H_{k}=\rho(A_{k})\quad(\rho=\mathrm{ReLU}),\qquad U:=A_{L}\in\mathbb{R}^{n\times d_{L}}, with H_{0}:=Z . The empirical risk is \mathcal{L}(W_{1:L})=\frac{1}{n}\sum_{i=1}^{n}\ell_{i}(u_{i}),\qquad u_{i}\in\mathbb{R}^{d_{L}}. Let \theta=\mathrm{vec}(W_{1},\dots,W_{L})\in\mathbb{R}^{p} and H_{\theta}:=\nabla^{2}_{\theta}\mathcal{L}\in\mathbb{R}^{p\times p} . We use \|\cdot\|_{2} for spectral norm and \|\cdot\|_{F} for the Frobenius norm. We denote the maximum eigen values as \lambda_{\max} Deviation From Isometry (DfI). For a matrix W , \mathrm{DfI}(W):=\|W^{\top}W-I\|_{F}^{2} . For each layer, set \nu_{k}:=1+\sqrt{\mathrm{DfI}(W_{k})},\qquad\alpha_{k}:=\sqrt{\nu_{k}}. First, let us examine the lemmas required for the proof of Theorem 2 . Lemma 1 (DfI controls the spectral norm) . For each layer k , \|W_{k}\|_{2}^{2}\leq\nu_{k} and \|W_{k}\|_{2}\leq\alpha_{k} . Proof. \|W\|_{2}^{2}=\lambda_{\max}(W^{\top}W)\leq\lambda_{\max}(W^{\top}W-I)+1\leq\|W^{\top}W-I\|_{2}+1\leq\|W^{\top}W-I\|_{F}+1=1+\sqrt{\mathrm{DfI}(W)} . ∎ Lemma 2 (Covariance/spectral growth through layers) . Let \Sigma_{H_{k}}=\tfrac{1}{n}H_{k}^{\top}H_{k} . Then \lambda_{\max}(\Sigma_{H_{k}})\;\leq\;\lambda_{\max}(\Sigma_{H_{k-1}})\,\|W_{k}\|_{2}^{2}\;\leq\;\lambda_{\max}(\Sigma_{H_{k-1}})\,\nu_{k}. Consequently, with \Sigma_{H_{0}}=\Sigma_{Z} , \lambda_{\max}(\Sigma_{H_{k}})\;\leq\;\lambda_{\max}(\Sigma_{Z})\,\prod_{j=1}^{k}\nu_{j}. In particular, under \Sigma_{Z}\approx I we have \lambda_{\max}(\Sigma_{H_{k}})\leq\prod_{j=1}^{k}\nu_{j} . Proof. ReLU is 1 -Lipschitz (applied elementwise), hence \|H_{k}\|_{2}\leq\|A_{k}\|_{2}\leq\|H_{k-1}\|_{2}\|W_{k}\|_{2} . Therefore \frac{1}{n}\|H_{k}\|_{2}^{2}\leq\frac{1}{n}\|H_{k-1}\|_{2}^{2}\|W_{k}\|_{2}^{2} , i.e., \lambda_{\max}(\Sigma_{H_{k}})\leq\lambda_{\max}(\Sigma_{H_{k-1}})\|W_{k}\|_{2}^{2} . Apply Lemma 1 . ∎ Lemma 3 (Block-Jacobian bound) . Let J\in\mathbb{R}^{(nd_{L})\times p} be the Jacobian of \mathrm{vec}(U) w.r.t. \theta , and J=[J_{1}\,\,J_{2}\,\,\cdots\,\,J_{L}] the block-columns corresponding to \mathrm{vec}(W_{k}) . Then \frac{1}{n}\|J_{k}\|_{2}^{2}\;\leq\;\Big(\prod_{j=1}^{k-1}\nu_{j}\Big)\,\Big(\prod_{j=k+1}^{L}\nu_{j}\Big)=\prod_{j\neq k}\nu_{j}. Consequently, \frac{1}{n}\|J\|_{2}^{2}\;\leq\;\sum_{k=1}^{L}\frac{1}{n}\|J_{k}\|_{2}^{2}\;\leq\;\sum_{k=1}^{L}\prod_{j\neq k}\nu_{j}. Proof. Consider a perturbation \Delta W_{k} . Without loss of generality, we may assume \|\Delta W_{k}\|_{F}=1 , since the operator norm is defined by the supremum over unit perturbations. With fixed ReLU gates (op. norm \leq 1 ), the output perturbation over all n samples satisfies \Delta U\;=\;H_{k-1}\,\Delta W_{k}\,B_{k+1:L},\quad B_{k+1:L}:=\underbrace{D_{k}W_{k+1}D_{k+1}\cdots D_{L-1}}_{\text{diag gates, }\|\cdot\|_{2}\leq 1}\,W_{L}. Thus \|\Delta U\|_{F}\leq\|H_{k-1}\|_{2}\,\|\Delta W_{k}\|_{F}\,\|B_{k+1:L}\|_{2} , so the operator norm of the linear map \Delta W_{k}\mapsto\Delta U is at most \|H_{k-1}\|_{2}\|B_{k+1:L}\|_{2} . Hence \|J_{k}\|_{2}\leq\|H_{k-1}\|_{2}\|B_{k+1:L}\|_{2} and \frac{1}{n}\|J_{k}\|_{2}^{2}\;\leq\;\frac{1}{n}\|H_{k-1}\|_{2}^{2}\,\|B_{k+1:L}\|_{2}^{2}\;=\;\lambda_{\max}(\Sigma_{H_{k-1}})\,\|B_{k+1:L}\|_{2}^{2}. Using Lemma 2 , \lambda_{\max}(\Sigma_{H_{k-1}})\leq\prod_{j=1}^{k-1}\nu_{j} . Also \|B_{k+1:L}\|_{2}\leq\prod_{j=k+1}^{L}\|W_{j}\|_{2}\leq\prod_{j=k+1}^{L}\alpha_{j} , thus \|B_{k+1:L}\|_{2}^{2}\leq\prod_{j=k+1}^{L}\nu_{j} . Multiplying the two bounds yields the claim. Finally, since J is a horizontal concatenation of blocks, \|J\|_{2}^{2}\leq\sum_{k}\|J_{k}\|_{2}^{2} . ∎ Lemma 4 (Gauss–Newton part) . For each sample, \nabla^{2}_{\theta}\ell_{i}=J_{i}^{\top}(\nabla^{2}_{u}\ell_{i})J_{i}+R_{i} with some remainder R_{i} . By (A2), \|\nabla^{2}_{u}\ell_{i}\|_{2}\leq\beta , hence \left\|\frac{1}{n}\sum_{i=1}^{n}J_{i}^{\top}(\nabla^{2}_{u}\ell_{i})J_{i}\right\|_{2}\;\leq\;\frac{\beta}{n}\,\|J\|_{2}^{2}\;\leq\;\beta\sum_{k=1}^{L}\prod_{j\neq k}\nu_{j}, where the last inequality uses Lemma 3 . Lemma 5 (Remainder term) . Let R:=\frac{1}{n}\sum_{i=1}^{n}R_{i} . Under (A2) and (A3), \|R\|_{2}\;\leq\;2\gamma\sum_{1\leq k<\ell\leq L}\prod_{j\notin\{k,\ell\}}\nu_{j}. Proof. Fix the ReLU gates locally (piecewise linear region) and \sum_{k}\|\Delta W_{k}\|_{F}=1 , since the operator norm is defined by the supremum over unit perturbations. Then the network output U is multilinear in \{W_{k}\}_{k=1}^{L} . For k<\ell , the mixed second derivative block maps (\Delta W_{k},\Delta W_{\ell}) to H_{k-1}\,\Delta W_{k}\,C_{k+1:\ell-1}\,\Delta W_{\ell}\,B_{\ell+1:L}, where C_{k+1:\ell-1} is the product of intermediate gated weights, and B_{\ell+1:L} the tail product as in Lemma 3 . By submultiplicativity, \|\!H_{k-1}\Delta W_{k}C_{k+1:\ell-1}\Delta W_{\ell}B_{\ell+1:L}\!\|_{F}\leq\|H_{k-1}\|_{2}\,\|\Delta W_{k}\|_{F}\,\|C_{k+1:\ell-1}\|_{2}\,\|\Delta W_{\ell}\|_{F}\,\|B_{\ell+1:L}\|_{2}. Using Lemma 2 and Lemma 1 , \|H_{k-1}\|_{2}\leq\sqrt{n}\,\prod_{j=1}^{k-1}\alpha_{j},\quad\|C_{k+1:\ell-1}\|_{2}\leq\prod_{j=k+1}^{\ell-1}\alpha_{j},\quad\|B_{\ell+1:L}\|_{2}\leq\prod_{j=\ell+1}^{L}\alpha_{j}. Therefore, after dividing by n (from the prefactor 1/n in \mathcal{L} ) and summing the symmetric contribution (\ell,k) , the bilinear remainder contributes at most \frac{2}{n}\sum_{k<\ell}\|H_{k-1}\|_{2}\|C_{k+1:\ell-1}\|_{2}\|B_{\ell+1:L}\|_{2}\,\|\Delta W_{k}\|_{F}\|\Delta W_{\ell}\|_{F}\;\leq\;2\sum_{k<\ell}\Big(\prod_{j\notin\{k,\ell\}}\alpha_{j}^{2}\Big)\,\|\Delta W_{k}\|_{F}\|\Delta W_{\ell}\|_{F}. Finally, by 2ab\leq a^{2}+b^{2} and \sum_{k}\|\Delta W_{k}\|_{F}^{2}=1 (unit parameter direction), the operator norm of the second-derivative map is bounded by \sum_{k<\ell}\prod_{j\notin\{k,\ell\}}\nu_{j} . Multiplying by \|\nabla_{u}\ell_{i}\|_{2}\leq\gamma and averaging over i gives the claim. ∎ Proof of Theorem 2 Proof. Combine Lemma 4 and Lemma 5 and use \|A+B\|_{2}\leq\|A\|_{2}+\|B\|_{2} . ∎ Corollary 1 (Near-interpolation or small-gradient regime) . If the training gradients at the outputs are small so that \gamma\approx 0 , then \big\|\nabla^{2}_{\theta}\mathcal{L}(W_{1:L})\big\|_{2}\;\lesssim\;\beta\sum_{k=1}^{L}\prod_{j\neq k}\bigl(1+\sqrt{\mathrm{DfI}(W_{j})}\bigr). Next, we present the lemma required for the proof of Theorem 3 . Lemma 6 (A basic spectral lemma from DfI) . Let \varepsilon=\sqrt{\mathrm{DfI}(W)} . Then \|W^{\top}W-I\|_{2}\;\leq\;\|W^{\top}W-I\|_{F}\;=\;\varepsilon, hence every eigenvalue \mu of W^{\top}W=S^{2} satisfies 1-\varepsilon\leq\mu\leq 1+\varepsilon . Equivalently, \sqrt{1-\varepsilon}\,I\;\preceq\;S\;\preceq\;\sqrt{1+\varepsilon}\,I. Proof. By the definition of \varepsilon , we have \|W^{\top}W-I\|_{2}\;\leq\;\|W^{\top}W-I\|_{F}\;=\;\varepsilon. Therefore, all eigenvalues \mu of W^{\top}W lie within the interval 1-\varepsilon\;\leq\;\mu\;\leq\;1+\varepsilon. Since W^{\top}W=S^{2} with S\succeq 0 , this is equivalent to the spectral bound \sqrt{1-\varepsilon}\,I\;\preceq\;S\;\preceq\;\sqrt{1+\varepsilon}\,I. ∎ Using this Lemma, we provide proof of Theorem 3 below: Proof of Theorem 3 Proof. Let Z\in\mathbb{R}^{n\times a} be the input matrix and W\in\mathbb{R}^{a\times b} a weight matrix. The resulting feature matrix is \Phi=ZW\in\mathbb{R}^{n\times b} and the empirical covariances are \Sigma_{Z}=\tfrac{1}{n}Z^{\top}Z\in\mathbb{R}^{a\times a},\qquad\Sigma_{\Phi}=\tfrac{1}{n}\Phi^{\top}\Phi=W^{\top}\Sigma_{Z}W\in\mathbb{R}^{b\times b}. Let W=QS denote the right polar decomposition of W , where Q\in\mathbb{R}^{a\times b} has orthonormal columns ( Q^{\top}Q=I_{b} ) and S=(W^{\top}W)^{1/2}\in\mathbb{R}^{b\times b} is positive definite. Then \Sigma_{\Phi}\;=\;W^{\top}\Sigma_{Z}W\;=\;S\,(Q^{\top}\Sigma_{Z}Q)\,S. Write M:=Q^{\top}\Sigma_{Z}Q\succeq 0 , and let its positive eigenvalues be \eta_{1}\geq\cdots\geq\eta_{d}>0 , where d=\operatorname{rank}(M)\leq\min\{b,\operatorname{rank}(\Sigma_{Z})\} . Let \sigma_{1}(\Phi)\geq\cdots\geq\sigma_{d}(\Phi)>0 denote the nonzero singular values of \Phi . For any x\in\mathbb{R}^{b} with \|x\|=1 , x^{\top}\Sigma_{\Phi}x\;=\;x^{\top}SMSx\;=\;(Sx)^{\top}M(Sx). Let y=Sx . Then x^{\top}\Sigma_{\Phi}x=y^{\top}My and, by Lemma 6 , \|y\|^{2}_{2}=\|Sx\|^{2}_{2}\in[\,1-\varepsilon,\,1+\varepsilon\,]. Therefore (1-\varepsilon)\,\lambda_{\min}^{+}(M)\;\leq\;x^{\top}\Sigma_{\Phi}x\;\leq\;(1+\varepsilon)\,\lambda_{\max}(M), where \lambda_{\min}^{+}(M)=\eta_{d} denotes the smallest positive eigenvalue of M (the lower bound is interpreted on the subspace where My\neq 0 ). Taking the maximum over unit x yields \lambda_{\max}(\Sigma_{\Phi})\;\leq\;(1+\varepsilon)\,\eta_{1}, and taking the minimum Rayleigh quotient over the orthogonal complement of \ker(\Sigma_{\Phi}) yields \lambda_{\min}^{+}(\Sigma_{\Phi})\;\geq\;(1-\varepsilon)\,\eta_{d}. Since \sigma_{\max}(\Phi)^{2}=n\,\lambda_{\max}(\Sigma_{\Phi}) and \big(\sigma_{\min}^{+}(\Phi)\big)^{2}=n\,\lambda_{\min}^{+}(\Sigma_{\Phi}) , below inequality holds with d=\operatorname{rank}(M) . \sqrt{n}\,\sqrt{1-\varepsilon}\,\sqrt{\eta_{d}}\;\;\leq\;\;\sigma_{\min}^{+}(\Phi)\;\;\leq\;\;\sigma_{\max}(\Phi)\;\;\leq\;\;\sqrt{n}\,\sqrt{1+\varepsilon}\,\sqrt{\eta_{1}}. Here \sigma_{\min}^{+}(\Phi) denotes the smallest positive singular value of \Phi (defined only when d\geq 1 ). Therefore, if d\geq 1 , then \rho_{\Phi}:=\frac{\sigma_{\max}(\Phi)}{\sigma_{\min}^{+}(\Phi)}\;\;\leq\;\;\sqrt{\frac{1+\varepsilon}{\,1-\varepsilon\,}}\;\cdot\;\sqrt{\frac{\eta_{1}}{\eta_{d}}}. (13) Consider the worst-case allocation of the nonzero singular values that maximizes the cumulative ratio \sum_{i=1}^{k}\sigma_{i}/\sum_{i=1}^{d}\sigma_{i} given a fixed condition number bound \rho_{\Phi} : the top k singular values all equal \sigma_{\max} and the remaining d-k equal \sigma_{\min}^{+} . Then \frac{\sum_{i=1}^{k}\sigma_{i}(\Phi)}{\sum_{i=1}^{d}\sigma_{i}(\Phi)}\;\leq\;\frac{k\,\sigma_{\max}}{k\,\sigma_{\max}+(d-k)\,\sigma_{\min}^{+}}\;=\;\frac{k\,\rho_{\Phi}}{k\,\rho_{\Phi}+(d-k)}. (14) To achieve a coverage level of 1-\delta with k singular values, it is necessary that \frac{k\,\rho_{\Phi}}{k\,\rho_{\Phi}+(d-k)}\;\geq\;1-\delta\quad\Longrightarrow\quad k\;\geq\;\frac{(1-\delta)\,d}{\delta\,\rho_{\Phi}+(1-\delta)}. Taking the ceiling and substituting the bound on \rho_{\Phi} from ( 13 ), establishes the left inequality in ( 3 ) When \Sigma_{Z}=I , we have M=Q^{\top}IQ=I , so \eta_{1}=\cdots=\eta_{b}=1 and d=b . Then, \sqrt{n}\sqrt{1-\varepsilon}\leq\sigma_{i}(\Phi)\leq\sqrt{n}\sqrt{1+\varepsilon}\qquad(\forall i), Which leads to \rho_{\Phi}\leq\sqrt{\tfrac{1+\varepsilon}{1-\varepsilon}} Substituting \rho_{\Phi} in first inequality of ( 3 ) leads to the right inequality ( 3 ). Without whitening, the achievable flattening is limited by the compressed input spectrum M=Q^{\top}\Sigma_{Z}Q . ∎ Next, we present the proof for Theorem 4 . Proof of Theorem 4 Proof. By isotropy and positive homogeneity, there exists a constant c_{\sigma}>0 such that \mathbb{E}_{z}[|\sigma(\langle z,w_{j}\rangle)|]=c_{\sigma}\|w_{j}\| for all j . Hence s_{j}=\|w_{j}\|/\big(\frac{1}{b}\sum_{k}\|w_{k}\|\big) . Let u_{j}=\|w_{j}\|^{2}=[W^{\top}W]_{jj} . Since \mathrm{DfI}(W)=\|W^{\top}W-I\|_{F}^{2}=\sum_{j=1}^{b}(u_{j}-1)^{2}+2\sum_{i0 is a scaling factor and \lambda is the regularization strength. It penalizes the deviation of WW^{\top} from a scaled identity matrix, encouraging the rows of each weight matrix W to be orthogonal and have controlled norms. This constraint keeps the singular values of W close to a constant, preventing gradient explosion or vanishing and leading to more stable and efficient optimization. We used s=1 in all experiments and only swept \lambda . Continual Backpropagation. Continual Backpropagation (CBP) selectively reinitializes low-utility hidden units using a contribution-utility measure ( 10 ) . Contribution-utility scores are computed as an exponential moving average of the unit’s activation magnitude multiplied by the summed magnitude of its outgoing weights. Units with persistently low contribution utility are considered uninformative and are periodically reset. CBP is controlled by two hyperparameters: the maturity threshold m , which protects units from reinitialization for at least m update steps to allow stable utility estimation, and the replacement rate \rho , which determines the expected fraction of units to reset at each update step via fractional accumulation. Recycling Dormant neurons. Recycling Dormant neurons (ReDo) ( 61 ) is another unit-reinitialization method that assigns a neuron score to each hidden unit in every layer, and resets units whose scores fall below the hyperparameter \tau . The neuron score s is computed as the ratio between a unit’s average activation magnitude and the average activation magnitude of all units in the same layer, formally defined as s_{i}^{\ell}=\frac{\mathbb{E}_{x\in D}{|h_{i}^{\ell}(x)|}}{\frac{1}{H^{\ell}}\sum_{k\in h}\mathbb{E}_{x\in D}|h_{k}^{\ell}(x)|} , where h_{i}^{\ell}(x) denotes the activation of neuron i in layer \ell for input x\in D , and H^{\ell} is the number of neurons in layer \ell . L2 Init. L2 Init ( 32 ) , as known as Regen (Regenerative regularization), is a weight regularization method designed to mitigate plasticity loss by leveraging the property that the initial network exhibits the highest plasticity. L2 Init regularizes the weights to stay close to the initial weights by adding a term \lambda\|W-W_{0}\|_{F}^{2} to the loss function, where \lambda is the regularization strength, W is the current weight matrix and W_{0} is initial weight matrix. Self-Normalized Resets. Self-Normalized Resets (SNR) ( 15 ) is a reset-based method that detects inactive neurons by monitoring their firing statistics and statistically testing whether a neuron’s activity is consistent with its past behavior. For each neuron, SNR maintains an empirical distribution of inter-firing times (the number of consecutive updates with zero activation). If the computed probability falls below a threshold 1-\tau , the neuron is classified as inactive and reset. This procedure adaptively replaces neurons whose activity has effectively vanished, mitigating plasticity loss without relying on a fixed, hand-tuned inactivity window. Muon. Muon ( 23 ) is an optimizer that augments SGD with momentum by orthogonalizing its update matrices. Concretely, Muon first forms the usual SGD-momentum update G for each weight matrix and then applies a small fixed number of Newton–Schulz iterations to approximate the closest semi-orthogonal matrix Ortho(G) , effectively replacing G by a matrix with singular values near one while staying close in Frobenius norm. Following the reference implementation, in our experiments we apply Muon only to the middle weight matrices of hidden layers, while scalar and vector parameters, as well as input and output layers, are optimized with AdamW. We set the momentum to 0.95 as recommended by the authors. Plasticity Injection. Plasticity injection ( 49 ) restores neural network’s plasticity by adding a fresh, randomly initialized copy of the prediction head while leaving current predictions unchanged at the moment of the change. The original prediction head is frozen, and two identical new heads are created, one that is allowed to learn and one that always stays fixed. At the start, the learned and fixed new heads cancel each other out, so the overall output of the agent stays exactly the same. As training continues, the learnable new head adapts to new data, giving the agent renewed flexibility, while the original and the fixed new head act as a stable reference. For DQN, we applied plasticity injection to MLP layers, and for SAC, we applied it to whole critic network which is known to suffer from severe plasticity loss ( 44 ) . Appendix E Detailed experiment settings E.1 Continual Visual Learning For continual visual learning, we report the results with 3 seeds. Table 2: Detailed settings in continual visual learning. Parameter Value Optimizer Adam ( 27 ) Learning Rate 1\mathrm{e}{-3} Learning Rate Scheduler Warmup Gradient norm clipping 0.5 Batch Size 256 Epochs per Chunk 100 Data Augmentation False In this section, we describe the detailed settings for conducting continual visual learning. We note that most of the hyperparameters we used were adopted from 33 . For the Warmup scheduler, the learning rate is increased linearly from 0 to the target learning rate during the first 10% of training on each dataset. In other words, in the case of Table 2 , the learning rate is gradually raised from 0 to 1\mathrm{e}{-3} over the first 10 epochs of each data chunk. In the warm-start scenario described in Section 4.1 , we trained for 1000 epochs before new data arrived and for 100 epochs after its arrival, in order to balance the total number of gradient updates before and after the introduction of new data. E.2 Continual Pretraining of LLMs For continual pretraining of LLMs, we report the results with 3 seeds. Table 3: Detailed settings in continual pretraining of LLMs. Parameter Value Optimizer AdamW ( 39 ) Weight Decay 1\mathrm{e}{-1} Learning Rate 6\mathrm{e}{-4} Minimum Learning Rate 6\mathrm{e}{-5} Learning Rate Scheduler Warmup + Linearly Decaying Gradient norm clipping 1.0 Batch Size 480 We used implementation and hyperparameters of nanoGPT from 24 . During first phase, the learning rate linearly increases from 0 to the target learning rate ( 6\mathrm{e}{-4} ) during 2,000 steps, then annealed to minimum learning rate ( 6\mathrm{e}{-5} ) until the end of the phase. In the second phase, the learning rate linearly increases from 0 to the target learning rate ( 6\mathrm{e}{-4} ) during 10% of training iterations of second phase. Then, it decreases linearly to minimum learning rate ( 6\mathrm{e}{-5} ) until the end of the phase. E.3 Reinforcement Learning For reinforcement learning, we report the results with 5 seeds. For S&P method, we apply S&P to the encoder and Reset to the fully connected layers ( 12 ) for discrete control, and S&P with \lambda=0.8 to whole parameters for continuous control tasks. We perform a single intervention (Full Reset, S&P, FIRE) at the midpoint of learning. We followed the hyperparameter configurations used in prior work ( 61 ) . Table 4: Hyperparameters used in the ALE environment with DQN algorithm. Parameter Value Optimizer Adam ( 27 ) Optimizer: \epsilon 1.5\mathrm{e}-4 Optimizer: Learning rate 6.25\mathrm{e}-5 Minimum \epsilon for training 0.01 Evaluation \epsilon 0.001 Discount factor \gamma 0.99 Replay buffer size 10^{6} Minibatch size 32 Initial collect steps 20000 Training iterations 10 Training environment steps per iteration 250K Updates per environment step (Replay Ratio) 1 Target network update period 2000 Loss function Huber Loss ( 22 ) For continuous tasks, the hyperparameter setting is followed by 34 . Table 5: Hyperparameters used in HumanoidBench environments with SimBa. Parameter Value Optimizer AdamW ( 39 ) Optimizer: Learning rate 1\mathrm{e}-4 Optimizer: Weight decay 0.01 Actor hidden dim 128 Actor num blocks 1 Critic hidden dim 512 Critic num blocks 2 Discount factor \gamma 0.99 Clipped Double-Q ( 16 ) True Replay buffer size 10^{6} Minibatch size 256 Initial collect steps 5000 Updates per environment step (Replay Ratio) 2 Soft target update factor \tau 0.005 E.4 Hyperparameter Search Space Table 6 presents the hyperparameter search space, and Tables 7 - 10 present their optimal values. Table 6: Hyperparameter search space for all experiments. Experiment Method Hyperparameters Search Space Warm-Starting S&P \lambda 0.2,0.4,0.6,0.8 DASH \alpha 0.1,0.3 \lambda 0.05,0.1,0.3 Parseval Reg. \lambda 1\mathrm{e}{-3},1\mathrm{e}{-4},1\mathrm{e}{-5} CBP \rho 1\mathrm{e}{-4},1\mathrm{e}{-5} m 100,1000 ReDo \tau 0.01,0.05,0.1,0.5 L2 Init \lambda 1\mathrm{e}{-3},1\mathrm{e}{-4},1\mathrm{e}{-5} SNR \tau 0.01,0.02,0.04,0.08 Continual Learning S&P \lambda 0.2,0.4,0.6,0.8 DASH \alpha 0.1,0.3 \lambda 0.05,0.1,0.3 Parseval Reg. \lambda 1\mathrm{e}{-3},1\mathrm{e}{-4},1\mathrm{e}{-5} CBP \rho 1\mathrm{e}{-4},1\mathrm{e}{-5} m 100,1000 ReDo \tau 0.01,0.05,0.1,0.5 L2 Init \lambda 1\mathrm{e}{-3},1\mathrm{e}{-4},1\mathrm{e}{-5} SNR \tau 0.01,0.02,0.04,0.08 Class-Incremental Learning S&P \lambda 0.2,0.4,0.6,0.8 DASH \alpha 0.1,0.3 \lambda 0.05,0.1,0.3 Parseval Reg. \lambda 1\mathrm{e}{-3},1\mathrm{e}{-4},1\mathrm{e}{-5} CBP \rho 1\mathrm{e}{-4},1\mathrm{e}{-5} m 100,1000 ReDo \tau 0.01,0.05,0.1,0.5 L2 Init \lambda 1\mathrm{e}{-3},1\mathrm{e}{-4},1\mathrm{e}{-5} SNR \tau 0.01,0.02,0.04,0.08 Continual pretraining of GPT-0.1B S&P \lambda 0.2,0.5,0.8 Table 7: Hyperparameters for Warm-Start setting. Dataset Method Value CIFAR-10 S&P \lambda = 0.8 (ResNet-18) DASH \alpha = 0.3 , \lambda = 0.05 Parseval Reg. \lambda = 1\mathrm{e}{-3} CBP \tau=1\mathrm{e}{-4},m=1000 ReDo \tau=0.5 L2 Init \lambda=1\mathrm{e}{-3} SNR \tau=0.01 FIRE \text{iter}=10 CIFAR-100 S&P \lambda = 0.8 (ViT-Tiny) DASH \alpha = 0.1 , \lambda = 0.05 Parseval Reg. \lambda = 1\mathrm{e}{-5} CBP \tau=1\mathrm{e}{-4},m=100 ReDo \tau=0.5 L2 Init \lambda=1\mathrm{e}{-3} SNR \tau=0.01 FIRE \text{iter}=10 Tiny ImageNet S&P \lambda = 0.8 (VGG-16) DASH \alpha = 0.1 , \lambda = 0.05 Parseval Reg. \lambda = 1\mathrm{e}{-3} CBP \tau=1\mathrm{e}{-4},m=1000 ReDo \tau=0.5 L2 Init \lambda=1\mathrm{e}{-3} SNR \tau=0.08 FIRE \text{iter}=10 Table 8: Hyperparameters for Continual Setting. Dataset Method Value CIFAR-10 S&P \lambda = 0.8 (ResNet-18) DASH \alpha = 0.3 , \lambda = 0.05 Parseval Reg. \lambda = 1\mathrm{e}{-3} CBP \tau=1\mathrm{e}{-5},m=1000 ReDo \tau=0.05 L2 Init \lambda=1\mathrm{e}{-5} SNR \tau=0.08 FIRE \text{iter}=10 CIFAR-100 S&P \lambda = 0.8 (ViT-Tiny) DASH \alpha = 0.1 , \lambda = 0.05 Parseval Reg. \lambda = 1\mathrm{e}{-5} CBP \tau=1\mathrm{e}{-5},m=1000 ReDo \tau=0.01 L2 Init \lambda=1\mathrm{e}{-4} SNR \tau=0.04 FIRE \text{iter}=10 Tiny ImageNet S&P \lambda = 0.8 (VGG-16) DASH \alpha = 0.1 , \lambda = 0.1 Parseval Reg. \lambda = 1\mathrm{e}{-4} CBP \tau=1\mathrm{e}{-5},m=1000 ReDo \tau=0.01 L2 Init \lambda=1\mathrm{e}{-5} SNR \tau=0.01 FIRE \text{iter}=10 Table 9: Hyperparameters for Class-Incremental Setting. Dataset Method Value CIFAR-100 S&P \lambda = 0.8 (ViT-Tiny) DASH \alpha = 0.3 , \lambda = 0.3 Parseval Reg. \lambda = 1\mathrm{e}{-5} CBP \tau=1\mathrm{e}{-5},m=1000 ReDo \tau=0.01 L2 Init \lambda=1\mathrm{e}{-5} SNR \tau=0.08 FIRE \text{iter}=10 Tiny ImageNet S&P \lambda = 0.8 (VGG-16) DASH \alpha = 0.3 , \lambda = 0.1 Parseval Reg. \lambda = 1\mathrm{e}{-3} CBP \tau=1\mathrm{e}{-4},m=1000 ReDo \tau=0.05 L2 Init \lambda=1\mathrm{e}{-4} SNR \tau=0.02 FIRE \text{iter}=10 Table 10: Hyperparameters for Continual pretraining of LLMs. Method CKPT Value S&P Best \lambda = 0.5 30k \lambda = 0.8 60k \lambda = 0.5 FIRE Best \text{iter}=5 30k \text{iter}=5 60k \text{iter}=5