Chebyshev Moment Regularization (CMR): Condition-Number Control with Moment Shaping
Jinwoo Baek · 2025
Read Chebyshev Moment Regularization (CMR): Condition-Number Control with Moment Shaping by Jinwoo Baek (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-21772.txt.
Introduction
Jinwoo Baek proposes a training aid for neural networks called Chebyshev Moment Regularization, or CMR. It works with any network shape. The added loss term does not just chase lower prediction error. It directly shapes the spread of values inside each layer. Keeping that spread under control is the whole point.
Baek proves several guarantees for the method. They include steady improvement of the control measure and bounded update sizes. In a harsh test, plain training on handwritten digits collapsed. The test used a deep fifteen-layer network. CMR cut the layer condition numbers by roughly a factor of one thousand. It restored test accuracy from about ten percent to about eighty-six percent. Steering networks toward tame internal ranges, Baek argues, makes learning more stable.
Catalog record
| CREATOR | Jinwoo Baek |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-21772
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Chebyshev Moment Regularization (CMR): Condition-Number Control with Moment Shaping Jinwoo Baek Abstract We introduce Chebyshev Moment Regularization (CMR) , a simple, architecture-agnostic loss that directly optimizes layer spectra. CMR jointly controls spectral edges via a log-condition proxy and shapes the interior via Chebyshev moments, with a decoupled, capped mixing rule that preserves task gradients. We prove strictly monotone descent for the condition proxy, bounded moment gradients, and orthogonal invariance. In an adversarial “ κ \kappa -stress” setting (MNIST, 15-layer MLP), compared to vanilla training , CMR reduces mean layer condition numbers by ∼ 10 3 \sim\!10^{3} (from ≈ × 10 3 \approx 3.9\!\times\!10^{3} to ≈ 3.4 \approx 3.4 in 5 epochs), increases average gradient magnitude, and restores test accuracy ( ≈ 10 % → ≈ 86 % \approx 10\%\!\to\!\approx 86\% ). These results support optimization-driven spectral preconditioning : directly steering models toward well-conditioned regimes for stable, accurate learning. \optauthor School of Electrical Engineering and Computer Science Department of Mathematics Oregon State University Corvallis, OR 97331, USA 1 Introduction Training very deep networks is brittle: layer spectra often turn pathological— \sigma_{\min} collapses, \sigma_{\max} inflates, and condition numbers explode—yielding poorly scaled gradients and stalled optimization. Common fixes such as residual connections and normalization ( He et al. 2016 ; Ioffe and Szegedy 2015 ) or careful initialization ( Glorot and Bengio 2010 ) help indirectly; they do not directly optimize the spectral geometry that governs numerical stability. We propose Chebyshev Moment Regularization ( CMR ) , a drop-in loss that treats the spectrum as a first-class training signal. CMR couples a log-condition proxy that targets the spectral edges ( \sigma_{\max},\sigma_{\min} ) with Chebyshev moments of a normalized Gram operator that shape the interior distribution. A decoupled, capped mixing rule scales spectral gradients relative to task gradients, preserving task signal while keeping spectral intervention bounded. Unlike spectral-norm regularization, which constrains only the operator norm \|W\|_{2} ( Miyato et al. 2018 ) , or orthogonality/Parseval penalties that drive layers toward (near-)isometries ( Cisse et al. 2017 ) , CMR provides fine-grained, orthogonally invariant control over both edges and mass with minimal implementation burden. Contributions. • Method. CMR is a lightweight, drop-in loss that directly reshapes layer spectra—controlling the edges and smoothing the interior—while mixing with task gradients under a small cap to preserve training dynamics; applicable across architectures and optimizers in formulation. • Theory. (a) Gradient flow on \rho_{\text{cond}} satisfies a strict descent identity (monotonic conditioning improvement); (b) moment gradients are bounded and scale-friendly; (c) the penalty is invariant under QWR for orthogonal Q,R . • Practice. In adversarial “ \kappa -stress,” CMR reconditions layers by orders of magnitude while increasing average gradient norms, restoring trainability and substantially improving test accuracy relative to vanilla training. 2 Method: Chebyshev Moment Regularization Let W\in\mathbb{R}^{m\times n} be a layer and G=W^{\top}W . Define \rho_{\text{cond}}(W)=\tfrac{1}{2}\Big(\log\left\lVert W\right\rVert_{2}^{2}-\log\lambda_{\min}(W^{\top}W+\epsilon I)\Big)=\log\sigma_{\max}(W)-\tfrac{1}{2}\log(\sigma_{\min}^{2}(W)+\epsilon), \rho_{\text{moment}}(W)=\sum_{k=3}^{K}w_{k}\,s_{k}(W)^{2},\qquad s_{k}(W)=\tfrac{1}{n}\mathrm{Tr}\!\big(T_{k}(\widehat{G})\big), where \epsilon>0 , w_{k}=\exp(\beta(k-3)) , n=\text{cols}(W) , and \widehat{G}=\frac{G-cI}{d} with c=\frac{\lambda_{\max}(G)+\lambda_{\min}(G)}{2} and d=\max\{\frac{\lambda_{\max}(G)-\lambda_{\min}(G)}{2},\epsilon\} so that \sigma(\widehat{G})\subseteq[-1,1] . The training objective is \mathcal{L}(\theta)=\mathcal{L}_{\text{task}}(\theta)\;+\;\lambda\sum_{\ell=1}^{L}\left(\alpha_{1}\,\rho_{\text{cond}}(W^{(\ell)})+\alpha_{2}\,\rho_{\text{moment}}(W^{(\ell)})\right). On using k\geq 3 moments. We use k\!\geq\!3 because s_{0},s_{1},s_{2} encode mass/mean/variance of the normalized spectrum and are largely determined by the edge normalization and the condition proxy (App. A ). Decoupled, capped mixing. We backprop the task loss to get g_{\text{task}} , then backprop the spectral penalty to get g_{\text{spec}} . Let \rho_{\text{spec}}\!\in(0,1] and set \widetilde{g}_{\text{spec}}=\min\!\Big\{1,\tfrac{\rho_{\text{spec}}\left\lVert g_{\text{task}}\right\rVert}{\left\lVert g_{\text{spec}}\right\rVert}\Big\}\,g_{\text{spec}}, and update with g=\widetilde{g}_{\text{spec}}+g_{\text{task}} . This preserves task signal while enforcing bounded spectral intervention . We use a small stabilizer \delta=10^{-12} in the cap denominator. {algorithm2e} [H] CMR -SGD with decoupled, capped spectral gradients \KwIn model \theta , weights (\lambda,\alpha_{1},\alpha_{2}) , cap \rho_{\text{spec}} , warmup T_{\text{w}} \For steps t=0,1,2,\dots Compute g_{\text{task}}\leftarrow\nabla_{\theta}\mathcal{L}_{\text{task}} \lambda_{t}\leftarrow\lambda\cdot\min\{1,t/T_{\text{w}}\} Compute g_{\text{spec}}\leftarrow\nabla_{\theta}\sum_{\ell}(\alpha_{1}\rho_{\text{cond}}+\alpha_{2}\rho_{\text{moment}}) Scale: \gamma\leftarrow\min\{1,\rho_{\text{spec}}\left\lVert g_{\text{task}}\right\rVert/(\left\lVert g_{\text{spec}}\right\rVert+\delta)\} ; \widetilde{g}_{\text{spec}}\!\leftarrow\!\lambda_{t}\gamma g_{\text{spec}} Update with g=\widetilde{g}_{\text{spec}}+g_{\text{task}} 3 Theory (proof sketches) We show: (i) the condition proxy decreases under its gradient flow, (ii) moment gradients are bounded under mild spread, and (iii) the penalty is orthogonally invariant. Proofs are in App. B ; the rationale for using k\!\geq\!3 moments is in App. A . Assumptions and notation. We take \epsilon>0 so that \lambda_{\min}(W^{\top}W+\epsilon I)>0 . Gradients of spectral terms are well-defined whenever the extremal singular values are simple; otherwise, statements hold almost everywhere and extend via Clarke subgradients of spectral functions ( Lewis 1996 ; Sun 1988 ; Bhatia 1997 ) . For Lemma 3 we assume a nontrivial spectral spread \lambda_{\max}(G)-\lambda_{\min}(G)\geq\theta\,\lambda_{\max}(G) for some \theta\in(0,1] ; see Remark 4 for the \theta\to 0 case. Theorem 1 (Monotone Descent for the Condition Proxy) . Let W(t) evolve under the gradient flow \dot{W}(t)=-\eta\nabla_{W}\rho_{\text{cond}}(W(t)) for \eta>0 . The condition proxy exhibits a strict descent property: \frac{d}{dt}\rho_{\text{cond}}(W(t))=-\eta\left\lVert\nabla_{W}\rho_{\text{cond}}(W(t))\right\rVert_{F}^{2}\leq 0. Proof Sketch. The result is an exact dissipation identity derived from the chain rule: \frac{d}{dt}\rho_{\text{cond}}=\langle\nabla\rho_{\text{cond}},\dot{W}\rangle_{F}=\langle\nabla\rho_{\text{cond}},-\eta\nabla\rho_{\text{cond}}\rangle_{F} . This confirms that minimizing \rho_{\text{cond}} is a well-posed objective that directly pushes the model towards better-conditioned states. \square Corollary 2 (Control over the Log-Condition Number) . The condition proxy \rho_{\text{cond}} and the true log-condition number \log\kappa(W) are related by the identity \log\kappa(W)=\rho_{\text{cond}}(W)+\frac{1}{2}\log(1+\epsilon/\sigma_{\min}^{2}(W)) . Thus, the monotone decrease of \rho_{\text{cond}} guaranteed by Theorem 1 forces a non-increasing trend in \log\kappa(W) , bounded by an additive term that vanishes as \epsilon\to 0 . Discrete-step behavior. While Theorem 1 is stated for gradient flow, in our experiments we observe a discrete-step monotonic trend for \rho_{\text{cond}} under standard optimizers (Sec. 5 ), supporting its use as a direct conditioning signal. Note. The identity holds pointwise; the monotonicity conclusion applies almost everywhere along trajectories where \rho_{\text{cond}} is differentiable, and extends in the sense of energy dissipation inequalities using Clarke subgradients ( Lewis 1996 ) . Lemma 3 (Moment Gradients are Bounded and Scale-Friendly) . Assume the spectral spread \lambda_{\max}(G)-\lambda_{\min}(G)\geq\theta\,\lambda_{\max}(G) for some \theta\in(0,1] . Then the Frobenius norm of the moment penalty’s gradient satisfies \left\lVert\nabla_{W}\rho_{\text{moment}}\right\rVert_{F}\leq\frac{C\cdot K}{\left\lVert W\right\rVert_{2}}+\mathcal{O}(\left\lVert W\right\rVert_{2}^{-3}). Moreover, without the spread assumption one always has the general bound \|\nabla_{W}\rho_{\text{moment}}\|_{F}\;\leq\;\frac{C^{\prime}}{\sqrt{n}\,\epsilon}\,K\,\|W\|_{2}, so the claimed 1/\|W\|_{2} decay is precisely the regime where the affine normalization is edge–dominated (i.e., d=\Theta(\|W\|_{2}^{2}) ). Proof Sketch. The proof proceeds in three steps. (1) The gradient of the trace term is \nabla_{\widehat{G}}\mathrm{Tr}(T_{k}(\widehat{G}))=T_{k}^{\prime}(\widehat{G}) . (2) The derivative of a Chebyshev polynomial satisfies T_{k}^{\prime}(x)=k\,U_{k-1}(x) with \sup_{x\in[-1,1]}|U_{k-1}(x)|=k , hence \|T_{k}^{\prime}(\cdot)\|_{\infty}\leq k^{2} ( Rivlin 1990 , Thm. 1.2) . (3) The affine scaling denominator d=\max\{\frac{\lambda_{\max}-\lambda_{\min}}{2},\epsilon\} scales as \Theta(\left\lVert W\right\rVert_{2}^{2}) under the spread assumption. Combining these facts via the chain rule for matrix calculus ( dG=W^{\top}dW+dW^{\top}W ) yields the stated 1/\left\lVert W\right\rVert_{2} decay; terms involving gradients of c,d are lower order ( \mathcal{O}(d^{-2}) ). The alternative bound follows by taking d=\epsilon . \square Remark 4 . When the spectrum is nearly degenerate (spread \to 0 ), d is set by \epsilon and the bound becomes linear in \|W\|_{2} . In practice we keep \epsilon small and rely on the condition proxy to widen the edges, quickly entering the favorable 1/\|W\|_{2} regime. Proposition 5 (Orthogonal Invariance) . The CMR penalty is invariant under orthogonal transformations. For any orthogonal matrices Q,R of appropriate dimensions, \rho_{\text{cond}}(QWR)=\rho_{\text{cond}}(W)\quad\text{and}\quad\rho_{\text{moment}}(QWR)=\rho_{\text{moment}}(W). Proof Sketch. The singular values of W are invariant to orthogonal transformations, which proves invariance of \rho_{\text{cond}} . For the moment term, the Gram matrix transforms as G_{QWR}=R^{\top}G_{W}R . By the functional calculus for matrix polynomials ( Higham 2008 , Chap. 1) , T_{k}(R^{\top}\widehat{G}_{W}R)=R^{\top}T_{k}(\widehat{G}_{W})R . Using the cyclic property of the trace, s_{k}(QWR)=\tfrac{1}{n}\mathrm{Tr}(T_{k}(\widehat{G}_{QWR}))=\tfrac{1}{n}\mathrm{Tr}(T_{k}(\widehat{G}_{W}))=s_{k}(W) . Thus \rho_{\text{moment}}