On the Stability of the Jacobian Matrix in Deep Neural Networks

Benjamin Dadoun, Soufiane Hayou, Hanan Salam, Mohamed El Amine Seddik, Pierre Youssef · 2025

Read On the Stability of the Jacobian Matrix in Deep Neural Networks by Benjamin Dadoun, Soufiane Hayou, Hanan Salam, Mohamed El Amine Seddik, Pierre Youssef (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-08764.txt.

Introduction

Benjamin Dadoun and four co-authors prove stability results for deep neural networks in a 2025 paper. As networks grow deeper, gradients can explode or vanish during training. That failure ties to how the network's input-output Jacobian behaves across layers.

Earlier work found safe starting settings only for plain fully connected nets with independent random weights. Real networks break both limits through pruning and through training itself. The new theorem covers sparse weights and weakly linked randomness as well. It leans on fresh advances in random matrix theory. The guarantees give modern initialization schemes a firmer base. Designers of pruned or structured networks gain theory that matches their practice more closely.

Catalog record

CREATORBenjamin Dadoun, Soufiane Hayou, Hanan Salam, Mohamed El Amine Seddik, Pierre Youssef
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-08764

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

On the Stability of the Jacobian Matrix in Deep Neural Networks Benjamin Dadoun, Soufiane Hayou, Hanan Salam, Mohamed El Amine Seddik, Pierre Youssef Abstract . Deep neural networks are known to suffer from exploding or vanishing gradients as depth increases, a phenomenon closely tied to the spectral behavior of the input-output Jacobian. Prior work has identified critical initialization schemes that ensure Jacobian stability, but these analyses are typically restricted to fully connected networks with i.i.d. weights. In this work, we go significantly beyond these limitations: we establish a general stability theorem for deep neural networks that accommodates sparsity (such as that introduced by pruning) and non-i.i.d., weakly correlated weights (e.g. induced by training). Our results rely on recent advances in random matrix theory, and provide rigorous guarantees for spectral stability in a much broader class of network models. This extends the theoretical foundation for initialization schemes in modern neural networks with structured and dependent randomness. [ style=plain, sibling=lemma ] 1. Introduction Despite their impressive performance across a wide range of applications, training deep neural networks (DNNs) remains a challenging task, requiring extensive hyperparameter tuning. A key factor that affects the trainability of DNNs is the behavior of the input-output Jacobian, which quantifies the sensitivity of the network’s output to perturbations in its input [ SMG13 ] . Improper initialization can lead to vanishing or exploding Jacobian/gradients as network depth increases, potentially causing the training process to converge to suboptimal local optima or diverge altogether. Prior research on the Jacobian in DNNs has primarily examined its behavior at initialization [ PSG17 , CH23 ] , where network weights are typically drawn from random distributions, such as independent and identically distributed (i.i.d) Gaussians, or random orthogonal matrices [ PSG17 , HN20 , CDK22 ] . These studies have demonstrated, both theoretically and empirically, a correlation between “stable” Jacobian behavior at initialization and improved training and generalization performance. Similar works have investigated other related quantities at initialization such as input-output covariance kernel and gradient propagation dynamics, to derive the “Edge of Chaos” initialization regime [ PLR+16 , SGGSD17 , HDR19 , HDR22 ] , which guarantees gradient stability and improved information propagation through depth. However, these analyses are limited to the setting of random, uncorrelated initializations and do not readily extend to more structured or correlated weight distributions. Yet in many practical scenarios, such as data-dependent initialization [ KDDD15 , CIG22 ] , meta-learning [ FAL17 ] , or fine-tuning from pretrained weights [ LFLY18 ] , networks are initialized with non-i.i.d., often correlated weights [ MM15 ] . Recent theoretical and empirical work has shown that even mild correlations in the weight matrices can significantly alter signal and gradient propagation dynamics, potentially improving convergence and generalization [ JYZ+20 ] . Still, the behavior of the Jacobian in the presence of correlated weights remains poorly understood, particularly in the large-depth regime where signal propagation is most sensitive. Understanding how weight correlations affect Jacobian stability is therefore crucial for explaining the training dynamics of deep networks in realistic settings. Similarly, another underexplored case is that of sparse networks. Pruning is a common technique that sparsifies neural networks to reduce their memory and computational footprint [ LDS90 , HSG93 , LAT18 , WZG20 , FC19 ] . While score-based pruning methods such as SNIP [ LAT18 ] , GraSP [ WZG20 ] , and their variants [ HTDT21 ] aim to preserve important connectivity at initialization, the effect of pruning on the input-output Jacobian remains poorly understood. Existing works suggest that unstructured sparsity can disrupt gradient flow [ EIKD22 ] , but lack theoretical guarantees on Jacobian stability. This gap motivates a closer analysis of Jacobian behavior in pruned networks and other non-standard settings. In this work, we focus on two scenarios: • Correlated weights : Investigating the Jacobian behavior in the large-depth regime with correlated weights can shed light on training stability during the early stages, particularly near initialization. • Sparse networks : Studying the Jacobian of pruned (sparse) networks can provide insights on how to adjust training algorithms to maintain stability after pruning . We address these questions by developing a general theoretical framework (stability theorem, Section 3 ) based on recent breakthroughs in random matrix theory for the products of random matrices in non-i.i.d. settings [ BvH24 ] . Our primary objective is to gain insights into the behavior of the Jacobian under the different scenarios stated above, in the case of Multi-Layer Perceptrons (MLPs). Our analysis reveals several interesting phenomena. Specifically, we show that in sparse networks, the Jacobian behaves similarly to that of a full (non-sparse) network if we scale the non-zero weights suitably, where the scaling factor depends on the pruning method . In Section 5 , we provide an application of our framework in the case of networks with non-i.i.d. initialization weights, and find that there exists a width-dependent correlation threshold that guarantees similar Jacobian properties to the i.i.d. case. Throughout this paper, we consider the Multi-Layer Perceptron (MLP) architecture given by: (1) \texttt{MLP}\triangleright\begin{cases}Y_{0}(x)=W_{\text{in}}\,x,\\ Y_{k}(x)=W_{k}\,\phi(Y_{k-1}(x)),\quad k=1,\dots,L,\\ Y_{\text{out}}(x)=W_{\text{out}}\,\phi(Y_{L}(x)),\end{cases} where x\in\mathbb{R}^{d} is the input, Y_{\text{out}}\in\mathbb{R}^{o} is the network output, W_{\text{in}}\in\mathbb{R}^{n\times d},W_{k}\in\mathbb{R}^{n\times n},W_{\text{out}}\in\mathbb{R}^{o\times n} are the network weights, and \phi is the ReLU activation function given by \phi(z)=\max(z,0) (acting coordinate-wise). For the sake of simplification, we omit here the bias terms in the definition of the MLP. We refer to Y_{k} as the pre-activations or the features and \phi(Y_{k}) as the activations. Hereafter, width and depth will be used to refer to n and L , respectively. In practice, neural networks are usually trained with gradient-based algorithms such as Stochastic Gradient Descent (SGD), Adam, etc. [ LBH15 , KB14 , Bot12 ] . This requires the calculation of the gradients of some loss function with respect to the weights (W_{k})_{1\leq k\leq L} using back-propagation. Let \ell be a loss function (e.g. mean-squared error for regression, and cross-entropy loss for classification) and \mathcal{D}=\{(x_{i},z_{i}),i=1\dots N\} be a fixed training dataset. DNN training aims to minimize the empirical objective \mathcal{L}(\mathbf{W})=N^{-1}\sum_{i=1}^{N}\ell(Y_{\text{out}}(x_{i}),z_{i}), where \mathbf{W}=\{W_{\text{in}},(W_{k})_{1\leq k\leq L},W_{\text{out}}\} . With GD, the parameters \mathbf{W} are updated with the rule: \mathbf{W}\leftarrow\mathbf{W}-\eta\frac{\partial\mathcal{L}}{\partial\mathbf{W}}. For a datapoint (x,z) , the gradient of the loss function evaluated at (x,z) w.r.t. the weights W_{k}^{i,j} (for some i,j\in\{1,\dots,n\} ) is given by: \frac{\partial\ell(Y_{\text{out}}(x),z)}{\partial W_{k}^{i,j}}=\frac{\partial\ell(Y_{\text{out}}(x),z)}{\partial Y_{k}^{i}(x)}\,\phi(Y_{k-1}^{j}(x))=\frac{\partial\ell(Y_{\text{out}}(x),z)}{\partial Y_{L}(x)}^{\top}\frac{\partial Y_{L}(x)}{\partial Y_{k}^{i}(x)}\,\phi(Y_{k-1}^{j}(x)). Hence, the gradients inherently depend on the Jacobian terms J_{k}(x)=\frac{\partial Y_{L}(x)}{\partial Y_{k-1}(x)}=\left(\frac{\partial Y_{L}^{i}(x)}{\partial Y_{k-1}^{j}(x)}\right)_{1\leq i,j\leq n}\in\mathbb{R}^{n\times n}, for k\in\{1,2,\dots,L\} . Using the chain rule, it is easy to see that for k\in\{1,\dots,L-1\} , the Jacobian satisfies the recursion J_{k}(x)=J_{k+1}(x)\times W_{k}\,D_{k-1}(x) , where D_{k}(x)=\textup{Diag}(\phi^{\prime}(Y_{k}(x)))\in\mathbb{R}^{n\times n} . Thus, we can express the Jacobian terms as a matrix product: (2) J_{k}(x)=\prod_{l=k}^{L}W_{l}D_{l-1}(x),\quad k\in\{1,\dots,L-1\}, Hereafter, we will denote the Jacobian without reference to its input x , and use J_{1}=\prod_{l=1}^{L}W_{l}D_{l-1} , as defined earlier. We assume a fixed non-zero input x for the analysis. Nevertheless, as we show in our empirical results, our findings hold for randomly selected inputs from the dataset, demonstrating that our conclusions are independent of the input choice 1 1 1 It should be noted that while the input can impact stability in a DNN, this impact is often negligible if the input is normalized, with the architecture and weight distribution playing a more significant role. . Due

Read the complete text