Internal Examiner : Abstract \thetitle by \theauthor A thesis presented in fulfillment of the requirements for the degree of Doctor of Philosophy at Ollscoil na Gaillimhe University of Galway Ireland and Taighde Éireann – Research Ireland Irish National Centre for Research Training in AI (CRT-AI) Doctoral Supervisors : Dr. Colm O’Riordan Dr. James McDermott Scoil na Ríomheolaíochta \thedate School of Computer Science Cite as: \theauthor . \thetitle . Doctoral Thesis, Research Ireland – Centre for Research Training in Artificial Intelligence (CRT-AI), University of Galway, Ireland, \thedate . To my beloved grandparents. Voyagers: Edge of the Bubble (NASA/JPL-Caltech) 1 1 1 Image use follows the NASA Images and Media Usage Guidelines . . Two tiny spacecraft — the Voyagers — manifest our spirit of exploration to an extraordinary level, a spirit that defines us as human beings. They have been sailing outward into vast interstellar space, reaching the frontier of our solar system, where the star winds meet the solar winds from our world, forming a vast bubble. Within this bubble lies every story, every life, every dream we have ever known — yet the Voyagers press onward, carrying our boldness and wonder into the vast, eternal, and spectacular sea between the stars, destined for an eternal journey, a testament to our instinct to go beyond the horizon where no human has gone before. I am only stardust — learning to think, to chase dreams. “Nunquam praescriptos transibunt sidera fines — Nothing exceeds the limits of the stars.” Jules Henri Poincaré, 1890 Research Funding This Ph.D. training program is a part of the Centre for Research Training in Artificial Intelligence ( CRT-AI ), funded by Taighde Éireann – Research Ireland under Grant Number 18/CRT/6223. Abstract Deep learning has achieved remarkable success across a wide range of domains, significantly expanding the frontiers of what is achievable in artificial intelligence. Yet, despite these advances, critical challenges remain — most notably, ensuring robustness to small input perturbations and generalization to out-of-distribution data. These critical challenges underscore the need to understand the underlying fundamental principles that govern robustness and generalization. This understanding is indispensable for establishing deep learning systems that are: reliable — performing consistently under expected conditions on in-distribution data; resilient — capable of recovering from unexpected conditions such as noise or adversarial attacks; and trustworthy — behaving transparently, ethically, in alignment with intended use, and technically robust, particularly in safety-critical applications. Among the theoretical tools available, Lipschitz continuity plays a pivotal role in governing the fundamental properties of neural networks related to robustness and generalization. It quantifies the worst-case sensitivity of network’s outputs to small input perturbations. While its importance is widely acknowledged, prior research has predominantly focused on empirical regularization approaches based on Lipschitz constraints, leaving the underlying principles less explored. This thesis seeks to advance a principled understanding of the principles of Lipschitz continuity in neural networks within the paradigm of machine learning, examined from two complementary perspectives : an internal perspective — focusing on the temporal evolution of Lipschitz continuity in neural networks during training ( i.e. , training dynamics ); and an external perspective — investigating how Lipschitz continuity modulates the behavior of neural networks with respect to features in the input data, particularly its role in governing frequency signal propagation ( i.e. , modulation of frequency signal propagation ). Guided by these perspectives, the thesis formulates three primary research questions: (RQ1) State of Knowledge — what is the state of knowledge of Lipschitz continuity in neural networks? (RQ2) Training Dynamics — how does Lipschitz continuity in neural networks evolve during the training process? (RQ3) Modulation of Frequency Signal Propagation — how does Lipschitz continuity modulate frequency signal propagation in neural networks? Declaration I, \theauthor , declare that this thesis, titled “ \thetitle ”, is composed by myself, that the work contained herein is my own except where explicitly stated otherwise in the text, and that this work has not been submitted for any other degree or professional qualification. Galway, \thedate \theauthor List of publications Articles Compiled in This Thesis This thesis compiles and presents the following selected articles — published or under review as of the date of submission — into a self-contained , modular , coherent , monograph-style presentation format, all centered on the principles of Lipschitz continuity in neural networks : I II III IV Articles Not Compiled in This Thesis Although the following published conference articles and the manuscript in preparation were produced during the Ph.D. training of the author, they are not included in this thesis due to their topics and the thesis’s length constraints: V 1 Introduction 2 Literature Review Copyright Disclaimer . This chapter includes content adapted from the following article: 1. Portions of the content are reused verbatim or with minor edits, in accordance with institutional guidelines and the policies of the relevant publishers. Permission from all co-authors has been obtained. Guiding Research Question. This chapter addresses the primary research question — (RQ1) State of Knowledge — by answering two interrelated secondary research questions through a comprehensive literature study: (SRQ1) Theoretical Foundations and (SRQ2) Regularization Mechanism . Reproducibility. https://github.com/roisincrtai/activation_lipschitz ABSTRACT Lipschitz continuity is a fundamental property of neural networks that characterizes their sensitivity to input perturbations. It plays a pivotal role in deep learning, governing robustness , generalization and optimization dynamics . Despite its importance, research on Lipschitz continuity is scattered across various domains, lacking a unified perspective. This paper addresses this gap by providing a systematic review of Lipschitz continuity in deep learning. We explore its theoretical foundations , estimation methods , regularization approaches , and certifiable robustness . By reviewing existing research through the lens of Lipschitz continuity, this survey serves as a comprehensive reference for researchers and practitioners seeking a deeper understanding of Lipschitz continuity and its implications in deep learning. Appendix 2.A Appendix Appendix 2.B Proofs: Lipschitz Constants of Activation Functions 2.b.1 Proof: Lipschitz Constant of Sigmoid Proof. Consider the sigmoid function: \displaystyle\mathrm{Sigmoid}(x)=f(x)=(1+e^{-x})^{-1}. (2.1) Let u=1+e^{-x} , so f(x)=u^{-1} . The derivative is: \displaystyle f^{\prime}(x)=-\frac{1}{u^{2}}\cdot\frac{du}{dx}. (2.2) Compute \frac{du}{dx} : \displaystyle u=1+e^{-x},\quad\frac{du}{dx}=-e^{-x}. (2.3) Thus: \displaystyle f^{\prime}(x)=-\frac{1}{(1+e^{-x})^{2}}\cdot(-e^{-x})=\frac{e^{-x}}{(1+e^{-x})^{2}}. (2.4) Express the derivative in terms of f(x) : \displaystyle 1-f(x)=\frac{e^{-x}}{1+e^{-x}}, (2.5) and: \displaystyle f^{\prime}(x)=\frac{1}{1+e^{-x}}\cdot\frac{e^{-x}}{1+e^{-x}}=f(x)(1-f(x)). (2.6) Thus: \displaystyle|f^{\prime}(x)|=f(x)(1-f(x)). (2.7) Maximize the Derivative. Since 00 and 1+e^{x}>1 , we have f^{\prime}(x)>0 , so: \displaystyle|f^{\prime}(x)|=\frac{e^{x}}{1+e^{x}}. (2.27) Maximize the Derivative. To find the Lipschitz constant, we need to compute: \displaystyle K=\sup_{x\in\mathbb{R}}\frac{e^{x}}{1+e^{x}}. (2.28) Notice that \frac{e^{x}}{1+e^{x}} is the sigmoid function, which ranges between 0 and 1. Analyze its behavior: 1. As x\to\infty , e^{x} grows large, so: \displaystyle\frac{e^{x}}{1+e^{x}}\approx\frac{e^{x}}{e^{x}}=1. (2.29) 2. As x\to-\infty , e^{x}\to 0 , so: \displaystyle\frac{e^{x}}{1+e^{x}}\to\frac{0}{1}=0. (2.30) To confirm the supremum, consider the function g(x)=\frac{e^{x}}{1+e^{x}} . Its derivative is: \displaystyle g^{\prime}(x)=\frac{e^{x}(1+e^{x})-e^{x}\cdot e^{x}}{(1+e^{x})^{2}}=\frac{e^{x}}{(1+e^{x})^{2}}. (2.31) Since g^{\prime}(x)>0 for all x , g(x) is strictly increasing, approaching 0 as x\to-\infty and 1 as x\to\infty . Thus: \displaystyle\sup_{x\in\mathbb{R}}\frac{e^{x}}{1+e^{x}}=1 (2.32) Hence: \displaystyle\mathrm{Lip}\left[\mathrm{Softplus}(x)\right]=\sup_{x}|f^{\prime}(x)|=1. (2.33) ∎ 2.b.4 Proof: Lipschitz Constant of Swish Proof. Let \displaystyle f(x)=x\,\sigma(x),\quad\sigma(x)=\frac{1}{1+e^{-x}}. (2.34) The derivative is \displaystyle g(x):=f^{\prime}(x)=\sigma(x)+x\,\sigma(x)\bigl(1-\sigma(x)\bigr). (2.35) Since \sigma(-x)=1-\sigma(x) , we have \displaystyle g(-x)=1-g(x). (2.36) Using \sigma(x)=\frac{1}{2}\bigl(1+\tanh(\frac{x}{2})\bigr) and \sigma(x)(1-\sigma(x))=\frac{1}{4}\operatorname{sech}^{2}(\frac{x}{2}) , \displaystyle g(x)=\frac{1}{2}\Bigl(1+\tanh\!\frac{x}{2}\Bigr)+\frac{x}{4}\,\operatorname{sech}^{2}\!\frac{x}{2}. (2.37) Differentiating, \displaystyle g^{\prime}(x)=\frac{1}{4}\,\operatorname{sech}^{2}\!\frac{x}{2}\,\Bigl(2-x\,\tanh\!\frac{x}{2}\Bigr), (2.38) so the maximizer x^{\star}>0 satisfies \displaystyle x^{\star}\,\tanh\!\frac{x^{\star}}{2}=2. (2.39) Using \tanh(\frac{x^\star}{2})=\frac{2}{x^{\star}} and \displaystyle\operatorname{sech}^{2}\!\frac{x^{\star}}{2}=\frac{(x^{\star})^{2}-4}{(x^{\star})^{2}}. (2.40) Thus \displaystyle K=g(x^{\star})=\frac{1}{2}+\frac{x^{\star}}{4}. (2.41) Since g(-x)=1-g(x) , x^{\star} gives the global maximum of |g| , solving x^{\star}\,\tanh\!\frac{x^{\star}}{2}=2 gives: \displaystyle x^{\star}\approx 2.3993572805\cdots. (2.42) Hence: \displaystyle\mathrm{Lip}\left[\mathrm{Siwsh}(x)\right]=\sup_{x}|f^{\prime}(x)|\approx 1.09983932\cdots (2.43) ∎ 2.b.5 Proof: Lipschitz constant of GELU Proof. Let: \displaystyle f(x)=x\,\Phi(x), (2.44) where: \displaystyle\Phi(x)=\tfrac{1}{2}\bigl(1+\operatorname{erf}(x/\sqrt{2})\bigr) (2.45) is the standard normal CDF and: \displaystyle\phi(x)=\tfrac{1}{\sqrt{2\pi}}e^{-x^{2}/2} (2.46) is its PDF. Differentiate: \displaystyle g(x):=f^{\prime}(x)=\Phi(x)+x\,\phi(x). (2.47) Note the symmetry \Phi(-x)=1-\Phi(x) and \phi(-x)=\phi(x) , hence \displaystyle g(-x)=1-g(x). (2.48) Compute the critical points: \displaystyle g^{\prime}(x)=\phi(x)+\phi(x)+x\,\phi^{\prime}(x)=2\phi(x)-x^{2}\phi(x)=\phi(x)\,(2-x^{2}). (2.49) Since \phi(x)>0 , we have g^{\prime}(x)>0 for |x|<\sqrt{2} and g^{\prime}(x)<0 for |x|>\sqrt{2} . Hence g attains its unique global maximum at x^{\star}=\sqrt{2} and minimum at -\sqrt{2} . Since g(-x)=1-g(x) , the minimum equals 1-g(\sqrt{2}) , so indeed \sup_{x}|g(x)|=g(\sqrt{2}) . Evaluate at: \displaystyle x=\sqrt{2}\approx 1.414224\cdots, (2.50) hence: \displaystyle\mathrm{Lip}\left[\mathrm{GELU}(x)\right] \displaystyle=\sup_{x}|f^{\prime}(x)| (2.51) \displaystyle=g(\sqrt{2}) (2.52) \displaystyle=\Phi(\sqrt{2})+\sqrt{2}\,\phi(\sqrt{2}) (2.53) \displaystyle=\tfrac{1}{2}\bigl(1+\operatorname{erf}(1)\bigr)+\frac{e^{-1}}{\sqrt{\pi}} (2.54) \displaystyle\approx 1.128904145. (2.55) ∎ 2.b.6 Proof: Lipschitz constant of Softmax Proof. Let \displaystyle\operatorname{softmax}(z)_{i}=\frac{e^{z_{i}}}{\sum_{j=1}^{n}e^{z_{j}}}, (2.56) and: \displaystyle p:=\operatorname{softmax}(z)\in\Delta^{n-1}, (2.57) where \Delta^{n-1} is the probability simplex in \mathbb{R}^{n} ( i.e. , the set of all probability vectors of length n ): \displaystyle\Delta^{n-1}\;:=\;\big\{\,p\in\mathbb{R}^{n}\ \big|\ p_{i}\geq 0,\ \sum_{i=1}^{n}p_{i}=1\,\big\}. (2.58) The Jacobian is \displaystyle J(z)=\nabla\operatorname{softmax}(z)=\mathrm{diag}(p)-p\,p^{\top}, (2.59) which is symmetric positive semidefinite (PSD) and satisfies: \displaystyle J(z)\,\mathbf{1}=0. (2.60) Hence the Lipschitz constant K is: \displaystyle K=\sup_{z\in\mathbb{R}^{n}}\|J(z)\|_{2}=\sup_{p\in\Delta^{n-1}}\lambda_{\max}\!\big(\mathrm{diag}(p)-p\,p^{\top}\big). (2.61) where \lambda_{\max} is the largest singular value of \mathrm{diag}(p)-p\,p^{\top} . For any unit vector v\in\mathbb{R}^{n} , \displaystyle v^{\top}J(z)\,v=v^{\top}\!\mathrm{diag}(p)\,v-(p^{\top}v)^{2}=\sum_{i=1}^{n}p_{i}v_{i}^{2}-\Big(\sum_{i=1}^{n}p_{i}v_{i}\Big)^{2}=\mathrm{Var}_{i\sim p}[v_{i}]. (2.62) Therefore \displaystyle\|J(z)\|_{2}=\sup_{\|v\|_{2}=1}\mathrm{Var}_{i\sim p}[v_{i}]. (2.63) By Popoviciu’s inequality on variances, for any random variable X supported in [a,b] , \displaystyle\mathrm{Var}[X]\leq\frac{(b-a)^{2}}{4}, (2.64) with equality attained by a two–point distribution at the endpoints. Applying this to X=v_{i} under p , we get \displaystyle\mathrm{Var}_{i\sim p}[v_{i}]\leq\frac{\big(\max_{i}v_{i}-\min_{i}v_{i}\big)^{2}}{4}. (2.65) Maximizing the RHS over \|v\|_{2}=1 yields \max_{i}v_{i}-\min_{i}v_{i}\leq\sqrt{2} , with equality for \displaystyle v=\frac{1}{\sqrt{2}}(e_{a}-e_{b})\quad\text{for some distinct }a,b, (2.66) so that \displaystyle\sup_{\|v\|_{2}=1}\mathrm{Var}_{i\sim p}[v_{i}]\leq\frac{(\sqrt{2})^{2}}{4}=\frac{1}{2}. (2.67) This upper bound is (arbitrarily) attainable by choosing p supported on the two indices a,b with p_{a}=p_{b}=\tfrac{1}{2} and p_{i}\to 0 for i\notin\{a,b\} (which is approached by softmax logits z_{a}=z_{b}\gg z_{i\notin\{a,b\}} ). In that case, \displaystyle J=\begin{bmatrix}\frac{1}{4}&-\frac{1}{4}\\[2.0pt] -\frac{1}{4}&\frac{1}{4}\end{bmatrix}\oplus 0_{n-2}, (2.68) whose largest eigenvalue is \tfrac{1}{2} . Hence: \displaystyle\mathrm{Lip}\left[\mathrm{Softmax}(x)\right]=\sup_{z}\|J(z)\|_{2}=\frac{1}{2}, (2.69) independent of n\geq 2 . ∎ 3 Operator-Theoretic Perturbation Analysis Copyright Disclaimer . This chapter includes content adapted from the following article: (i) Portions of the content are reused verbatim or with minor edits, in accordance with institutional guidelines and the policies of the relevant publishers. Permission from all co-authors has been obtained. Guiding Research Question. This chapter addresses the secondary research question (SRQ3) Parameter Space Perturbation , in service of the primary research question — (RQ2) Training Dynamics . Using tools from operator theory and perturbation theory, it establishes a theoretical framework for analyzing perturbations in matrices. In the next chapter, the analytic operator-norm Hessian formulation enables further theoretical analysis of the temporal evolution of Lipschitz continuity in neural networks. Reproducibility. https://github.com/roisincrtai/highorder_spectral_variation_analysis ABSTRACT Higher-order derivatives of singular values in real rectangular matrices arise naturally in both numerical simulation and theoretical analysis, with applications in areas such as statistical physics and optimization in deep learning. Deriving closed-form expressions beyond first order has remained a difficult problem within classical matrix analysis, and no general framework has been available. To address this gap, we present an operator-theoretic framework that extends Kato’s analytic perturbation theory from self-adjoint operators to real rectangular matrices, thereby yielding general n -th order Fréchet derivatives of singular values. As a special case, we obtain a closed-form Kronecker-product representation of the singular-value Hessian, not previously found in the literature. This framework bridges abstract perturbation theory with matrix analysis and provides a systematic tool for higher-order spectral analysis. 4 Stochastic Dynamic Analysis of Lipschitz Continuity Copyright Disclaimer . This chapter includes content adapted from the following article: 1. Portions of the content are reused verbatim or with minor edits, in accordance with institutional guidelines and the policies of the relevant publishers. Permission from all co-authors has been obtained. Authorship Statement . This chapter is the result of collaborative research. Contributions from the authors are stated as below: (i) Róisín Luo, affiliated with University of Galway – Ireland and Irish National Centre for Research Training in AI , made the primary contributions, including conceptualization, theoretical analysis, drafting, etc. (ii) Dr. James McDermott, affiliated with University of Galway – Ireland and Irish National Centre for Research Training in AI , provided supervision, contributed to the refinement of the manuscript, offered academic guidance, and critically reviewed the content. (iii) Dr. Christian Gagné, affiliated with Université Laval – Canada , Canada – CIFAR AI Chair , and Mila – Québec AI Institute , contributed to the refinement of the manuscript, offered academic guidance, and critically reviewed the content. (iv) Dr. Qiang Sun, affiliated with University of Toronto - Canada , and MBZUAI – UAE , contributed to the refinement of the manuscript, offered academic guidance, and critically reviewed the content. (v) Dr. Colm O’Riordan, affiliated with University of Galway – Ireland and Irish National Centre for Research Training in AI , provided supervision, contributed to the refinement of the manuscript, offered academic guidance, and critically reviewed the content. I confirm that the contributions stated above accurately reflect the work I completed and the assistance provided by co-authors. All collaborators have reviewed, approved the manuscript to which they contributed, and consent to its inclusion in this thesis. Guiding Research Question. This chapter addresses the secondary research question (SRQ4) Stochastic Dynamic Analysis of Lipschitz Continuity , in service of the primary research question — (RQ2) Training Dynamics . Building on the analytic operator-norm Hessian formulation introduced in the previous chapter, and incorporating high-dimensional stochastic differential equations (SDEs), this chapter establishes — to the best of our knowledge — the first theoretical framework for modeling the temporal evolution of Lipschitz continuity in neural networks during training. The proposed framework is validated through numerical experiments with a convolutional neural network trained on CIFAR-10 and CIFAR-100 with multiple regularization configurations. It also offers theoretical insights into neural network behaviors, including near convergence dynamics, the role of hyper-parameters, and the impact of noisy supervision. Reproducibility. https://anonymous.4open.science/r/lipschitz_dynamics_reproducibility-4721 ABSTRACT Lipschitz continuity characterizes the worst-case sensitivity of neural networks to small input perturbations; yet its dynamics ( i.e. temporal evolution) during training remains under-explored. We present a rigorous mathematical framework to model the temporal evolution of Lipschitz continuity during training with stochastic gradient descent (SGD). This framework leverages a system of stochastic differential equations (SDEs) to capture both deterministic and stochastic forces. Our theoretical analysis identifies three principal factors driving the evolution: (i) the projection of gradient flows, induced by the optimization dynamics, onto the operator-norm Jacobian of parameter matrices; (ii) the projection of gradient noise, arising from the randomness in mini-batch sampling, onto the operator-norm Jacobian; and (iii) the projection of the gradient noise onto the operator-norm Hessian of parameter matrices. Furthermore, our theoretical framework sheds light on how noisy supervision, parameter initialization, batch size, and mini-batch sampling trajectories, among other factors, shape the evolution of the Lipschitz continuity of neural networks. Our experimental results demonstrate strong agreement between the theoretical implications and the observed behaviors. Appendix 4.A Appendix Appendix 4.B Network configuration Function Block # of Layer Module Block #1 1 nn.Conv2d(3, 32, kernel_size=3, padding=1) 2 nn.ReLU() 3 nn.MaxPool2d(kernel_size=2, stride=2) Block #2 4 nn.Conv2d(32, 64, kernel_size=3, padding=1) 5 (*) nn.Dropout(p=0.2) 6 nn.ReLU() 7 nn.MaxPool2d(kernel_size=2, stride=2) Block #3 8 nn.Conv2d(64, 128, kernel_size=3, padding=1, stride=2) 9 (*) nn.Dropout(p=0.2) 10 (*) nn.BatchNorm2d(128, track_running_stats=False) 11 nn.ReLU() Classifier 12 nn.Linear(128 * 4 * 4, 256) 13 nn.ReLU() 14 (*) nn.Dropout(p=0.3) 15 nn.Linear(256, num_classes) Table 4.1 : ConvNet configuration. The layer marked as ‘(*)’ is configurable with respect to experimental needs. There are five parameterized layers ( i.e. # 1, # 4, # 8, # 12 and # 15). # of Layer Module 1 nn.Linear(28*28, 512) 2 nn.ReLU() 3 nn.Linear(512, 256) 4 nn.ReLU() 5 nn.Linear(256, 10) Table 4.2 : MLP configuration. There are three parameterized layers ( i.e. # 1, # 3, and # 5). Appendix 4.C Regularization configuration Regularization Configuration mixup \alpha=0.4 label smoothing \epsilon=0.1 adversarial training w/ FGSM \epsilon=0.03 weight-decay \lambda=0.001 auto-augment AutoAugment(policy=AutoAugmentPolicy.CIFAR10) AutoAugment(policy=AutoAugmentPolicy.CIFAR100) Table 4.3 : Regularization configuration. Appendix 4.D Full validation experiment Results of full validation experiments are shown as in Figure 4.1 . Figure 4.1 : Full numerical validation of our mathematical framework on CIFAR-10. 5 Fourier Analysis of Lipschitz Continuity Guiding Research Question. This chapter explores the relationship between frequency-domain features, robustness, and Lipschitz continuity. It addresses the secondary research question (SRQ5) Fourier Analysis of Lipschitz Continuity , in service of the primary research question — (RQ3) Modulation of Frequency Signal Propagation . Using tools from Fourier analysis, information theory, and Shapley value theory, the chapter first establishes a frequency-domain upper bound on the Lipschitz continuity of neural networks. It then analyzes how input-space frequency perturbations are modulated by this bound. As a direct corollary, it is shown that the variational mutual information induced by frequency perturbations is bounded above by spectral contributions to Lipschitz continuity. As a complement to the theory, empirical studies are also presented to support the connection between frequency-domain perturbations and Lipschitz continuity during training. Notes . Theoretical claims appear in the main text only when supported by numerical experiments, while purely theoretical analyses are compiled in the appendix. This chapter also serves as a conceptual transition to the next chapter, which interprets global robustness of image classifiers using Shapley value theory to quantify the effects of frequency-domain perturbations. Reproducibility. https://anonymous.4open.science/r/lipschitz_fourier_analysis-DCF2/ ABSTRACT Lipschitz continuity plays a central role in understanding the robustness and generalization of neural networks. In this work, we investigate how Lipschitz properties interact with the frequency structure of data. We begin with a theoretical analysis, showing that the global Lipschitz constant can be decomposed as an integral over frequency-specific contributions, revealing how Lipschitz continuity modulates frequency signal propagation. We further demonstrate that perturbing input frequencies leads to bounded changes in mutual information, where the bounds are determined by local spectral Lipschitz coefficients. Complementing this, we provide an empirical analysis that examines the relationship between frequency structures and Lipschitz continuity. Our experiments show that training on high-frequency signals results in larger Lipschitz constants and sharper loss landscapes. These findings bridge the gap between frequency-domain analysis and learning dynamics, offering new tools to interpret model behavior through spectral properties. 6 Interpreting Global Robustness Copyright Disclaimer . This chapter includes content adapted from the following article: 1. Portions of the content are reused verbatim or with minor edits, in accordance with institutional guidelines and the policies of the relevant publishers. Permission from all co-authors has been obtained. Guiding Research Question. This chapter addresses the secondary research question (SRQ6) Interpreting Global Robustness , in service of the primary research question — (RQ3) Modulation of Frequency Signal Propagation . Building on the theoretical analysis and empirical study from the previous chapter, it presents a method for interpreting the global robustness of image models to input perturbations. The results further support and extend the theoretical and empirical insights established in the previous chapter. Reproducibility. https://anonymous.4open.science/r/lipschitz_dynamics_reproducibility-4721 ABSTRACT Perturbation robustness evaluates the vulnerabilities of models, arising from a variety of perturbations, such as data corruptions and adversarial attacks. Understanding the mechanisms of perturbation robustness is critical for global interpretability. We present a model-agnostic, global mechanistic interpretability method to interpret the perturbation robustness of image models. This research is motivated by two key aspects. First, previous global interpretability works, in tandem with robustness benchmarks, e.g. mean corruption error (mCE), are not designed to directly interpret the mechanisms of perturbation robustness within image models. Second, we notice that the spectral signal-to-noise ratios (SNR) of perturbed natural images exponentially decay over the frequency. This power-law-like decay implies that: Low-frequency signals are generally more robust than high-frequency signals — yet high classification accuracy can not be achieved by low-frequency signals alone. By applying Shapley value theory, our method axiomatically quantifies the predictive powers of robust features and non-robust features within an information theory framework. Our method, dubbed as I-ASIDE ( I mage A xiomatic S pectral I mportance D ecomposition E xplanation), provides a unique insight into model robustness mechanisms. We conduct extensive experiments over a variety of vision models pre-trained on ImageNet, including both convolutional neural networks ( e.g. AlexNet , VGG , GoogLeNet/Inception-v1 , Inception-v3 , ResNet , SqueezeNet , RegNet , MnasNet , MobileNet , EfficientNet , etc.) and vision transformers ( e.g. ViT , Swin Transformer , and MaxViT ), to show that I-ASIDE can not only measure the perturbation robustness but also provide interpretations of its mechanisms. Appendix 6.A Appendix Appendix 6.B Fairness division axioms Symmetry axiom : Let \widetilde{\mathcal{I}}\in 2^{\mathcal{I}} be some spectral player coalition. For \forall\penalty\ \mathcal{I}_{i},\mathcal{I}_{j}\in\mathcal{I}\land\mathcal{I}_{i},\mathcal{I}_{j}\notin\widetilde{\mathcal{I}} , the statement v(\widetilde{\mathcal{I}}\cup\{\mathcal{I}_{i}\})=v(\widetilde{\mathcal{I}}\cup\{\mathcal{I}_{j}\}) implies \psi_{i}(\mathcal{I},v)=\psi_{j}(\mathcal{I},v) . This axiom restates the statement ‘ equal treatment of equals ’ principle mathematically. This axiom states that the ‘names’ of players should have no effect on the ‘treatments’ by the characteristic function in coalition games ( Roth 1988 ) . Linearity axiom : Let u and v be two characteristic functions. Let (\mathcal{I},u) and (\mathcal{I},v) be two coalition games. Let (u+v)(\widetilde{\mathcal{I}}):=u(\widetilde{\mathcal{I}})+v(\widetilde{\mathcal{I}}) where \widetilde{\mathcal{I}}\in 2^{\mathcal{I}} . The divisions of the new coalition game (\mathcal{I},u+v) should satisfy: \psi_{i}(\mathcal{I},u+v)=\psi_{i}(\mathcal{I},u)+\psi_{i}(\mathcal{I},v) . This axiom is also known as ‘ additivity axiom’ and guarantees the uniqueness of the solution of dividing payoffs among players ( Roth 1988 ) . Efficiency axiom : This axiom states that the sum of the divisions of all players must be summed to the worth of the player set (the grand coalition): \sum\limits_{i=0}^{M-1}\psi_{i}(\mathcal{I},v)=v(\mathcal{I}) . Dummy player axiom : A dummy player (null player) \mathcal{I}_{*} is the player who has no contribution such that: \psi_{*}(\mathcal{I},v)=0 and v(\widetilde{\mathcal{I}}\cup\{\mathcal{I}_{*}\})\equiv v(\widetilde{\mathcal{I}}) for \forall\penalty\ \mathcal{I}_{*}\notin\widetilde{\mathcal{I}}\land\mathcal{I}_{*}\subseteq\mathcal{I} . Remark 6.B.1 . In the literature ( Roth 1988 ) , the efficiency axiom and the dummy player axiom are also combined and relabeled as carrier axiom. 6.b.1 Spectral signal-to-noise ratio (SNR) Discrete Fourier Transform . The notion ‘frequency’ measures how ‘fast’ the outputs can change with respect to inputs. High frequency implies that small variations in inputs can cause large changes in outputs. In terms of images, the ‘inputs’ are the pixel spatial locations while the ‘outputs’ are the pixel values. Let x:(i,j)\mapsto\mathbb{R} be some 2D image with dimension M\times N which sends every location (i,j) to some real pixel value where (i,j)\in[M]\times[N] . Let \mathscr{F}:\mathbb{R}^{2}\mapsto\mathbb{C}^{2} be some DFT functional operator. The DFT of x is given by: \displaystyle\mathscr{F}(x)(u,v)=\sum_{j=0}^{N-1}\sum_{i=0}^{M-1}x(i,j)e^{-\mathbb{i}2\pi(\frac{u}{M}i+\frac{v}{N}j)}. (6.1) Point-wise energy spectral density (ESD) . The ESD measures the energy quantity at a frequency. To simplify discussions, we use radial frequency , which is defined as the radius r with respect to zero frequency point ( i.e. the frequency center). The energy is defined as the square of the frequency magnitude according to Parseval’s Power Theorem. Let L_{r} be a circle with radius r on the spectrum of image x , as illustrated in Figure . The r is referred to as radial frequency. The point-wise ESD function is given by: \displaystyle ESD_{r}(x):=\frac{1}{|L_{r}|}\cdot\sum\limits_{(u,v)\in L_{r}}|\mathscr{F}(x)(u,v)|^{2} (6.2) where (u,v) is the spatial frequency point and |L_{r}| is the circumference of L_{r} . Spectral signal-to-noise ratio (SNR) . The SNR can quantify signal robustness. We define the spectral SNR at radius frequency r as: \displaystyle SNR(r):=\frac{ESD_{r}(x)}{ESD_{r}(\Delta x)} (6.3) where \Delta x is some perturbation. We have characterized the SNRs of some corroptions and adversarial attacks in Figure . 6.b.2 Absence assignment scheme There exist multiple choices for the assignments of the absences of spectral layers in coalition filtering design: (1) Assigning to constant zeros (Zeroing), (2) assigning to complex Gaussian noise (Complex Gaussian) and (3) assigning to the corresponding frequency components randomly sampled from other images at the same dataset (Replacement). Zeroing . The {\bf\it b} in ( ) is set to zeros. Complex Gaussian . The {\bf\it b} in ( ) is sampled from a i.i.d. complex Gaussian distribution: \mathcal{N}(\mu,\frac{\sigma^{2}}{2})+i\mathcal{N}(\mu,\frac{\sigma^{2}}{2}) . Replacement . The {\bf\it b} in ( ) is set to: {\bf\it b}=\mathscr{F}(x^{*}) (where x^{*}\thicksim\mathcal{X} is a randomly sampled image from some set \mathcal{X} ). In our implementation, we simply choose ‘zeroing’: {\bf\it b}={\bf\it 0} . Figure 6.1 shows the filtered image examples by using the above three strategies and also show the examples of measured spectral importance distributions. Empirically, the three strategies have rather similar performance. In this research, we do not unfold the discussions regarding the masking strategy choices. ((a)) ((b)) ((c)) ((d)) ((e)) ((f)) ((g)) ((h)) ((i)) ((j)) ((k)) ((l)) Figure 6.1 : Three absence assignment strategies: (1) Assigning the spectral absences with constant zeros (Zeroing), (2) assigning the spevtral absences with Gaussian noise (Complex Gaussian) and (3) randomly sampling spectral components from the same image datasets (Replacement). The standard complex Gaussian distribution is given by: \mathcal{N}(0,\frac{1}{2})+i\mathcal{N}(0,\frac{1}{2}) . The figures (a), (b) and (c) show the coalition filtering results with the spectral coalition: \{\mathcal{I}_{0}\} . The figures (d), (e) and (f) show the coalition filtering results with the spectral coalition: \{\mathcal{I}_{1},\mathcal{I}_{2},\mathcal{I}_{3},\mathcal{I}_{4},\mathcal{I}_{5},\mathcal{I}_{6},\mathcal{I}_{7}\} . The figures (g) to (l) show the examples of the measured spectral importance distributions of a resnet18 and a efficientnet_v2_s (both are pre-trained on ImageNet ) with the three assignment strategies. 6.b.3 Proof for Spectral Coalition Information Identity Theorem Proof for Spectral Coalition Information Identity. Suppose the probability measures P(x) , P(x,y) , P(y|x) , and Q(y|x) are absolutely continuous with respect to x on domain \mathcal{X}\bowtie\widetilde{\mathcal{I}} . \displaystyle\mathbb{I}(\mathcal{X}\bowtie\widetilde{\mathcal{I}},\mathcal{Y}) \displaystyle=\int\limits_{\mathcal{X}\bowtie\widetilde{\mathcal{I}}}\sum_{y\in\mathcal{Y}}P(x,y)\cdot\log\frac{P(x,y)}{P(x)\cdot P(y)}dx (6.4) \displaystyle=\int\limits_{\mathcal{X}\bowtie\widetilde{\mathcal{I}}}\sum_{y\in\mathcal{Y}}P(x,y)\cdot\log\left(\frac{P(y|x)\cdot P(x)}{P(y)\cdot P(x)}\cdot\frac{Q(y|x)}{Q(y|x)}\right)dx (6.5) \displaystyle=\int\limits_{\mathcal{X}\bowtie\widetilde{\mathcal{I}}}\sum_{y\in\mathcal{Y}}P(x,y)\cdot\log\left(\frac{P(y|x)}{Q(y|x)}\cdot\frac{1}{P(y)}\cdot Q(y|x)\right)dx (6.6) \displaystyle=\int\limits_{\mathcal{X}\bowtie\widetilde{\mathcal{I}}}P(x)\left(\sum_{y\in\mathcal{Y}}P(y|x)\cdot\log\frac{P(y|x)}{Q(y|x)}\right)dx (6.7) \displaystyle\quad\quad-\sum_{y\in\mathcal{Y}}\left(\int\limits_{\mathcal{X}\bowtie\widetilde{\mathcal{I}}}P(x,y)dx\right)\log P(y) (6.8) \displaystyle\quad\quad+\int\limits_{\mathcal{X}\bowtie\widetilde{\mathcal{I}}}\sum_{y\in\mathcal{Y}}P(x,y)\cdot\log Q(y|x)dx (6.9) \displaystyle=\mathop{\mathbb{}}{E}\limits_{x\in\mathcal{X}\bowtie\widetilde{\mathcal{I}}}\underbrace{KL(P(y|x)||Q(y||x))}_{\mathrm{point-wise}}+H(\mathcal{Y})+\int\limits_{\mathcal{X}\bowtie\widetilde{\mathcal{I}}}P(x)\left(\sum_{y\in\mathcal{Y}}P(y|x)\cdot\log Q(y|x)\right)dx (6.10) \displaystyle=\mathop{\mathbb{}}{E}\limits_{x\in\mathcal{X}\bowtie\widetilde{\mathcal{I}}}\underbrace{KL(P(y|x)||Q(y||x))}_{\mathrm{point-wise}}+H(\mathcal{Y})+\mathop{\mathbb{}}{E}\limits_{x\in\mathcal{X}\bowtie\widetilde{\mathcal{I}}}\sum_{y\in\mathcal{Y}}P(y|x)\cdot\log Q(y|x) (6.11) \displaystyle=\mathop{\mathbb{}}{E}\limits_{x\in\mathcal{X}\bowtie\widetilde{\mathcal{I}}}\underbrace{KL(P(y|x)||Q(y||x))}_{\mathrm{point-wise}}+H(\mathcal{Y})+v(\widetilde{\mathcal{I}})+C (6.12) where H(\mathcal{Y}) is the Shannon entropy of the label set \mathcal{Y} . ∎ Appendix 6.C Information quantity relationship in spectral coalitions Figure 6.2 : Information quantity relationship. This shows the theoretical information quantity relationship between what the characteristic function v measures and the mutual information \mathbb{I}(\mathcal{X}\bowtie\widetilde{\mathcal{I}},\mathcal{Y}) . For a given coalition \widetilde{\mathcal{I}} , a dataset \langle\mathcal{X},\mathcal{Y}\rangle and a classifier Q , the v measures how much information the classifier Q utilizes in decisions. The measured results are then used to compute the marginal contributions of features. 6.c.1 Partitioning spectrum with \ell_{\infty} ball over \ell_{2} ball Figure 6.3 : Two spectral band partitioning schemes. This shows the motivation we choose \ell_{\infty} ball over \ell_{2} ball in partitioning the frequency domain into the M bands (i.e., M ‘spectral players’) over 2D Fourier spectrum. The frequency data density of the spectral players with \ell_{\infty} remains a constant. However, the frequency data density of the spectral players with \ell_{2} is not a constant since some frequency components do not present. This motives us to empirically choose \ell_{\infty} metric to form spectral players in implementation. 6.c.2 Normalizing summarized SIDs We normalize the above result and set: \displaystyle S(v) \displaystyle:=\frac{\absolutevalue{{\vb*\beta}^T \bar\Psi(v) - \frac{||\vb*\beta||_1}{M}}}{\sup\absolutevalue{{\vb*\beta}^T \bar\Psi(v) - \frac{||\vb*\beta||_1}{M}}} (6.13) \displaystyle=\frac{\absolutevalue{{\vb*\beta}^T \bar\Psi(v) - \frac{||\vb*\beta||_1}{M}}}{\sup\absolutevalue{||\vb*\beta||_2 \cdot|| \bar\Psi(v)||_2 - \frac{||\vb*\beta||_1}{M}}} (6.14) \displaystyle=\frac{\absolutevalue{{\vb*\beta}^T \bar\Psi(v) - \frac{||\vb*\beta||_1}{M}}}{\absolutevalue{||\vb*\beta||_2 - \frac{||\vb*\beta||_1}{M}}} (6.15) \displaystyle=\absolutevalue{\frac{{{\vb*{\bar\beta}}}^T \bar\Psi(v) - \frac{1}{M} \frac{||\vb*\beta||_1}{||\vb*\beta||_2}}{1 - \frac{1}{M} \frac{||\vb*\beta||_1}{||\vb*\beta||_2}}}. (6.16) where {{\bf\it\bar{\beta}}}=\frac{{\bf\it\beta}}{||{\bf\it\beta}||_{2}} and \sup\absolutevalue{{\vb*\beta}^T \bar\Psi(v) - \frac{||\vb*\beta||_1}{M}} is derived by: \displaystyle\sup\absolutevalue{{\vb*\beta}^T \bar\Psi(v) - \frac{||\vb*\beta||_1}{M}} \displaystyle=\absolutevalue{\sup{\vb*\beta}^T \bar\Psi(v) - \frac{||\vb*\beta||_1}{M}} (6.17) \displaystyle=\absolutevalue{\sup||{\vb*\beta}||_2 \cdot||\bar\Psi(v)||_2 - \frac{||\vb*\beta||_1}{M}}\penalty\ \penalty\ \penalty\ \penalty\ \mathrm{s.t.}\penalty\ ||\bar{\Psi}(v)||_{1}=1 (6.18) \displaystyle=\absolutevalue{||{\vb*\beta}||_2 - \frac{||\vb*\beta||_1}{M}}\penalty\ \penalty\ \penalty\ \penalty\ \mathrm{since}\penalty\ ||\bar{\Psi}(v)||_{2}^{2}\leq||\bar{\Psi}(v)||_{1}^{2}. (6.19) Set \eta=\frac{1}{M}\frac{||{\bf\it\beta}||_{1}}{||{\bf\it\beta}||_{2}} : \displaystyle S(v)=\absolutevalue{\frac{{{\vb*{\bar\beta}}}^T \bar\Psi(v) - \eta}{1 - \eta}}. (6.20) ∎ 6.c.3 How many samples are sufficient? ((a)) ((b)) ((c)) Figure 6.4 : Convergence of relative estimation errors converge with respect to the numbers of samples K . The errors are measured by: \frac{1}{M}||\Psi^{(i+1)}(v)-\Psi^{(i)}(v)||_{1} where \Psi^{(i)}(v) denotes the i -th measured spectral importance distribution with respect to characteristic function v . The experiments are conducted on CIFAR10 , CIFAR100 and ImageNet with resnet18 . Error bound analysis . Let K be the number of the samples of some baseline dataset. Let: \displaystyle\Delta v(\tilde{\mathcal{I}},\mathcal{I}_{i}):=v(\tilde{\mathcal{I}}\cup\{\mathcal{I}_{i}\})-v(\tilde{\mathcal{I}}) (6.21) and \displaystyle\Delta v(\mathcal{I}_{i}):=\left(\Delta v(\tilde{\mathcal{I}},\mathcal{I}_{i})\right)_{\tilde{\mathcal{I}}\subseteq\mathcal{I}} (6.22) and \displaystyle W:=\left(\frac{1}{M}\binom{M-1}{|\tilde{\mathcal{I}}|}^{-1}\right)_{\tilde{\mathcal{I}}\subseteq\mathcal{I}}. (6.23) Hence: \displaystyle\psi_{i}(\mathcal{I},v)=W^{T}\Delta v(\mathcal{I}_{i}) (6.24) where ||W||_{1}\equiv 1 since W is a probability distribution. Let \bar{\psi}_{i} , \Delta\bar{v}(\mathcal{I}_{i}) and \Delta\bar{v}(\tilde{\mathcal{I}},\mathcal{I}_{i}) be estimations with K samples using Monte Carlo sampling. The error bound with \ell_{1} norm is given by: \displaystyle\epsilon \displaystyle\stackrel{{\scriptstyle\text{def}}}{{=}}\sup_{i}||\bar{\psi}_{i}(\mathcal{I},v)-\psi_{i}(\mathcal{I},v)||_{1}=\sup_{i}||W^{T}\Delta\bar{v}(\mathcal{I}_{i})-W^{T}\Delta v(\mathcal{I}_{i})||_{1} (6.25) \displaystyle\leq\sup_{i}||W||_{1}\cdot||\Delta\bar{v}(\mathcal{I}_{i})-\Delta v(\mathcal{I}_{i})||_{\infty}\quad\quad\quad\left(\mathrm{H\"{o}lder^{\prime}s\penalty\ inequality}\right) (6.26) \displaystyle=\sup_{i}||\sum\limits_{\tilde{\mathcal{I}}\subseteq\mathcal{I}\setminus\mathcal{I}_{i}}\left(\Delta\bar{v}(\tilde{\mathcal{I}},\mathcal{I}_{i})-\Delta v(\tilde{\mathcal{I}},\mathcal{I}_{i})\right)||_{\infty} (6.27) \displaystyle\leq\sup_{i}2^{M-1}\cdot\sup_{\tilde{\mathcal{I}}}||\Delta\bar{v}(\tilde{\mathcal{I}},\mathcal{I}_{i})-\Delta v(\tilde{\mathcal{I}},\mathcal{I}_{i})||_{\infty} (6.28) \displaystyle=\sup_{i}2^{M-1}\cdot\sup_{\tilde{\mathcal{I}}}||\Delta\bar{v}(\tilde{\mathcal{I}},\mathcal{I}_{i})-\Delta v(\tilde{\mathcal{I}},\mathcal{I}_{i})||_{1} (6.29) \displaystyle\leq 2^{M-1}\cdot\left\{\frac{Var(\Delta\bar{v})}{K}\right\}^{\frac{1}{2}} (6.30) where Var(\Delta\bar{v}) gives the upper bound of the variance of \Delta\bar{v}(\tilde{\mathcal{I}},\mathcal{I}_{i}) . 7 Future work 8 Conclusions 205 205 206 206 204 204 203 203 202 202 205 205 206 206 204 204 203 203