1-Lipschitz Network Initialization for Certifiably Robust Classification Applications: A Decay Problem

Marius F. R. Juston, Ramavarapu S. Sreenivas, William R. Norris, Ahmet Soylemezoglu, Dustin Nottage · 2025

Read 1-Lipschitz Network Initialization for Certifiably Robust Classification Applications: A Decay Problem by Marius F. R. Juston, Ramavarapu S. Sreenivas, William R. Norris, Ahmet Soylemezoglu, Dustin Nottage (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-00240.txt.

Introduction

Juston and four co-authors examined a hidden flaw in tamper-proof neural networks in twenty twenty-five. These one-Lipschitz networks limit how much an input tweak can change the output, enabling certified robustness against adversarial attacks. Two standard layer designs dominate the field.

The authors analyzed how the layers' weight settings behave at initialization, when weights are drawn randomly. They derived exact and upper bounds for weight variance under normal and generalized normal starting distributions. Surprisingly, weight variance does not affect output variance at all; only the matrix dimensions do. Worse, they prove the initialization always makes deep networks of this kind decay toward zero output. The paper flags a fundamental startup problem that designers of certified defenses must engineer around.

Catalog record

CREATORMarius F. R. Juston, Ramavarapu S. Sreenivas, William R. Norris, Ahmet Soylemezoglu, Dustin Nottage
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE66 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-00240

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

1-Lipschitz Network Initialization for Certifiably Robust Classification Applications: A Decay Problem Marius F. R. Juston, Ramavarapu S. Sreenivas, William R. Norris, Ahmet Soylemezoglu, Dustin Nottage Abstract This paper discusses the weight parametrization of two standard 1-Lipschitz network architectures, the Almost-Orthogonal-Layers (AOL) and the SDP-based Lipschitz Layers (SLL). It examines their impact on initialization for deep 1-Lipschitz feedforward networks, and discusses underlying issues surrounding this initialization. These networks are mainly used in certifiably robust classification applications to combat adversarial attacks by limiting the impact of perturbations on the classification output. Exact and upper bounds for the parameterized weight variance were calculated assuming a standard Normal distribution initialization; additionally, an upper bound was computed assuming a Generalized Normal Distribution, generalizing the proof for Uniform, Laplace, and Normal distribution weight initializations. It is demonstrated that the weight variance holds no bearing on the output variance distribution and that only the dimension of the weight matrices matters. Additionally, this paper demonstrates that the weight initialization always causes deep 1-Lipschitz networks to decay to zero. [type=editor, auid=000,bioid=1, role=Researcher, orcid=0009-0005-7216-4180] [type=editor, auid=000,bioid=1, orcid=0000-0002-8242-0839] [orcid=0000-0002-4940-4458 ] [ orcid=0000-0003-4950-099X ] keywords 1-Lipschitz Network ,Kaiming initialization ,Almost-Orthogonal-Layers ,Generalized Normal Distribution † † credit: Writing – review & editing, Writing – original draft, Methodology, Formal analysis, Conceptualization, Software † † credit: Writing – review & editing † † credit: Writing - Review † † credit: Writing - Review † † credit: Writing - Review † † corresponding: Corresponding author † † This author has passed away. † † note: The code is available for replication of the results at https://github.com/Marius-Juston/SLLLipschitzInitialization 1 Introduction The robustness of deep neural networks, primarily against adversarial attacks, has been a significant challenge in the field of modern applications of machine learning Nguyen et al. (2015) ; Szegedy et al. (2013) ; Biggio et al. (2013) by manipulating the input so that the model produces incorrect output. The problem of network robustness in deep networks stems mainly from the fact that large network weight magnitudes have an exponential impact on the output, the deeper it goes. The significant weight magnitudes thus enable a small perturbation to the input to cause a drastic change in the classification output Costa et al. (2024) . The design of the 1-Lipschitz neural network has provided a reliable solution to certifying the network to be robust, such that the decision output remains the same within a sphere of perturbation Tsuzuku et al. (2018) . For the design, multiple approaches have been proposed, ranging from utilizing Spectral Normalization (SN) Miyato et al. (2018) ; Roth et al. (2020) , Orthogonal Parametrization Trockman and Kolter (2021) , Convex Potential Layers (CPL) Meunier et al. (2022) , Almost-Orthogonal-Layers (AOL) Prach and Lampert (2022) , Sandwich layers Wang and Manchester (2023) and the recent SDP-based Lipschitz Layers (SLL) Araujo et al. (2023) . All these previous techniques assume standard Kaiming initialization scheme for its parameterized layered weights, which might or might not be appropriate for these networks. This remains under-studied for these networks; paper aims to further the theoretical initialization dynamics understanding of AOL and SLL networks. This paper explores the impact of the weight parameterization of 1-Lipschitz networks employing Almost-Orthogonal-Layers and SDP-based Lipschitz Layers on the initialization of deep neural networks. Exploring the challenges in applying certifiably robust neural networks, such as 1-Lipschitz neural networks, is crucial, especially as neural network attacks become more frequent and robust classification results become more important. As such, discussing ways to improve training for deeper neural network architectures is important to understand and address. This article aims to illuminate some of the issues underlying these weight-normalizing networks. • An extended derivation for the network layer variance while accounting for the bias term and its recursive definition using the ReLU activation function is provided. • Given the structure for the Almost-Orthogonal-Layers and SDP-based Lipschitz Layers feedforward network structure and weight parameterization, an upper bound and exact network weight variance is derived assuming a normal distribution initialization • A general upper bound based on the Generalized Normal Distribution for the parameterized network weight variance is derived. • Based on the calculated weight variance, insights for the 1-Lipschitz network are discussed as to potential issues in this network’s initialization. The initial work for the initialization analysis is inspired by the works of Kaiming He et al. (2015) and Xavier Glorot and Bengio (2010) , while the 1-Lipschitz network structure is derived from Araujo et al. (2023) . Understanding the initialization characteristics of these networks is extremely important as it allows researchers to better understand how to generate deeper Lipschitz networks, which enable robust certification guarantees, thus the bounds derived in the paper can help with researchers to develop more novel initialization or parameterization methods to help mitigate the issues that arise from the current state of the art methods. 2 Related Work The starting work from Xavier Glorot and Bengio (2010) was a pivotal moment for deep neural networks with the methodology to properly initialize deep neural networks such that they would converge, assuming hyperbolic tangent activation functions; however, their work posed simplifying assumptions on the activation functions which caused issues when transition to more modern activation functions such as the commonly used ReLU Nair and Hinton (2010) . The works by Kaiming expanded on this concept and developed a method to generate weight initializations for deep networks using the Parameterized ReLU family He et al. (2015) ; this work demonstrated the ability to ensure that the network would converge and train properly, regardless of depth. Since then, all modern machine learning has used Kaiming initialization for its networks, and modifications to the initialization gain have been activation-specific to ensure the stability criteria derived by Kaiming remain satisfied, as in the work on SELUs Klambauer et al. (2017) . An issue with the works above is the assumption of a bias term initialized to zero. From these works the connection to the correct initialization choice to enabling gradient stability establishes a direct link to impact the training accuracy and thus the robustness of networks. In conjunction with the works for network initialization, Araujo et al. (2023) developed a unifying methodology to combine multiple existing 1-Lipschitz network structures into a unifying framework. This framework provides a guideline for creating a new, certifiably robust neural network. The authors achieve this by formulating feedforward networks as a nonlinear robust control Lur’e system Lur’e and Postnikov (1944) and enforcing conditions on the generalized residual network’s weights via SDP constraints. From this work, they can demonstrate general conditions for enforcing a multilayered 1-Lipschitz network and combine previous works from Spectral Normalization (SN) Miyato et al. (2018) ; Roth et al. (2020) , Orthogonal Parameterization Trockman and Kolter (2021) , Convex Potential Layers (CPL) Meunier et al. (2022) , and Almost-Orthogonal-Layers (AOL) Prach and Lampert (2022) into a single constraint. From the framework, they generate an augmented version of the AOL with additional parameterization, called SDP-based Lipschitz Layers, which improves the network’s generalizability. However, previous work on robust networks also uses standard Kaiming initialization for network weights. In addition, because 1-Lipschitz activation functions are required, the ReLU activation function is commonly used and will also be used in the proofs below. This article explores the impact of

Read the complete text