LipNeXt: Scaling up Lipschitz-based Certified Robustness to Billion-parameter Models

Kai Hu, Haoqi Hu, Matt Fredrikson · 2026

Read LipNeXt: Scaling up Lipschitz-based Certified Robustness to Billion-parameter Models by Kai Hu, Haoqi Hu, Matt Fredrikson (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-18513.txt.

Introduction

Kai Hu, Haoqi Hu, and Matt Fredrikson build a new net that can prove its own strength in the face of tricks. Some nets come with a bound that shows no small shift in a shot can flip their call. Past nets with that proof were small, slow to train, or weak on big shot sets. Their net drops two common parts to break that trade.

The net has no folds of the classic kind and no hard caps that slow its growth. It trains by steps that keep its weights on a curved space where the bound holds by form. A new part shifts the shot to catch form with no folds at all. The team bill it as the first net of its kind to reach huge scale. Such nets could serve fields where a proof of strength is a must, not a plus.

Catalog record

CREATORKai Hu, Haoqi Hu, Matt Fredrikson
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-18513

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LipNeXt: Scaling up Lipschitz-based Certified Robustness to Billion-parameter Models Kai Hu, Haoqi Hu, Matt Fredrikson Abstract Lipschitz-based certification offers efficient, deterministic robustness guarantees but has struggled to scale in model size, training efficiency, and ImageNet performance. We introduce LipNeXt , the first constraint-free and convolution-free 1-Lipschitz architecture for certified robustness. LipNeXt is built using two techniques: (1) a manifold optimization procedure that updates parameters directly on the orthogonal manifold and (2) a Spatial Shift Module to model spatial pattern without convolutions. The full network uses orthogonal projections, spatial shifts, a simple 1-Lipschitz β \beta -Abs nonlinearity, and L 2 L_{2} spatial pooling to maintain tight Lipschitz control while enabling expressive feature mixing. Across CIFAR-10/100 and Tiny-ImageNet, LipNeXt achieves state-of-the-art clean and certified robust accuracy (CRA), and on ImageNet it scales to 1–2B large models, improving CRA over prior Lipschitz models (e.g., up to + 8 % +8\% at ε = 1 \varepsilon{=}1 ) while retaining efficient, stable low-precision training. These results demonstrate that Lipschitz-based certification can benefit from modern scaling trends without sacrificing determinism or efficiency. 1 Introduction Adversarial robustness represents a fundamental challenge in machine learning ( 36 ) . Numerous defense mechanisms have been developed to enhance model robustness against adversarial attacks ( 10 ; 19 ; 27 ; 23 ; 26 ) . However, these approaches are predominantly empirical defenses that cannot provide formal guarantees of robust predictions. Consequently, models deemed robust under current evaluation protocols may remain vulnerable to more sophisticated attack strategies as they emerge. This limitation is particularly concerning for safety-critical applications such as autonomous driving ( 15 ) , medical image processing ( 21 ) , and malware classification ( 30 ) , where failures can have severe consequences. To address this challenge, certified robustness has emerged as a promising research direction. Certifying the robustness of a test case requires a mathematical guarantee that the model’s outputs remain unchanged within a predefined \ell_{p} -norm ball of radius \varepsilon around the input. The performance of these certification methods is typically measured by the certified robust accuracy (CRA), which quantifies the proportion of correctly predicted inputs that are also provably robust within a specified radius. Research on robustness certification largely follows two methodological strands. The first, randomized smoothing (RS) ( 7 ; 45 ; 17 ; 5 ) , provides probabilistic guarantees by averaging a classifier’s predictions under additive noise. The second exploits the Lipschitz properties of neural networks to yield deterministic (worst-case) certificates ( 14 ; 3 ; 12 ) . In this work, we focus on advancing the latter direction; a detailed comparison between the two appears in Appendix A . Despite their theoretical appeal, Lipschitz-based certification methods have struggled to scale in practice. A central critique is the weak performance on large-scale benchmarks: models often underfit even small-sized datasets such as CIFAR-100 and degrade markedly on ImageNet ( 13 ) . Most systems still rely on small , VGG-style ( 34 ) architectures with \leq 32M parameters. Although recent work has explored deeper/larger architectures for certified robustness ( 13 ; 3 ) , the gains plateau quickly as model size increases. Examining existing work, we find that orthogonal matrices are fundamental to building 1-Lipschitz networks because they enable tight Lipschitz bounds. However, they are also a major bottleneck that prevents Lipschitz-based certification from scaling. Existing methods either explicitly re-parameterize orthogonal matrices or implicitly re-parameterize Lipschitz-bounded operations to learn near-orthogonal matrices; both introduce substantial computational overhead (see Section 2.1 ). To address this issue, we propose directly optimizing orthogonal matrices on the orthogonal manifold. Although orthogonal manifold optimization is a mature technique, to our knowledge it has not been exploited for certification. We further observe that in the large-model regime, where learning rates are small, the matrix exponential can be accurately and efficiently approximated. Combining these ideas, we reduce the additional per-update overhead to at most five matrix multiplications. A natural path to scalability is transformers ( 39 ) , which scale to billion-parameter models and exhibit emergent abilities across a wide range of tasks ( 41 ) . However, attention lacks straightforward Lipschitz control. Fortunately, ConvNeXt ( 24 ) and MetaFormer ( 47 ) suggest that Lipschitz-bounded architectures can benefit from transformer-era design choices. Motivated by the simple token-mixing mechanisms in these modern architectures ( 47 ) , we design a convolution-free shifting module paired with positional encoding that uses simple shift operations to model spatial relations as the building block of our model. Compared to prior work using heavy FFT-based convolution designs ( 38 ; 20 ) or power-iteration-based Lipschitz regularization ( 22 ; 13 ) , the shifting module adds minimal computational cost while preserving the capacity to model spatial patterns. Combining the aforementioned techniques, we propose LipNeXt , the first convolution-free and constraint-free architecture for certified robustness. Benefiting from this design, we scale to billion-parameter models and observe non-saturating gains with increasing model size. Our experiments show that LipNeXt outperforms state-of-the-art methods in both clean accuracy and certified robustness across CIFAR-10, CIFAR-100, Tiny-ImageNet, and ImageNet, highlighting the effectiveness of our approach for scalable, provably robust deep learning. Contributions. (i) A constraint-free manifold optimization scheme for learning orthogonal parameters at scale with efficient exponential approximations and stabilization; (ii) a theoretically motivated, convolution-free Spatial Shift Module that preserves Lipschitz control while enabling spatial mixing; and (iii) state-of-the-art CRA and clean accuracy on standard benchmarks, with successful scaling to billion-parameter models. 2 Preliminary and Related Work 2.1 Preliminaries Lipschitz-based Method for Certified Robustness Consider a neural network f:\mathbb{R}^{d}\rightarrow\mathbb{R} and its corresponding binary classifier F(x)=\text{sign}(f(x)) . We say that classifier F is \varepsilon -locally robust at point x if for all perturbations x^{\prime} satisfying \|x-x^{\prime}\|_{p}\leq\varepsilon , we have F(x)=F(x^{\prime}) . Certified robustness represents a stronger condition where the robustness property cannot only be empirically verified but also mathematically guaranteed. In this work, we only focus on p=2 , i.e., certified robustness under \ell_{2} norm. Given an upper bound K on the Lipschitz constant of f , we can certify that F is locally robust at x with a guaranteed robustness radius of |f(x)|/K . This certification follows from the Lipschitz property, which bounds the maximum change in f(x) within the \varepsilon -ball around x . For multi-class classification with N classes, we decompose the problem into N-1 binary classification tasks using a one-vs-rest approach. The certified robustness radius of the N -class classifier is then defined as the minimum certified radius across all constituent binary classifiers, thereby ensuring robustness for the prediction. To compute Lipschitz bounds for deep networks, we leverage the compositional property that the Lipschitz constant of a composite function satisfies \text{Lip}(f\circ g)\leq\text{Lip}(f)\text{Lip}(g) , where (f\circ g)(x)=f(g(x)) . Consequently, for a feed-forward neural network, an upper bound on the overall Lipschitz constant can be efficiently computed as the product of the Lipschitz constants of all constituent layers. However, this bound can be very loose, leading to an extremely small certified radius. Therefore, designing a network architecture for which a tight Lipschitz bound can be computed is crucial to achieving satisfactory certified performance. Orthogonal Manifold Optimization The orthogonal manifold is defined as the set of all orthogonal matrices: \mathcal{M}_{d}=\{\mathbf{X}\in\mathbb{R}^{d\times d}\mid\mathbf{X}^{\top}\mathbf{X}=\mathbf{I}_{d}\} , where \mathbf{I}_{n} denotes the d\times d identity matrix. Optimization problems on

Read the complete text