RCR-AF: Enhancing Model Generalization via Rademacher Complexity Reduction Activation Function
Yunrui Yu, Kafeng Wang, Hang Su · 2025
Read RCR-AF: Enhancing Model Generalization via Rademacher Complexity Reduction Activation Function by Yunrui Yu, Kafeng Wang, Hang Su (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-22446.txt.
Introduction
Yunrui Yu and colleagues propose a new activation function for neural networks in this two thousand twenty-five paper. Activation functions shape how each unit in a network responds. They affect both accuracy on new data and resistance to attacks. Crafted inputs can fool networks in ways that matter for safety. The authors aim to build a function that improves both traits at once.
Their design, RCR-AF, blends traits of two known functions. It keeps smooth gradients and some negative signal, like GELU, and steady growth, like ReLU. Built-in clipping, set by two dials, controls sparsity and model size. Theory based on Rademacher complexity shows how these dials govern generalization. The paper offers network builders a new knob for robust design.
Catalog record
| CREATOR | Yunrui Yu, Kafeng Wang, Hang Su |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-22446
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
RCR-AF: Enhancing Model Generalization via Rademacher Complexity Reduction Activation Function Yunrui Yu, Kafeng Wang, Hang Su Abstract Despite their widespread success, deep neural networks remain critically vulnerable to adversarial attacks, posing significant risks in safety-sensitive applications. This paper investigates activation functions as a crucial yet underexplored component for enhancing model robustness. We propose a Rademacher Complexity Reduction Activation Function ( RCR-AF ), a novel activation function designed to improve both generalization and adversarial resilience. RCR-AF uniquely combines the advantages of GELU (including smoothness, gradient stability, and negative information retention) with ReLU’s desirable monotonicity, while simultaneously controlling both model sparsity and capacity through built-in clipping mechanisms governed by two hyperparameters, α \alpha and γ \gamma . Our theoretical analysis, grounded in Rademacher complexity, demonstrates that these parameters directly modulate the model’s Rademacher complexity, offering a principled approach to enhance robustness. Comprehensive empirical evaluations show that RCR-AF consistently outperforms widely-used alternatives (ReLU, GELU, and Swish) in both clean accuracy under standard training and in adversarial robustness within adversarial training paradigms. 1 Introduction Deep learning has revolutionized artificial intelligence, achieving groundbreaking results across computer vision [ 1 , 2 ] , natural language processing [ 3 ] , and speech recognition [ 4 ] . However, the discovery of adversarial vulnerabilities [ 5 ] has revealed a critical weakness: carefully crafted imperceptible perturbations can deceive even state-of-the-art models. This security flaw poses substantial risks as deep learning systems become increasingly deployed in safety-critical applications. The emergence of large language models like ChatGPT [ 6 ] further amplifies these concerns, as their widespread adoption could magnify the societal impact of robustness failures. Significant efforts have been made to develop defense mechanisms [ 7 , 8 , 9 , 10 , 11 , 12 , 13 ] against adversarial attacks, alongside increasingly sophisticated evaluation methodologies [ 10 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 ] . However, the arms race between evolving attack strategies and defense mechanisms underscores the critical need for theoretically grounded approaches to robustness enhancement. Defense strategies fall into three main categories: (1) adversarial training-based approaches [ 10 , 16 ] , which optimize models using clean and adversarial examples; (2) input transformation methods [ 12 , 11 ] , which purify inputs to remove perturbations; and (3) model enhancement techniques [ 20 , 24 ] , which focus on architectural improvements and feature space regularization to inherently strengthen model robustness. Among these, adversarial training stands out as the most empirically validated, enhancing robustness by exposing models to adversarial perturbations during training [ 10 ] . Recent studies [ 25 ] highlight the crucial role of activation functions in adversarial robustness. GELU consistently outperforms ReLU in adversarial training due to its continuous differentiability and retention of negative inputs, enabling stable gradient flow and improved robustness. However, its non-monotonic behavior reduces interpretability. In contrast, ReLU’s hard zero-clipping promotes sparsity but discards negative information, potentially limiting robustness. We propose the RCR-AF activation function, which synergistically combines GELU’s desirable properties (smoothness , gradient stability, and negative information retention) with ReLU’s monotonicity. Notably, RCR-AF incorporates built-in clipping with two hyperparameters ( \alpha , \gamma ) that simultaneously control feature sparsity and regulate model capacity via Rademacher complexity. This dual mechanism enables both practical output bounding and theoretically-grounded complexity control. To our knowledge, RCR-AF represents the first activation function that systematically enhances adversarial robustness through explicit complexity regularization. Experiments demonstrate RCR-AF ’s advantages over standard activation functions (GELU, ReLU, and Swish): under standard training, it achieves higher clean accuracy, while in adversarial training settings, it yields improved robustness against AutoAttack. These consistent improvements across different training paradigms suggest that RCR-AF can simultaneously enhance both generalization and adversarial robustness. 1. The RCR-AF activation function synergistically combines GELU’s desirable properties (smoothness, gradient stability, and negative information retention) with ReLU’s monotonicity. 2. RCR-AF incorporates built-in clipping with two hyperparameters ( \alpha , \gamma ) that simultaneously control feature sparsity and regulate model capacity via Rademacher complexity. 3. Theoretical framework proving RCR-AF ’s hyperparameters directly modulate model’s Rademacher complexity. 4. Comprehensive experiments showing RCR-AF outperforms baseline activations in both standard and adversarial training scenarios. 2 Preliminaries & Related Work 2.1 Taxonomy of Defense Strategies Modern adversarial defense strategies fall into three main categories: (1) Adversarial training-based approaches, the most empirically validated paradigm, which builds on PGD-AT [ 26 ] with subsequent improvements like TRADES’ theoretical loss decomposition [ 27 ] , MART’s margin maximization [ 28 ] , and enhanced regularization [ 29 ] ; (2) Input transformation methods that modify inputs pre-processing, including GAN-based purification [ 12 ] and stochastic randomization [ 30 ] , offering computational efficiency but weaker robustness guarantees; and (3) Model enhancement techniques like specialized activations [ 31 ] , robust architectures [ 32 ] , and feature regularization [ 24 ] , where our work contributes through novel activation function design while maintaining adversarial training compatibility. 2.2 Adversarial Training Framework Among these, adversarial training stands out as the most empirically validated, enhancing robustness by exposing models to adversarial perturbations during training. Given a classifier f_{\bm{\theta}}:\mathbb{R}^{C\times H\times W}\rightarrow\mathbb{R}^{K} with parameters \bm{\theta} , the adversarial training objective can be formulated as a min-max optimization problem: \min_{\bm{\theta}}\mathbb{E}_{(\mathbf{x},y)\sim\mathcal{D}}\left[\max_{\|\mathbf{\delta}\|_{p}\leq\epsilon}L(f_{\bm{\theta}}(\mathbf{x}+\mathbf{\delta}),y)\right] (1) where \mathcal{D} represents the data distribution and L denotes the classification loss. The inner maximization generates adversarial perturbations within an \ell_{p} -norm ball of radius \epsilon , typically implemented through Projected Gradient Descent (PGD) [ 26 ] : \mathbf{\hat{x}}_{i+1}=\text{Proj}_{\mathcal{B}_{\epsilon}(\mathbf{x})}\left(\mathbf{\hat{x}}_{i}+\alpha_{i}\cdot\text{sign}\left(\nabla_{\mathbf{\hat{x}}_{i}}L(f_{\bm{\theta}}(\mathbf{\hat{x}}_{i}),y)\right)\right), (2) where \mathbf{\hat{x}}_{0}=\text{Proj}_{\mathcal{B}_{\epsilon}(\mathbf{x})}(\mathbf{x}+\mathbf{u}) is the initial adversarial example, with \mathbf{u}\sim\text{Uniform}[-\epsilon,\epsilon] being a random perturbation. \alpha_{i} is the step size at iteration i . \text{Proj}_{\mathcal{B}_{\epsilon}(\mathbf{x})} denotes the projection operator that ensures the adversarial example remains within the \epsilon -ball centered at \mathbf{x} under the \ell_{p} -norm (typically \ell_{\infty} or \ell_{2} ). \nabla_{\mathbf{\hat{x}}_{i}}L is the gradient of the loss function L with respect to the adversarial example \mathbf{\hat{x}}_{i} . 2.3 Activation Functions in Robust Learning Activation functions play a pivotal role in determining neural network behavior during adversarial training. Modern activation functions can be categorized into two distinct classes based on their gradient properties: (1) piecewise-linear functions with gradient discontinuities (e.g., ReLU), and (2) smooth functions with continuous gradients (e.g., GELU and Swish). The ReLU activation ( \sigma(x)=\max(0,x) ) [ 2 ] represents the first category, introducing sparsity and computational efficiency through its simple thresholding operation. However, its non-differentiability at zero and complete suppression of negative activations can lead to training instability and dead neurons. These characteristics become particularly problematic in adversarial training scenarios, where gradient-based attacks exploit the discontinuous decision boundaries created by ReLU’s hard thresholding. In contrast, both GELU ( \sigma(x)=x\Phi(x) where \Phi is the standard normal CDF) [ 31 ] and Swish ( \sigma(x)=x/(1+e^{-\beta x}) with \beta=1 ) [ 33 ] belong to the second category, providing continuous gradients across their entire domains. This shared smoothness property enables more stable gradient flow during both standard and adversarial training, while their retention of negative activations prevents complete neuron deactivation. Recent work by [ 25 ] demonstrates that these properties collectively contribute to improved adversarial robustness compared to ReLU. 3 Methods We propose a novel activation function, termed \text{${{\text{RCR-AF}}}$}(x;\alpha;\gamma) , defined as: \text{${{\text{RCR-AF}}}$}(x;\alpha;\gamma)=\frac{1}{\alpha}\ln\left(1+e^{-\alpha x}\right)+x\quad with\quad\text{clip}(x,[-\gamma/\alpha,\gamma/\alpha]) (3) where \alpha and