λ \lambda -GELU: Learning Gating Hardness for Controlled ReLU-ization in Deep Networks

Cristian Pérez-Corral* Universitat Politècnica de València Valencia, Spain [email protected] Alberto Fernández-Hernández Universitat Politècnica de València Valencia, Spain [email protected] Jose I. Mestre Universitat Jaume I Castelló de la Plana, Spain [email protected] Manuel F. Dolz Universitat Jaume I Castelló de la Plana, Spain [email protected] Enrique S. Quintana-Ortí Universitat Politècnica de València Valencia, Spain [email protected] · 2026

Read λ \lambda -GELU: Learning Gating Hardness for Controlled ReLU-ization in Deep Networks by Cristian Pérez-Corral* Universitat Politècnica de València Valencia, Spain [email protected] Alberto Fernández-Hernández Universitat Politècnica de València Valencia, Spain [email protected] Jose I. Mestre Universitat Jaume I Castelló de la Plana, Spain [email protected] Manuel F. Dolz Universitat Jaume I Castelló de la Plana, Spain [email protected] Enrique S. Quintana-Ortí Universitat Politècnica de València Valencia, Spain [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-21991.txt.

Introduction

Pérez-Corral and co-authors add a learnable sharpness knob to GELU, easing the path from smooth networks to ReLU ones. Many deployment and compression tools expect ReLU-style piecewise linear models. Their lambda-GELU scales the input inside the Gaussian curve, and a larger lambda makes the gate act more like ReLU. Learning lambda proved unstable with naive updates, so they add a constrained form and an optimizer-aware rule. Across MLPs, CNNs, and Transformers, layerwise hardness profiles emerged. A hardening step then swaps in ReLUs.

Catalog record

CREATORCristian Pérez-Corral* Universitat Politècnica de València Valencia, Spain [email protected] Alberto Fernández-Hernández Universitat Politècnica de València Valencia, Spain [email protected] Jose I. Mestre Universitat Jaume I Castelló de la Plana, Spain [email protected] Manuel F. Dolz Universitat Jaume I Castelló de la Plana, Spain [email protected] Enrique S. Quintana-Ortí Universitat Politècnica de València Valencia, Spain [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40896
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.21991

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

λ \lambda -GELU: Learning Gating Hardness for Controlled ReLU-ization in Deep Networks Cristian Pérez-Corral* Universitat Politècnica de València Valencia, Spain [email protected] Alberto Fernández-Hernández Universitat Politècnica de València Valencia, Spain [email protected] Jose I. Mestre Universitat Jaume I Castelló de la Plana, Spain [email protected] Manuel F. Dolz Universitat Jaume I Castelló de la Plana, Spain [email protected] Enrique S. Quintana-Ortí Universitat Politècnica de València Valencia, Spain [email protected] Abstract gelu ( gelu ) is a widely used smooth alternative to relu ( relu ), yet many deployment, compression, and analysis toolchains are most naturally expressed for piecewise-linear ( relu -type) networks. We study a hardness-parameterized formulation of gelu , f ⁡ ( x , λ ) = x ​ Φ ​ ( λ ​ x ) f(x;\lambda)=x\,\Phi(\lambda x) , where Φ \Phi is the Gaussian cdf ( cdf ) and λ ∈ [ 1 , ∞ ) \lambda\in[1,\infty) controls gate sharpness, with the goal of turning smooth gated training into a controlled path toward relu -compatible models. Learning λ \lambda is non-trivial: naive updates yield unstable dynamics and effective gradient attenuation, so we introduce a constrained reparameterization and an optimizer-aware update scheme. Empirically, across a diverse set of model–dataset pairs spanning MLPs, CNNs, and Transformers, we observe structured layerwise hardness profiles and assess their robustness under different initializations. We further study a deterministic relu -ization strategy in which the learned gates are progressively hardened toward a principled target, enabling a post-training substitution of λ \lambda - gelu by relu with reduced disruption. Overall, λ \lambda - gelu provides a minimal and interpretable knob to profile and control gating hardness, bridging smooth training with relu -centric downstream pipelines. AI Artificial Intelligence AUC Area Under the Curve CDF Cumulative Distribution Function CNN Convolutional Neural Network DL Deep Learning DNN Deep Neural Network GELU Gaussian Error Linear Unit GLAI GreenLightningAI KD Knowledge Distillation ML Machine Learning MLP Multilayer Perceptron MVA Maximum Validation Accuracy MHA Multihead Attention MoE Mixture-of-Experts Mish Mish NLP Natural Language Processing NN Neural Network OUI Overfitting-Underfitting Indicator PDF Probability Density Function PReLU Parametric Rectified Linear Unit PoC Proof-of-Concept ReLU Rectified Linear Unit SiLU Sigmoid Linear Unit SGD Stochastic Gradient Descent SVD Singular Value Decomposition Swish Swish SReLU S-shaped ReLU VT Vision Transformer ViT Vision Transformer WD Weight Decay I Introduction Activation functions shape both the optimization dynamics and the functional form of deep neural networks. Modern practice is largely dominated by rectified nonlinearities such as relu [ 18 , 4 , 13 ] , and by smooth, self-gated alternatives (most notably gelu ) that have become standard in several Transformer architectures [ 7 ] . While smooth gates can be convenient during training, many downstream pipelines in efficient deployment and model analysis are most naturally expressed for piecewise-linear networks and often assume relu -type structure (e.g., model compression through pruning and quantization, and piecewise-linear interpretability and verification toolchains) [ 9 , 8 , 2 , 10 ] . This tension motivates mechanisms that retain the benefits of smooth training recipes while enabling a controlled transition to relu -compatible models. In this work, we study a hardness-parameterized variant of gelu defined as f(x;\lambda)=x\,\Phi(\lambda x) , where \Phi is the Gaussian cdf ( cdf ) and \lambda\in[1,\infty) controls the sharpness of the gate. Learning \lambda is non-trivial and requires a constrained parameterization together with a dedicated optimization scheme to avoid unstable updates and gradient attenuation, which we develop in this paper. Beyond enabling hardness profiling across layers, the learned parameter provides a practical control knob for relu -ization: starting after the first 25\% of training epochs, we anneal \lambda toward a large target and evaluate whether the resulting model can be replaced by relu with minimal loss in the validation metric. We emphasize that \lambda - gelu plays two complementary roles in this work. First, when \lambda is learned jointly with the weights, it acts as a lightweight probe of layerwise gating hardness, enabling us to characterize consistent hardness hierarchies across architectures and initializations. Second, the same parameter provides a practical control knob for relu -ization: once an initial smooth-training phase is completed, \lambda can be deterministically annealed to enforce relu -like gating and enable a clean activation replacement for downstream compatibility or inference deployment. In summary, our contributions are: • We introduce \lambda - gelu as a minimal hardness-controlled family x\,\Phi(\lambda x) with \lambda\in[1,\infty) . • We develop a stable constrained optimization scheme to learn \lambda jointly with the network parameters, based on softplus reparameterization and a dedicated learning-rate scaling for this hardness variable. • We propose a deterministic hardening-and-replacement protocol: after a smooth-training phase, we freeze \lambda and linearly anneal it toward a principled target \lambda_{\mathrm{target}} , and we evaluate relu substitution at the best-validation checkpoint without further weight updates. The remainder of the paper is organized as follows. Section II reviews related work on smooth and parametric activation functions and on relu -centric downstream pipelines in compression, interpretability, and verification. Section III introduces the proposed framework and the constrained optimization strategy for learning \lambda . Section IV reports an empirical evaluation across multiple datasets and architectures and studies the proposed annealing procedure. Finally, Section V concludes and outlines future research directions. II Related Work Rectified activations, and in particular relu , mitigate gradient attenuation effects that hinder deep optimization and have become a default choice in many pipelines [ 18 , 4 , 13 ] . At the same time, their non-differentiability at the origin and issues such as inactive (“dead”) units have motivated a large body of work on smooth alternatives. Representative examples include gelu [ 7 ] , silu [ 3 ] , Swish [ 20 ] , and Mish [ 17 ] . Many of these can be interpreted as self-gated activations of the form x\,\phi(x) , where the gate \phi(x)\in(0,1) is a smooth approximation to a Heaviside-type decision. In particular, gelu has become prevalent in Transformer-based architectures, where it is frequently adopted as a default activation [ 7 ] . A natural extension of both rectifiers and smooth gates is to make the nonlinearity adaptive by introducing learnable parameters controlling slope, curvature, saturation, or sharpness. Examples include parametric rectifiers such as prelu [ 5 ] and other variants that learn thresholds or slopes, as well as smooth gates with tunable “temperature”-like parameters. These approaches are motivated by improved optimization and task adaptivity, but they also raise stability concerns: unconstrained parameters may drift to degenerate regimes, and naive parameterizations can lead to vanishing or overly unstable gradients. The closest formulation to our own is the learnable- \beta variant of Swish [ 20 ] , x\cdot\sigma(\beta x) , which also introduces a scalar sharpness-like parameter. However, \beta is unconstrained (it can take negative values, collapsing toward zero), and no annealing protocol toward a piecewise-linear target has been proposed for it. prelu [ 5 ] is also related in that it learns a per-layer scalar, but it modifies the slope of the negative half-plane rather than gating sharpness, and does not converge to relu as the parameter grows. Our \lambda - gelu differs from both in three concrete ways: (i) \lambda\geq 1 is enforced by construction, grounding the family at standard gelu ; (ii) we derive a principled annealing target \lambda_{\mathrm{target}} from

Read the complete text