IGLU : The Integrated Gaussian Linear Unit Activation Function

Mingi Kang, Zai Yang, Jeova Farias Sales Rocha Neto · 2026

Read IGLU : The Integrated Gaussian Linear Unit Activation Function by Mingi Kang, Zai Yang, Jeova Farias Sales Rocha Neto (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-06861.txt.

Introduction

Kang and colleagues introduce IGLU, an activation function for networks, where activations control gradient flow and stability. Modern transformers favor smoother choices over ReLU, like GELU, yet the math linking them is unclear. IGLU comes from mixing GELU gates with a half-normal distribution, giving a closed-form Cauchy gate. That yields a one-parameter family sliding between identity-like and ReLU-like shape via sharpness. Its Cauchy gate falls as a power law for negative side, so gradients never hit zero for finite inputs, unlike GELU's Gaussian gate.

Catalog record

CREATORMingi Kang, Zai Yang, Jeova Farias Sales Rocha Neto
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41062
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.06861

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

IGLU : The Integrated Gaussian Linear Unit Activation Function Mingi Kang, Zai Yang, Jeova Farias Sales Rocha Neto Abstract Activation functions are fundamental to deep neural networks, governing gradient flow, optimization stability, and representational capacity. Within historic deep architectures, while ReLU has been the dominant choice for the activation function, modern transformer-based models increasingly are adopting smoother alternatives such as GELU and other self-gated alternatives. Despite their empirical success, the mathematical relationships among these functions and the principles underlying their effectiveness remains only partially understood. We introduce IGLU , a parametric activation function derived as a scale mixture of GELU gates under a half-normal mixing distribution. This derivation yields a closed-form expression whose gating component is exactly the Cauchy CDF, providing a principled one-parameter family that continuously interpolates between identity-like and ReLU-like behavior via a single sharpness parameter σ \sigma . Unlike GELU ’s Gaussian gate, IGLU ’s heavy-tailed Cauchy gate decays polynomially in the negative tail, guaranteeing non-zero gradients for all finite inputs and offering greater robustness to vanishing gradients. We further introduce IGLU-Approx , a computationally efficient rational approximation of IGLU expressed entirely in terms of ReLU operations that eliminates transcendental function evaluation. Through evaluations on CIFAR-10, CIFAR-100, and WikiText-103 across ResNet-20, ViT-Tiny, and GPT-2 Small, IGLU achieves competitive or superior performance on both vision and language datasets against ReLU and GELU baselines, with IGLU-Approx recovering this performance at substantially reduced computational cost. In particular, we show that employing a heavy-tailed gate leads to considerable performance gains in heavily imbalanced classification datasets. 1 Introduction Activation functions are a central component of the representational power and optimization dynamics of deep neural networks. By introducing non-linearity between layers, they enable models to approximate complex functions and learn hierarchical structures across a wide range of tasks. While the Rectified Linear Unit (ReLU) [ 1 ] became the dominant activation function in early deep learning systems, its inherent operational issues and the rise of transformer architectures brought smoother alternatives such as the Gaussian Error Linear Unit (GELU) [ 18 ] , Sigmoid Linear Unit (SiLU) [ 13 ] and Mish [ 28 ] , to mention a few. These smooth activation functions offer improved gradient flow and strong empirical performance, yet the theoretical principles that distinguish them remain only partially understood. Despite their widespread adoption, relatively little work has examined how different smooth activation functions relate to one another mathematically, or how systematic modifications to their components influence network behavior. The literature reflects this: of the hundreds of activation functions proposed since the early days of deep learning [ 23 ] , the vast majority are motivated by empirical intuition rather than principled theoretical reasoning. Existing work largely focuses on benchmarking performance across tasks, with activation design often reduced to heuristic modification of existing functions without a unifying framework. A principled approach, one that grounds activation design in well-understood mathematical structures and offers clear control over the resulting behavior, remains notably uncommon in the literature. In this work, we introduce IGLU , a novel parametric activation function derived a continuous mixture of GELU activations, leading to arctangent-based function [ 30 ] . By this construction, IGLU provides a unified formulation that unifies ReLU and GELU through a single tunable parameter \sigma . It further offers a novel connection between neural network training and heavy-tailed data modeling, as its artangent-based design directly implies the use of a Cauchy CDF gate as an alternative to the Gaussian-one found in GELU . In practice, due to its slower decay this formulation guarantees larger gradient magnitudes than GELU at the activation, a property that promotes robustness against vanishing gradients. Finally, inspired by GELU in its more computationally efficient \operatorname{tanh}(\cdot) -based approximation, we propose rational approximation of IGLU that does not require the execution of any transcendental function, leading to a faster gating mechanism. We provide the first systematic evaluation of IGLU on image classification and language modeling tasks, examining its mathematical properties, proposing an efficient approximation suitable for practical deployment, and comparing its empirical performance against ReLU, GELU, and other mainstream activation functions across standard benchmarks. We also provide speed tests that demonstrate IGLU ’s computational superiority against other common activations and also show how the heavy-tailed modeling within IGLU ’s formulation contributes better performances at highly unbalanced datasets. Through both theoretical analysis and experimental validation, we demonstrate IGLU ’s advantages in representational flexibility, optimization stability, and overall model performance. 2 IGLU Formulation Activation functions enable neural networks to learn non-linear representations, and their mathematical structures determine gradient flow, optimization stability, and representational capacity. In this section, we formulate the proposed IGLU activation function and demonstrate how it generalizes ReLU and GELU . We then analyse the theoretical implications of our proposed activations. Finally, we derive a rational approximation to IGLU and offer a further connection between ReLU and itself. -5 -4 -3 -2 -1 0 1 2 3 4 5 -1 0 1 2 x ReLU GELU IGLU -5 -4 -3 -2 -1 0 1 2 3 4 5 -1 0 1 2 x a=0 a=0.1 a=0.5 a=1.0 a=5.0 a=10.0 Figure 1 : Relevant activation functions in this work. (Left) Comparison of standard activation functions including our proposed IGLU ( \sigma=1 ). (Right) Parameterized \textsc{GELU}_{a} for various a values showing convergence to ReLU as a\to\infty . 2.1 Background The Rectified Linear Unit ( ReLU ) [ 1 ] is a piecewise linear activation function that is non-smooth at the origin and outputs zero for all negative inputs: \operatorname{\text{ReLU}}(x)=\max(0,x)=x\cdot\operatorname{step}(x), (1) where \operatorname{step}(x) equals 0 if x<0 and 1 otherwise. This step function acts as a hard gating mechanism, determining which activations propagate forward through the network. ReLU is computationally efficient and easy to optimize, but it suffers from the “dying ReLU ” problem due to its zero-gradient region for negative inputs. A widely used smooth alternative is the Gaussian Error Linear Unit ( GELU ) [ 18 ] , which replaces the non-differentiable \operatorname{step}(\cdot) in ReLU with a smooth, monotonically increasing function that approximates it. It introduces a soft probabilistic gate based on the Gaussian cumulative distribution function (CDF): \textsc{GELU}(x)=x\cdot\Phi(x),\quad\Phi(x)=\int_{-\infty}^{x}\frac{1}{\sqrt{2\pi}}e^{-t^{2}/2}dt. (2) Unlike ReLU , which deterministically suppresses negative inputs, GELU scales inputs according to their probability under a standard normal distribution, yielding a smooth transition between attenuation and amplification, as show in Fig. 1 (left). GELU also carries a probabilistic interpretation as it can be understood as applying a stochastic regularizer to the identity mapping: inputs are retained proportionally to the likelihood that a standard normal random variable falls below x [ 18 ] . This perspective connects GELU to noise-injection and dropout-like mechanisms, but in a deterministic and differentiable form. GELU has been widely adopted in transformer architectures such as BERT [ 9 ] and GPT [ 6 ] due to its smoother gradient behavior and improved optimization stability in large-scale models. In practice, the exact formulation in ( 2 ) is often approximated for computational efficiency using the following closed-form expression: \textsc{GELU}(x)\approx\frac{x}{2}+\frac{x}{2}\tanh\!\left(\!\sqrt{\frac{2}{\pi}}\left(x+0.044715x^{3}\right)\!\right), (3) which avoids the explicit computation of the Gaussian CDF while closely matching its shape. Despite its empirical success, GELU presents certain disadvantages. First, it is computationally more expensive than ReLU due to the evaluation of transcendental functions (e.g., \tanh , \operatorname{erf}

Read the complete text