Gompertz Linear Units: Leveraging Asymmetry for Enhanced Learning Dynamics

Indrashis Das, Mahmoud Safari, Steven Adriaensen, Frank Hutter · 2025

Read Gompertz Linear Units: Leveraging Asymmetry for Enhanced Learning Dynamics by Indrashis Das, Mahmoud Safari, Steven Adriaensen, Frank Hutter (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-03654.txt.

Introduction

Four researchers introduce a new switch-like function for training neural networks, in a 2025 paper. Such functions, called activation functions, shape how each unit in a network responds and how smoothly learning flows. The best-known one, ReLU, can leave units stuck off for good. Newer smooth rivals eased that flaw, and the team aims to beat them.

Their function, named GoLU, multiplies a value by a curve drawn from the Gompertz function, which leans to one side. That lean, the authors argue, trims spread in the hidden layers better than rival curves while keeping gradients healthy. They tested GoLU across many tasks, from image sorting and language modeling to video-style scene labeling and image making. In those tests it topped the leading rival functions, which the team offers as proof it is a sturdy all-round choice.

Catalog record

CREATORIndrashis Das, Mahmoud Safari, Steven Adriaensen, Frank Hutter
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-03654

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Gompertz Linear Units: Leveraging Asymmetry for Enhanced Learning Dynamics Indrashis Das, Mahmoud Safari, Steven Adriaensen, Frank Hutter Abstract Activation functions are fundamental elements of deep learning architectures as they significantly influence training dynamics. ReLU, while widely used, is prone to the dying neuron problem, which has been mitigated by variants such as LeakyReLU, PReLU, and ELU that better handle negative neuron outputs. Recently, self-gated activations like GELU and Swish have emerged as state-of-the-art alternatives, leveraging their smoothness to ensure stable gradient flow and prevent neuron inactivity. In this work, we introduce the Gompertz Linear Unit (GoLU), a novel self-gated activation function defined as GoLU ⁡ ( x ) = x ​ Gompertz ​ ( x ) \mathrm{GoLU}(x)=x\,\mathrm{Gompertz}(x) , where Gompertz ⁡ ( x ) = e − e − x \mathrm{Gompertz}(x)=e^{-e^{-x}} . The GoLU activation leverages the right-skewed asymmetry in the Gompertz function to reduce variance in the latent space more effectively compared to GELU and Swish, while preserving robust gradient flow. Extensive experiments across diverse tasks, including Image Classification, Language Modeling, Semantic Segmentation, Object Detection, Instance Segmentation, and Diffusion, highlight GoLU’s superior performance relative to state-of-the-art activation functions, establishing GoLU as a robust alternative to existing activation functions. 1 Introduction Developing effective activation functions has been a longstanding area of research in deep learning. In the early days, the Sigmoid ( Verhulst, 1838 ; Rumelhart et al., 1986 ) and Tanh ( LeCun et al., 2002 ) functions were popular choices. However, these activations can suffer from the vanishing gradient problem due to their tendency to saturate. The introduction of ReLU ( Nair & Hinton, 2010 ) marked a turning point, as it allowed for more efficient training by alleviating the vanishing gradient problem and inducing intensity equivariance ( Nair & Hinton, 2010 ) . However, ReLU comes with its own challenges, notably the dying-ReLU problem. To address these challenges, several ReLU variants have been developed, including LeakyReLU ( Maas et al., 2013 ) , PReLU ( He et al., 2015 ) and ELU ( Clevert et al., 2015 ) . Despite the emergence of these alternatives, ReLU remains one of the most widely used activation functions today, owing to its simplicity as a piecewise linear function and its computational efficiency. In the deep learning community, the landscape of activation functions has gradually shifted towards self-gated activations such as Gaussian Error Linear Units (GELU) ( Hendrycks & Gimpel, 2016 ) , Swish ( Ramachandran et al., 2017 ) , and Mish ( Misra, 2019 ) . These activations provide probabilistic interpretations while enhancing robustness when combined with normalization techniques ( Ioffe & Szegedy, 2015 ; Ba et al., 2016 ; Ulyanov et al., 2016 ; Wu & He, 2018 ; Zhang & Sennrich, 2019 ) . Unlike ReLU, which strictly enforces gradient preservation due to its piecewise-linear nature, Swish, Mish and GELU, as smooth activation functions, relax these constraints. Their smoothness allows for improved gradient flow without strictly adhering to intensity equivariance. In this work we introduce Gompertz Linear Units (GoLU), a new activation function of the self-gated family based on the Gompertz function ( Gompertz, 1825 ) as its gating mechanism. The Gompertz function was initially developed to model human mortality rates, and has since been widely applied in biology. Notably, it also possesses a probabilistic interpretation, as it represents the cumulative distribution function (CDF) of the standard Gumbel distribution. While both the Sigmoid function and the Gaussian CDF exhibit reflection symmetry around the point (0, 0.5), the Gompertz function manifests a subtle rightward asymmetry, leading to distinct qualitative behavior. Our experiments indicate that GoLU, compared to existing self-gated activations, effectively reduces variance in the latent representation. Moreover, it contributes to a smoother loss landscape , making it less sensitive to small perturbations in the model parameters. Additionally, an analysis of the learned weights in our trained models reveals that GoLU induces a more spread weight distribution compared to commonly used activations (see Section 2.2 for details). A more spread weight distribution may indicate the network’s ability to capture a diverse range of features from the data. On the other hand, variance reduction in activation outputs can help eliminate irrelevant information, allowing the network to focus on distinguishing patterns and potentially mitigate overfitting. However, overly broad weight distributions may introduce instability, while excessive variance reduction could result in underfitting and the loss of essential features, ultimately degrading performance. Extensive, task-specific evaluations, suggest that GoLU effectively addresses this trade-off by achieving a balanced level of both weight distribution and variance reduction, leading to improved performance over baseline activations (see Section 3 ). To facilitate reproducibility, we have made our code available at https://github.com/automl/GoLU . 2 Gompertz Linear Unit 2.1 Definition and Properties In this section, we introduce the GoLU activation function and discuss its properties. GoLU is defined through Equations 1 and 2 and visualized in Figure 1 (Left) as the red curve, alongside other activation functions for comparison. \mathrm{GoLU}(x)=x\,\mathrm{Gompertz}(x) (1) \mathrm{Gompertz(x)}=e^{-e^{-x}} (2) The gate function \mathrm{Gompertz(x)} refers to the Gompertz function introduced in ( Gompertz, 1825 ) and is plotted in red in Figure 1 (Right), alongside the gate functions of other gated activations. Figure 1 : Activation functions (Left) and their corresponding gate functions (Right). GoLU and its gate, the Gompertz function, are highlighted in red. Note the slight rightward shift of the Gompertz gate. The Gompertz function can also be interpreted probabilistically, as it corresponds to the CDF of the standard Gumbel distribution, \mathrm{Gumbel}(0,1) , with probability density function \mathrm{Gumbel}(x)=e^{-(x+e^{-x})} (3) From Equations 1 , 2 and Figure 1 , we understand that, contrary to ReLU and its variants which are monotonic and non-smooth at zero, GoLU is a smooth and non-monotonic self-gated activation, similar to Swish and GELU. In fact the formulation of GoLU using exponentials makes it infinitely differentiable. However, in contrast to Sigmoid and the Gaussian CDF (i.e. the gate functions of Swish and GELU), the Gompertz function is asymmetric, as it does not mirror evenly around a central point 1 1 1 Formally, we refer to a scalar function f(x) as symmetric if there exists a point x^{*} such that for any input x we have f(x^{*}+x)-f(x^{*})=f(x^{*})-f(x^{*}-x) . . This asymmetry, which has a bias towards the right, arises from the inherent asymmetry of the Gumbel distribution, which favors positive input values, as illustrated in Figure 2 (Left). In fact, the right-leaning asymmetry of the Gumbel distribution leads to smaller gate values across the entire input range, inducing a compression effect on the output distribution. This behavior extends to GoLU, yielding output values closer to zero, both for positive and negative inputs, when compared to other gated activation functions, effectively reducing the magnitude of the activation output. We note that, while Mish also exhibits an asymmetric distribution, it is skewed to the left, producing the opposite effect relative to GoLU 2 2 2 See Appendix B for an interesting case of flipped Mish distribution with right-leaning asymmetry. . From a more localized perspective, the Gompertz gate exhibits a reduced value in particular at the origin. This leads to a decreased steepness of GoLU near this point, as indicated by \mathrm{GoLU}^{\prime}(0)=\mathrm{Gompertz}(0) from Equation 1 . This property of reduced slope magnitude is not

Read the complete text