Stochastic activations

Maria Lomeli, Matthijs Douze, Gergely Szilvasy, Loic Cabannes, Jade Copet, Sainbayar Sukhbaatar, Jason Weston, Gabriel Synnaeve, Pierre-Emmanuel Mazaré, Hervé Jégou · 2025

Read Stochastic activations by Maria Lomeli, Matthijs Douze, Gergely Szilvasy, Loic Cabannes, Jade Copet, Sainbayar Sukhbaatar, Jason Weston, Gabriel Synnaeve, Pierre-Emmanuel Mazaré, Hervé Jégou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-22358.txt.

Introduction

Maria Lomeli and nine co-authors introduce stochastic activations, a small change to how language models compute. In a model's feed-forward layers, an activation function shapes each signal. The fast ReLU function has a flaw: its flat shape for negative inputs blocks gradient flow during training, yet its sparsity at run time is valuable. Their 2025 strategy flips a coin at each use, choosing randomly between ReLU and the smoother SiLU function during training.

The team uses the trick two ways. First, pre-train with the coin flips, then fine-tune and deploy with pure ReLU: the resulting sparse activations cut inference cost and speed the model up, while beating models trained with ReLU alone. Second, they test the random choice for text generation itself. Output diversity rose, and quality stayed close to the best fixed activation paired with temperature sampling.

Catalog record

CREATORMaria Lomeli, Matthijs Douze, Gergely Szilvasy, Loic Cabannes, Jade Copet, Sainbayar Sukhbaatar, Jason Weston, Gabriel Synnaeve, Pierre-Emmanuel Mazaré, Hervé Jégou
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-22358

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Stochastic activations Maria Lomeli, Matthijs Douze, Gergely Szilvasy, Loic Cabannes, Jade Copet, Sainbayar Sukhbaatar, Jason Weston, Gabriel Synnaeve, Pierre-Emmanuel Mazaré, Hervé Jégou Abstract We introduce stochastic activations. This novel strategy randomly selects between several non-linear functions in the feed-forward layer of a large language model. In particular, we choose between SILU or RELU depending on a Bernoulli draw. This strategy circumvents the optimization problem associated with RELU , namely, the constant shape for negative inputs that prevents the gradient flow. We leverage this strategy in two ways: (1) We use stochastic activations during pre-training and fine-tune the model with RELU , which is used at inference time to provide sparse latent vectors. This reduces the inference FLOPs and translates into a significant speedup in the CPU. Interestingly, this leads to much better results than training from scratch with the RELU activation function. (2) We evaluate stochastic activations for generation. This strategy performs reasonably well: it is only slightly inferior to the best deterministic non-linearity, namely SILU combined with temperature scaling. This offers an alternative to existing strategies by providing a controlled way to increase the diversity of the generated text. † † affiliation: Meta FAIR † † affiliation: Ecole Normale Supérieure Paris Saclay † † affiliation: Paris Cité University † † date: September 25 th 2025 RELU : 1-p SILU : p Figure 1: Stochastic activation randomly selects one of two activations when x<0 : (1) RELU selected with probability 1-p ; otherwise (2) another activation, in particular SILU . 1 Introduction Large language models (LLMs) ( Devlin et al. 2019 ; Chowdhery et al. 2022 ; Brown et al. 2020 ; Vaswani et al. 2017 ) have revolutionized natural language processing, enabling unprecedented capabilities in text generation, comprehension, and reasoning. Their success stems from scaling model parameters and leveraging vast amounts of data, but this comes with a significant computational complexity. As the demand for more efficient and powerful models grows, researchers are increasingly focused on optimizing their training processes to balance performance with resource constraints. The majority of the LLM parameters are in the Feed-Forward Network (FFN) layers, where they memorize the training data. FFNs are two linear layers separated by an activation function , and sometimes an additional linear layer that serves as a gating operation. The activation is a \mathbb{R}\rightarrow\mathbb{R} non-linear function. In this context, the choice of activation function plays a crucial role for both the model’s expressivity and efficiency. The simplest activation is RELU (Rectified Linear Unit), that allows positive values to pass through and forces negative inputs to zero. RELU is sparsity-inducing , since, on average, half of its outputs are zero (in practice significantly more). Within a two-layer Multilayer Perceptron, this means that inference on the second layer is a matrix-sparse vector multiplication, so it can be implemented with fewer FLOPs than a matrix-dense vector multiplication. Note that effectively improving the runtime with this sparsity pattern remains challenging. In practice, the Sigmoid Linear Unit ( SILU ) activation, combined with a gated design, has consistently outperformed RELU in terms of model accuracy ( Shazeer 2020 ) . Unfortunately, SILU does not induce sparsity. One plausible explanation for RELU ’s underperformance is that its gradient for negative input values is zero, which hinders optimization by preventing weight updates in a significant portion of the network. Solutions like Leaky RELU ( Maas et al. 2013 ) circumvent this problem by ensuring non-zero gradient almost everywhere, but they are inferior to SILU and involve abandoning sparsity. In contrast, if the so-called “dying RELU problem” optimization challenge could be effectively addressed, RELU ’s theoretical advantages – such as sparsity and computational efficiency – may translate into performance comparable to SILU for a lower number of FLOPs. This disparity presents a challenge: how to harness the efficiency benefits of sparse activations, such as RELU , without sacrificing the empirical advantages of SILU . This motivates our exploration of alternative training strategies that mitigate RELU ’s limitations while preserving its benefits. In this work, we consider two ways to approach this problem. The first approach is activation fine-tuning, denoted Swi+FT : we pre-train the model with an activation that facilitates efficient large language model optimization, then we change the activation to RELU and adapt the model by fine-tuning it further. Our second approach, referred to as StochA (stochastic activations), is a novel technique that randomly selects between multiple activations, either at train or test time. Both approaches allow models to benefit from the superior optimization properties of SILU . These hybrid strategies combine the best of both worlds – maintaining high model performance while unlocking the computational efficiency of sparse activations. In summary, this paper makes the following contributions: • We introduce and analyze two strategies that employ different activation functions at training and inference time, namely Swi+FT and StochA . Both are complementary and make it possible to use activations at inference time that differ from those employed during pre-training. • We produce RELU -based models that are much better than those obtained with regular training, i.e., our methods significantly outperform training with RELU only. • We show that stochastic activations, when used at inference time, provides an alternative way to generate diverse sequences compared to traditional temperature sampling or other variants. 2 related work Standard activation functions Activations are at the core of deep learning, in that they are needed to depart from defining a linear function with limited expressivity. While early neural network architectures were inspired by logistic regression, such as sigmoidal and tanh activations, many activation functions have been evaluated for the Feed Forward layers (FFNs) of transformers. Vaswani et al. 2017 used RELU ( Glorot et al. 2011 ) . However, using the RELU activation function leads to some neurons getting stuck in the negative region. As a consequence, they stop learning entirely, since the gradient is zero for negative inputs, and their weights do not get updated. In contrast, Touvron et al. 2023 used SILU as the activation function for the FFN layers of the transformer for the first Llama models. Shazeer 2020 discusses the benefits of SWIGLU , which consists of SILU with gating. There exist many other activation functions such as the Gaussian Error Linear Unit ( GELU ) ( Hendrycks & Gimpel 2016 ) , Scaled Exponential Linear Unit ( SELU ) ( Klambauer et al. 2017 ) , Swish ( Ramachandran et al. 2018 ) and gated SILU , among others. In particular, the Leaky RELU ( Maas et al. 2013 ; Redmon et al. 2015 ; Ridnik et al. 2021 ; Guo et al. 2024 ) tried tackling the dying RELU problem by allowing a small, non-zero gradient when the input is negative in order to keep the neurons active and the gradients flowing, reducing the risk of dead neurons. Adaptive activation functions Lee et al. 2022 propose an activation, called Adaptive Swish ( ASH ), that uses stochastic sampling of the top-k percentile elements. It is a generalization of the Swish ( Ramachandran et al. 2018 ) activation function, that uses adaptive thresholding and selects only the values in the top percentiles and sets these to zero otherwise. The threshold is not fixed but depends on

Read the complete text