A Composite Activation Function for Learning Stable Binary Representations
Seokhun Park 1 , Choeun Kim 1 , Kwanho Lee 1 , Sehyun Park 1 , Insung Kong 2 , Yongdai Kim 1 1 Department of Statistics, Seoul National University 2 Department of Applied Mathematics, University of Twente {shrdid, kimchoeun, khlee0527}@snu.ac.kr , [email protected] , [email protected], [email protected] · 2026
Read A Composite Activation Function for Learning Stable Binary Representations by Seokhun Park 1 , Choeun Kim 1 , Kwanho Lee 1 , Sehyun Park 1 , Insung Kong 2 , Yongdai Kim 1 1 Department of Statistics, Seoul National University 2 Department of Applied Mathematics, University of Twente {shrdid, kimchoeun, khlee0527}@snu.ac.kr , [email protected] , [email protected], [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-11558.txt.
Introduction
Park and five colleagues propose the Heavy Tailed Activation Function, HTAF, for learning stable binary representations. Binary activations aid compute, memory and interpretability, but Heaviside activations block gradient training because they cannot be differentiated. HTAF smooths the Heaviside function as a sigmoid hyperbolic tangent composite, keeping large gradient mass near zero with slower tail decay. Spiking, binary and deep Heaviside networks all train stably with HTAF. The authors also build Implicit Concept Bottleneck Models, interpretable image models using HTAF for discrete features with performance matching or beating standard models.
Catalog record
| CREATOR | Seokhun Park 1 , Choeun Kim 1 , Kwanho Lee 1 , Sehyun Park 1 , Insung Kong 2 , Yongdai Kim 1 1 Department of Statistics, Seoul National University 2 Department of Applied Mathematics, University of Twente {shrdid, kimchoeun, khlee0527}@snu.ac.kr , [email protected] , [email protected], [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 79253 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.11558
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Composite Activation Function for Learning Stable Binary Representations Seokhun Park 1 , Choeun Kim 1 , Kwanho Lee 1 , Sehyun Park 1 , Insung Kong 2 , Yongdai Kim 1 1 Department of Statistics, Seoul National University 2 Department of Applied Mathematics, University of Twente {shrdid, kimchoeun, khlee0527}@snu.ac.kr , [email protected] , [email protected], [email protected] Abstract Activation functions play a central role in neural networks by shaping internal representations. Recently, learning binary activation representations has attracted significant attention due to their advantages in computational and memory efficiency, as well as interpretability. However, training neural networks with Heaviside activations remains challenging, as their non-differentiability obstructs standard gradient-based optimization. In this paper, we propose Heavy-Tailed Activation Function (HTAF) , a smooth approximation to the Heaviside function that enables stable training with gradient-based optimization. We construct HTAF as a sigmoid–hyperbolic tangent composite function and theoretically show that it maintains a large gradient mass around zero inputs while exhibiting slower gradient decay in the tail regions. We show that Spiking Neural Networks, Binary Neural Networks and Deep Heaviside neural Networks can be trained stably using HTAF with gradient-based optimization. Finally, we introduce Implicit Concept Bottleneck Models (ICBMs), an interpretable image model that leverages HTAF to induce discrete feature representations. Extensive experiments across various architectures and image datasets demonstrate that ICBM enables stable discretization while achieving prediction performance comparable to or better than standard models. 1 Introduction Activation functions play a central role in neural networks by transforming continuous signals into structured representations that enable effective learning and prediction. Through this transformation, activation functions critically influence not only prediction performance ( ( 2 ; 20 ) ), but also the robustness ( ( 64 ; 74 ) ) and interpretability of learned representations ( ( 7 ; 15 ; 30 ; 17 ) ). Early neural network models ( 55 ) were based on threshold-based activation functions, which are equivalent to the Heaviside activation function x\mapsto\mathbb{I}(x\geq 0) . However, such non-differentiable activations made gradient-based learning intractable. To address this issue, differentiable surrogate activation functions such as the sigmoid function were introduced ( 57 ). Subsequently, to mitigate the gradient vanishing problem, Rectified Linear Units (ReLU) became widely adopted ( 19 ). Recently, the Heaviside activation function has attracted increasing attention again in the AI community due to their potential benefits in memory and computational efficiency ( ( 77 ; 12 ; 51 ) ), interpretability ( ( 73 ; 79 ; 39 ) ), and connections to biological neural networks ( ( 83 ; 40 ; 65 ) ). Furthermore, the Heaviside activation function is closely related to quantized and binarized neural networks ( ( 9 ; 25 ; 51 ) ), which have become increasingly important for energy-efficient deployment of large deep neural networks such as large language models (LLMs, ( 70 ; 43 ) ). Although the Heaviside activation function offers several advantages, training neural networks that use it with gradient-based optimization remains difficult because the function is inherently non-differentiable. To address this issue, a substantial body of work ( 1 ; 40 ; 83 ; 27 ; 47 ) has focused on developing methods for learning neural networks with the Heaviside activation function. 1 proposed Straight-Through Estimator (STE), which enables gradient-based optimization of non-differentiable functions by using surrogate gradients. While STE serves as a practical workaround, it introduces a mismatch between the forward and backward passes and often leads to biased or suboptimal optimization ( 77 ). Another approach is to replace the Heaviside activation function with a smooth surrogate ( 27 ), most notably the sigmoid function, and encourage the activations to approach binary values by increasing the slope or scale of the nonlinearity. While this approach enables the use of standard gradient-based optimization, it suffers from severe optimization instability ( ( 18 ; 6 ) ) including the gradient vanishing problem. In this paper, we propose a novel smooth activation function, called Heavy-Tailed Activation Function (HTAF) , which closely approximates the Heaviside function while enabling stable and accurate training with standard gradient-based optimization. The key idea is to design an activation function that preserves a large gradient near zero (to approximate the Heaviside function closely), while allowing the gradient to decay slowly for large-magnitude inputs (to avoid the gradient vanishing problem). Specifically, HTAF has two parameters that allow the gradient magnitude in the tail regions to be adjusted without affecting the gradient near zero. That is, by adjusting these two parameters, HTAF allows the gradient in the tail region to decay more slowly, while still preserving a high gradient near zero. As an application of HTAF, we consider two problems. Firstly, using HTAF, we propose a general learning framework for neural networks with the Heaviside activation function, including Spiking Neural Networks (SNNs, ( 65 ; 14 ; 44 ) ), Binary Neural Networks (BNNs, 51 ; 54 ; 41 ), and Deep Heaviside Neural Networks (DHNs, 33 ; 12 ), enabling stable end-to-end optimization without surrogate gradients for activation functions. Second, we propose Implicit Concept Bottleneck Models (ICBMs), an extension of Concept Bottleneck Models (CBMs, 31 ), which employ HTAF as the activation function in the final layer of the backbone feature extractor during training. HTAF encourages each feature to take values close to either zero or one. At inference time, HTAF is replaced by the Heaviside function, yielding exact binary activations and resulting in fully discrete and interpretable features without substantially sacrificing accuracy. To sum up, our key contributions are summarized as follows. 1. We introduce Heavy-Tailed Activation Function (HTAF) , a novel smooth activation function, which approximates the Heaviside activation function. 2. We theoretically show that, by appropriately tuning its two parameters, HTAF can control the rate of gradient decay in tail regions while keeping a large gradient near zero, thereby enabling stable learning of deep neural networks (DNNs) with HTAF activation with standard gradient-based optimization. 3. We demonstrate that HTAF enables gradient-based learning of DNNs with the Heaviside activation function, allowing stable end-to-end optimization and improving prediction performance over existing training methods, including those based on surrogate gradients. 4. We propose Implicit Concept Bottleneck Models (ICBM), an interpretable image model that leverages HTAF to induce binary latent representations corresponding to implicit concepts. We show that the resulting model is interpretable while achieving competitive or improved prediction performance across various image datasets compared to non-interpretable models. 2 Related works 2.1 Deep Neural Networks with the Heaviside activation function Heaviside-Activated Neural Networks and their advantages. DNNs with the Heaviside activation function have been widely studied in various neural network models, including Spiking Neural Networks (SNNs, 83 ; 14 ; 26 ), Binary Neural Networks (BNNs, 51 ; 54 ), and Deep Heaviside Neural Networks (DHNs, 33 ). SNNs use the Heaviside function to model discrete spike generation, offering a biologically inspired and energy-efficient computation paradigm. Similarly, in BNNs, activations and weights are constrained to binary values, allowing significant reductions in memory usage and computational cost. DHNs are standard deep neural networks employing the Heaviside activation function. A key advantage of such models is their efficiency during inference. The output of the Heaviside activation function can be represented using a single bit, leading to substantial memory savings compared to floating-point