Topology-Aware Activation Functions in Neural Networks

Pavel Snopov 1 and Oleg R. Musin 1 University of Texas Rio Grande Valley - School of Mathematical and Statistical Sciences Brownsville - USA · 2025

Read Topology-Aware Activation Functions in Neural Networks by Pavel Snopov 1 and Oleg R. Musin 1 University of Texas Rio Grande Valley - School of Mathematical and Statistical Sciences Brownsville - USA (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-12874.txt.

Introduction

Pavel Snopov and Oleg R Musin design new activation functions for neural networks in this 2025 study. An activation function is the small curve each unit applies to its input, and standard curves such as ReLU can only bend data in limited ways. The authors argue that this limit hurts when a network must squeeze data through narrow layers, where the shape of the data cloud must be cut and rearranged.

Their two new functions add a splitting action that can cut the data shape on purpose. In tests on synthetic and real data, the tunable version beat classic functions in narrow-layer setups and stayed level in wider ones. The authors share their code and frame the result as a step toward networks that manage data shape by design, not by accident.

Catalog record

CREATORPavel Snopov 1 and Oleg R. Musin 1 University of Texas Rio Grande Valley - School of Mathematical and Statistical Sciences Brownsville - USA
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE15 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-12874

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Topology-Aware Activation Functions in Neural Networks Pavel Snopov 1 and Oleg R. Musin 1 University of Texas Rio Grande Valley - School of Mathematical and Statistical Sciences Brownsville - USA Abstract This study explores novel activation functions that enhance the ability of neural networks to manipulate data topology during training. Building on the limitations of traditional activation functions like ReLU \relu , we propose SmoothSplit \smoothsplit and ParametricSplit \parametricsplit , which introduce topology ¡¡cutting¿¿ capabilities. These functions enable networks to transform complex data manifolds effectively, improving performance in scenarios with low-dimensional layers. Through experiments on synthetic and real-world datasets, we demonstrate that ParametricSplit \parametricsplit outperforms traditional activations in low-dimensional settings while maintaining competitive performance in higher-dimensional ones. Our findings highlight the potential of topology-aware activation functions in advancing neural network architectures. The code is available via https://github.com/Snopoff/Topology-Aware-Activations . 1 Introduction and Related Work Despite significant advances, the underlying mechanisms of neural network learning remain only partially understood. Studies such as [ 1 ] have shown that a well-trained network (i.e., one achieving near-zero generalization error) progressively transforms a complex dataset M=M_{a}\cup M_{b} into a topologically simpler one. This transformation is largely attributed to the non-injective nature of \relu , which tends to ¡¡glue¿¿ points together, whereas functions like \tanh preserve the input topology. Additional work [ 2 , 3 , 4 , 5 ] suggests that untangling latent manifolds is crucial for improving classification performance, emphasizing the role of topological transformations in deep learning. Moreover, topology-aware activation functions have been applied successfully in tasks such as image segmentation [ 6 ] and graph neural networks [ 7 ] . Motivated by these insights, we propose novel non-homeomorphic activation functions that ¡¡split¿¿ data manifolds—effectively ¡¡cutting¿¿ topology rather than simply compressing it. The functions, \smoothsplit and \parametricsplit , are designed to enhance a network’s ability to restructure its internal representations in a topology-aware manner. Through experiments on both synthetic and real-world datasets, we demonstrate that \parametricsplit notably improves performance in low-dimensional settings while remaining competitive in higher dimensions, underscoring the potential of topology-aware activation functions for advancing neural network architectures. 2 Non-Homeomorphic Activation Functions As discussed in the introduction, the effectiveness of \relu (as detailed in [ 1 ] ) is largely due to its topological properties when considered as a function from \mathbb{R} to \mathbb{R} . Since \relu is not a homeomorphism, it alters the topology by compressing it; specifically, it ¡¡eliminates¿¿ non-trivial cycles in the homology of the underlying data manifold, thereby simplifying its structure. Topology simplification, however, can also be achieved by ¡¡cutting¿¿ the data manifold. When applied appropriately (e.g., along non-trivial cycles), this process divides the manifold into simpler components. A function capable of such cutting must also be non-homeomorphic; unlike \relu , which is non-injective, it must be non-surjective. For example, consider the function, termed \signsplit : \signsplit(x)=x+\mathrm{sign}(x)c, where c is a learnable parameter, initialized randomly between 0 and 1, that controls the distortion at x=0 . When used as an activation function, \signsplit divides the original data manifold along each dimension. Although effective for splitting data, \signsplit is non-differentiable and therefore unsuitable for use in neural networks. To overcome this limitation, we propose a smooth approximation, \smoothsplit , defined as \smoothsplit(x)=x+\tanh(\alpha x)c, where \alpha (optimized during training and initialized randomly between 0 and 1) determines the sharpness of the approximation. For sufficiently large \alpha , \smoothsplit closely approximates \signsplit while maintaining differentiability, thus making it compatible with gradient-based training. While splitting data manifolds can simplify their topology, neural networks may still require the ability to ¡¡glue¿¿ points together, particularly in the final layers. To enable both splitting and compressing operations, we introduce a parametric activation function, \parametricsplit , defined as: \parametricsplit(x)=\begin{cases}bx+b\cos a-\sin a,&\text{if }x\leq-\cos a,\\[2.84526pt] x\tan a,&\text{if }-\cos a\leq x\leq\cos a,\\[2.84526pt] x+\sin a-\cos a,&\text{if }x\geq\cos a.\end{cases} This function can emulate \relu , \signsplit , or \smoothsplit under appropriate parameter settings: • For a=0,\,b=0 , \parametricsplit(x) approximates \relu(x-1) . • For a=\frac{\pi}{2},\,b=1 , \parametricsplit(x) recovers \signsplit(x) . • For a\in\left[\frac{\pi}{4},\frac{\pi}{2}\right) with b=1 , \parametricsplit(x) approximates \smoothsplit(x) , with \alpha uniquely defined by a . (a) Original Circles dataset (b) \signsplit with c=0.2 (c) \smoothsplit with c=0.2,\,\alpha=40 (d) \parametricsplit with a=\frac{3\pi}{4},\,b=1 Figure 1 : Comparison of different transformations applied to the Circles dataset. In summary, there are two primary types of manifold deformations: compression and splitting. While \relu primarily compresses the data manifold, \signsplit and \smoothsplit effect splitting. The proposed \parametricsplit function unifies these operations by enabling both compression and splitting, depending on its parameter settings. Figure 1 illustrates these transformations on the Circles dataset. Since the parameters of \parametricsplit are learnable during training, it integrates seamlessly into the neural network pipeline. In the following sections, we compare \parametricsplit with \relu and other widely used activation functions on both synthetic and real-world datasets. 3 Experiments We evaluated the proposed activation functions in binary classification tasks, comparing their performance with \relu , \tanh , and \prelu . The experiments involved two synthetic datasets, CurvesOnTorus and Circles, and one real-world dataset, the Breast Cancer Wisconsin dataset. In the synthetic datasets, each class is sampled from a distinct manifold that is intertwined with others, making linear separability impossible. The data manifolds in these synthetic datasets are one-dimensional and immersed in \mathbb{R}^{2} and \mathbb{R}^{3} for Circles and CurvesOnTorus, respectively (illustrated in Figure 2 ). As for the loss function, both in training setting and validation setting, we used binary cross-entropy loss. (a) Dataset Circles (b) Dataset CurvesOnTorus Figure 2 : Synthetic datasets We utilized fully connected neural networks for the experiments. The activation functions being compared were applied to all layers except the last one, which used \relu . This design ensured that the network ¡¡glued¿¿ data components at the final layer, as mentioned earlier. However, further ablation studies are needed to assess the impact of this configuration. To investigate the effects of network depth and layer dimensionality on performance, we varied the number of hidden layers and the dimensions of each layer. The number of hidden layers was set to \{1,2,3\} , and layer dimensions varied depending on the dataset: \{2,3,4\} for Circles, \{3,4,5,6,7\} for CurvesOnTorus, and \{30,40,80,100\} for Breast Cancer Wisconsin. The models were trained for 100 epochs with weights initialized using Xavier normal initialization, except for networks with \relu , which used Xavier uniform initialization. The learning rate was set to 0.05, and datasets were split into training and test sets in a 70/30 ratio. To ensure robustness, each experiment was repeated 10 times. # of layers Activation functions Circles CurvesOnTorus 2 3 4 3 4 5 1 \tanh 0.536 (\pm 0.066) 0.506 (\pm 0.059) 0.457 (\pm 0.038) 0.464 (\pm 0.068) 0.302 (\pm 0.056) \mathbf{0.128} \mathbf{(\pm 0.057)} \relu 0.569 (\pm 0.075) 0.563 (\pm 0.074) 0.522 (\pm 0.053) 0.599 (\pm 0.070) 0.429 (\pm 0.112) 0.292 (\pm 0.156) \prelu \mathbf{0.512} \mathbf{(\pm 0.093)} 0.507 (\pm 0.071) 0.505 (\pm 0.076) 0.484 (\pm 0.112) 0.337 (\pm 0.118) 0.168 (\pm 0.107) \smoothsplit 0.543 (\pm 0.085) \mathbf{0.482} \mathbf{(\pm 0.081)} 0.461 (\pm 0.052) 0.534 (\pm 0.045) 0.377 (\pm 0.140) 0.274 (\pm 0.106) \parametricsplit 0.527 (\pm 0.012) 0.494 (\pm 0.087) \mathbf{0.398} \mathbf{(\pm 0.128)} \mathbf{0.462} \mathbf{(\pm 0.144)} \mathbf{0.251} \mathbf{(\pm 0.193)} 0.202 (\pm 0.122) 2 \tanh 0.591 (\pm 0.048) 0.530 (\pm 0.066) 0.490 (\pm 0.086) 0.452 (\pm 0.118) 0.242 (\pm

Read the complete text