Task-Specific Activation Functions for Neuroevolution using Grammatical Evolution
Benjamin David Winter 1 , William John Teahan 2 · 2025
Read Task-Specific Activation Functions for Neuroevolution using Grammatical Evolution by Benjamin David Winter 1 , William John Teahan 2 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-10879.txt.
Introduction
Benjamin Winter and William Teahan evolve fresh activation functions for neural nets tuned to each task. Stock curves like the rectified line work well, yet they may not be best for all data. Their grammar based evolution grows new curves shaped to a net and its data.
On known two class sets the grown curves beat the stock curve by clear margins on a key score. Gains came with no extra net size, which suits small edge gear. Both train and run stay lean on energy. Task fit curves may be a cheap path to stronger small nets.
Catalog record
| CREATOR | Benjamin David Winter 1 , William John Teahan 2 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-10879
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Task-Specific Activation Functions for Neuroevolution using Grammatical Evolution Benjamin David Winter 1 , William John Teahan 2 Abstract Activation functions play a critical role in the performance and behaviour of neural networks, significantly impacting their ability to learn and generalise. Traditional activation functions, such as ReLU, sigmoid, and tanh, have been widely used with considerable success. However, these functions may not always provide optimal performance for all tasks and datasets. In this paper, we introduce Neuvo GEAF – an innovative approach leveraging grammatical evolution (GE) to automatically evolve novel activation functions tailored to specific neural network architectures and datasets. Experiments conducted on well-known binary classification datasets show statistically significant improvements in F1-score (between 2.4% and 9.4%) over ReLU using identical network architectures. Notably, these performance gains were achieved without increasing the network’s parameter count, supporting the trend toward more efficient neural networks that can operate effectively on resource-constrained edge devices. This paper’s findings suggest that evolved activation functions can provide significant performance improvements for binary classification networks. I Introduction Choosing the right activation function for a neural network for any particular task or dataset can be a tedious process and may require substantial research as there is no rule of thumb for selecting an activation function because the choice of activation functions is dependent on the context/task [ 1 ] . The value from an activation function that is passed on to the loss function is extremely important. In a standard neural network, the activation function passes its value on to a loss function. A problem many researchers face is not knowing which activation function to use for each task. There are ten activation functions that are most prominent: Binary step , Linear , Sigmoid , Tanh , ReLU , Leaky ReLU , ELU , Swish , Softmax and Softplus . They all have their own issues and trade-offs, but their efficiency is dependent on the data/task they are performing. However, a poorly chosen activation function can have detrimental effects on the network’s results. The vanishing gradient problem is a phenomenon that occurs when inputs that are close to 1 automatically snap to 1 whilst inputs that are close to 0 automatically snap to 0 or -1 respectively [ 2 ] . This results in a lack of gradient for results, hence the name vanishing gradient’. Its opposite, the exploding gradient’ is the result of the error gradients accumulating during an update and resulting in typically exponentially large error gradients each epoch. This in turn results in an unstable network and can even change the network’s weights to NaN values. Furthermore, some of the activation functions previously mentioned are momentum based, resulting in a disparity in convergence time over a non-momentum based activation function. This in itself has a large effect on the time it takes to converge to a minima, as well as determining if the value will get stuck on a saddle point. Various papers [ 3 , 1 ] have shown that even if a catastrophic error is avoided in the selection of a neural network activation function, the neural network’s accuracy can be vastly improved by the design of custom activation functions. These are activation functions that are not classed as typical and may only improve results for one specific dataset. As the field of machine learning increasingly pushes toward deployment on edge devices and energy-efficient computing, optimizing smaller neural networks becomes critically important. While large language models (LLMs) excel at general tasks, they are impractical for many real-world applications requiring rapid, energy-efficient classification on resource-constrained hardware. The ability to achieve higher accuracy with compact networks through optimized activation functions represents a significant advancement for applications like medical diagnostics, sensor networks, and IoT devices, where both accuracy and efficiency are paramount concerns. This paper outlines a novel method of creating custom activation functions through Grammatical Neuroevolution on a task-specific basis – Neuvo GEAF, in which we create the search space using a Backas-Naur form grammar and recurse through the grammar to map evolvable genotypes to phenotypes which represent a neural network’s activation functions. Our approach delivers substantial performance improvements without increasing model complexity or computational requirements during inference, making it particularly valuable for deployment scenarios where energy consumption and computational resources are limited. The paper is organised as follows. Section two describes the background of Grammatical Evolution and its benefits over its competitor algorithms, whilst also discussing some of the more prominent Neuroevolution algorithms and models, specifically ones that feature Grammatical Neuroevolution or the creation of hand crafted activation functions. Section three summarises the grammar that is used for this project, and more specifically its usefulness in creating both emergent activation functions and well known activation functions. The methodology of the creation of this model regarding implementation is defined in section four of this paper. Section five regards the experimental setup, which datasets are used, the baseline network parameters used for fair testing as well as providing the hardware specifications used to make, train, validate and test the model. Section six discusses the results, the benefits and drawbacks of this approach and a general discussion surrounding the future of Grammatical Neuroevolution. Lastly, section seven concludes the paper and describes avenues that Grammatical Neuroevolution should focus on in the future. II Background and Related Work II-A Grammatical Evolution Grammatical evolution (GE) is an evolutionary algorithm that can evolve complete programs in an arbitrary language using a variable-length string that represents the genome. The genome determines which production rules in a Backus-Naur form grammar definition are used in a genotype-to-phenotype mapping process to a program [ 4 ] . GE has several benefits over Genetic Programming (GP), the first of which being an efficiency gain due to the ability to constrict the search space and thus avoid bloat. Bloat is typically defined as an increase in the size of the program without a corresponding improvement in fitness. Whilst many researchers have created models to combat bloat in GP, it still exists [ 5 , 6 , 7 , 8 ] . A major benefit of GE is that it works in a way that is more similar to how the role of DNA in natural evolution works than other EA’s, and therefore can exploit the natural divide between genotype and phenotype. Humans look and behave relatively similar to each other in comparison to other organisms. However, according to a DNA-sequencing analysis of the Y chromosome, it has been found that every time human DNA is passed from one generation to the next, it accumulates 100–200 new mutations [ 9 ] . Most of these mutations are known as silent mutations, which have no observable effect on the organism’s phenotype. GE utilises a concept called degeneracy which can produce these silent mutations within a genotype which have no effect on the phenotype [ 10 ] . A further feature of GE is deliberate bias, which allows the developer to create the grammar in such a way that certain features are more likely to be chosen. The grammar used in this project (seen in Table 1 ) was unbiased as there was an equal amount of chance for each feature. However, one could add deliberate bias by making one feature