Growing Networks with Autonomous Pruning Journal: Neural Networks

Charles de Lambilly, Stefan Duffner · 2026

Read Growing Networks with Autonomous Pruning Journal: Neural Networks by Charles de Lambilly, Stefan Duffner (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-19759.txt.

Introduction

De Lambilly and Duffner introduce networks that grow and prune themselves while they train. Most image classifiers keep a fixed size from start to finish. Their GNAP models start small and add capacity when progress stalls. Between growth steps, gradient descent prunes weights with no outside control. The aim is high accuracy with as few parameters as possible. On MNIST the method reached 99.44 percent accuracy with 6.2k parameters. On CIFAR10 it reached 92.2 percent with 157.8k parameters.

Catalog record

CREATORCharles de Lambilly, Stefan Duffner
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42428
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.19759

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Growing Networks with Autonomous Pruning Journal: Neural Networks Charles de Lambilly, Stefan Duffner Abstract This paper introduces Growing Networks with Autonomous Pruning (GNAP) for image classification. Unlike traditional convolutional neural networks, GNAP change their size, as well as the number of parameters they are using, during training, in order to best fit the data while trying to use as few parameters as possible. This is achieved through two complementary mechanisms: growth and pruning. GNAP start with few parameters, but their size is expanded periodically during training to add more expressive power each time the network has converged to a saturation point. Between these growing phases, model parameters are trained for classification and pruned simultaneously, with complete autonomy by gradient descent. Growing phases allow GNAP to improve their classification performance, while autonomous pruning allows them to keep as few parameters as possible. Experimental results on several image classification benchmarks show that our approach can train extremely sparse neural networks with high accuracy. For example, on MNIST, we achieved 99.44% accuracy with as few as 6.2k parameters, while on CIFAR10, we achieved 92.2% accuracy with 157.8k parameters. 1 Introduction Convolutional Neural Networks (CNNs) have achieved great successes in numerous computer vision tasks such as image classification. While some of these achievements can be attributed to the introduction of new ideas for the training phase of CNNs, many fundamental improvements have come from the search of more effective neural architectures. Historically, new architectures were developed by researchers through a difficult, time-consuming, trial-and-error method. However, in recent years, there has been a growing interest in ways to automate and optimise this architecture design phase. A popular approach is called Neural Architecture Search (NAS), and it has achieved great progress recently, spawning over 1000 papers in the last four years [ 23 ] . NAS methods can be distinguished through their search space, search strategy and performance evaluation strategy. The search space is the realm of possible architecture NAS can select. The search strategy is the strategy NAS uses to explore the search space. Finally, the performance evaluation strategy is the way NAS evaluate architectures coming under its consideration. NAS has been shown to be able to surpass human designers in many tasks. However, despite various improvements in its efficiency they remain computationally expensive because many models with different hyperparameters need to be trained. Another common way of automating the architecture design phase is pruning. By training a highly over-parameterised neural network compared to what would theoretically be necessary to adequately model the data, and then pruning its weights, it is possible to keep the performance of the large network while only keeping few of its weights. However, training the very large network is still a computationally expensive task, and it can also easily lead to overfitting. Moreover, the design and sizing of the architecture of the initial network is still a manual process that may be sub-optimal. For example, the choice of the number of layers is critical and is usually not reduced by pruning. Alternatively, it is also possible to automate the design phase with a growing network. The idea is to do the exact opposite of pruning, i.e. train a small network, and then periodically add neurons to it to help it model better the data. There are three different questions that every growing network architecture must address: when to add neurons, where to add neurons, and how to initialise the weights of the new neurons [ 3 ] . Many existing such architectures have taken inspiration from real-world neural growth of biological nervous systems. However, steering the growth of the neural network in a way that makes sure that the added neurons are strictly necessary remains a challenge. Therefore, the risk is that the resulting network is larger than their fixed-size counterparts with comparable classification accuracy as each growing step may add some redundancy in the model. To address these issues, in this paper, we use growing and pruning in a combined optimization process that changes the topology and size of the neural network during training while minimising its classification loss. We introduce Growing Networks with Autonomous Pruning (GNAP), a novel way of using pruning to keep a growing network as small as possible while still letting the growing phases improve classification performance. The overall approach consists in jointly training the weights and the topology of a specific CNN architecture encouraging sparse networks while repeatedly growing its structure whenever the sparsity-inducing pruning approaches saturation. Our contributions are the following: 1. a new approach for training neural networks adapting dynamically their size and topology during training by means of an alternating growing and pruning strategy and 2. a specific dynamic and highly flexible CNN architecture that allows sparse connections from any layer to any succeeding layer. We experimentally show on three public image classification benchmarks that our proposed approach leads to extremely sparse CNN with a high classification accuracy compared to the state of the art in sparse and efficient CNN models. Finally, we also show that our method can easily be extended to structured pruning, where instead of automatically removing single weights, we prune entire convolution kernels ( i.e . 2D matrices). 2 Related Work Several methods have been proposed in the literature to produce very sparse neural networks. Many of them start from dense architectures and apply pruning in such a way that very high pruning rates ( >90\% ) can be obtained, e.g . either before training [ 11 ] via a weights sensitivity analysis or based on the lottery ticket hypothesis [ 4 , 22 , 19 , 5 ] that effectively identifies sub-networks from a trained dense network and retrains these sub-networks. Another family of methods is based on NAS, like MCUNet [ 12 ] , where sparse networks are obtained by formulating an optimisation problem searching for the best configuration (hyperparameters, network architecture etc.) based on a pre-defined search space. In the following, we will focus on methods based on sparsity-inducing pruning and growing neural networks. Sparsity-inducing training: Sparsity in deep neural networks can be achieved through different techniques. One of them is to encourage sparse connectivity or topologies of the neural network during training by introducing an additional term in the loss function, e.g. the \ell_{1} loss [ 17 ] . Pruning is then performed by removing the components that have a value close to zero. This term can be defined for fully-connected layers but also for feature maps or convolution kernels/filters in CNNs [ 14 , 25 , 20 , 10 ] . In ART [ 6 ] , the sparsity-inducing regularisation term is adaptive during training and allows to gradually shift from exploration and exploitation in the optimisation process. Another approach proposed by [ 15 ] , called Sparse Evolutionary Training (SET), is to repeatedly prune a certain fraction of weights in each layer during training and replace them by new random values. This has then been extended in [ 16 ] by a dynamic sparse reparameterisation (DSR) algorithm, where this pruning rate and allocation of weights is dynamically set across layers thus leading to more sparse architectures. The method introduced in [ 27 ] achieves neural network pruning during training through gradient

Read the complete text