Pruning as Evolution: Emergent Sparsity Through Selection Dynamics in Neural Networks

Zubair Shah, Noaman khan · 2026

Read Pruning as Evolution: Emergent Sparsity Through Selection Dynamics in Neural Networks by Zubair Shah, Noaman khan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-10765.txt.

Introduction

Zubair Shah and Noaman Khan frame neural network pruning as an evolutionary process. Networks are often trained with far more parameters than they need. Many weights turn redundant as learning proceeds. Most pruning methods cut by decree, using importance scores or penalty terms set from above.

That view treats pruning as a central decision imposed on a trained model. Shah and Khan instead stress selection dynamics inside training itself. Sparsity can emerge as some connections win and others fade. The frame borrows the logic of natural selection for model design. It may lead to leaner networks that grow sparse on their own, without a separate cutting stage.

Catalog record

CREATORZubair Shah, Noaman khan
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-10765

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Pruning as Evolution: Emergent Sparsity Through Selection Dynamics in Neural Networks Zubair Shah, Noaman khan Abstract Neural networks are commonly trained in highly overparameterized regimes, yet empirical evidence consistently shows that many parameters become redundant during learning. Most existing pruning approaches impose sparsity through explicit intervention, such as importance-based thresholding or regularization penalties, implicitly treating pruning as a centralized decision applied to a trained model. This assumption is misaligned with the decentralized, stochastic, and path-dependent character of gradient-based training. We propose an evolutionary perspective on pruning: parameter groups (neurons, filters, heads) are modeled as populations whose influence evolves continuously under selection pressure. Under this view, pruning corresponds to population extinction: components with persistently low fitness gradually lose influence and can be removed without discrete pruning schedules and without requiring equilibrium computation. We formalize neural pruning as an evolutionary process over population masses, derive selection dynamics governing mass evolution, and connect fitness to local learning signals. We validate the framework on MNIST using a population-scaled MLP (784–512–256–10) with 768 prunable neuron populations. All dynamics reach dense baselines near 98% test accuracy. We benchmark post-training hard pruning at target sparsity levels (35–50%): pruning 35% yields ≈ \approx 95.5% test accuracy, while pruning 50% yields ≈ \approx 88.3–88.6%, depending on the dynamic. These results demonstrate that evolutionary selection produces a measurable accuracy–sparsity tradeoff without explicit pruning schedules during training. 1 Introduction Neural networks are often trained with more parameters than ultimately required, as evidenced by successful weight pruning and the existence of sparse trainable subnetworks ( Han et al. 2015 ; Frankle and Carbin 2019 ) . Pruning methods exploit this redundancy to reduce model size and computational cost. Most existing approaches, however, impose sparsity through explicit intervention, such as importance-based thresholding ( LeCun et al. 1990 ; Hassibi and Stork 1993 ) , regularization penalties ( Louizos et al. 2018 ) , or discrete pruning schedules ( Zhu and Gupta 2017 ) . These methods implicitly assume that pruning is a deliberate decision made by a centralized optimizer or an external controller acting on a trained model. This assumption sits uneasily with the decentralized, stochastic, and path-dependent nature of neural network training. Individual neurons or filters do not reason about global model structure, optimize long-term objectives, or coordinate explicitly with one another. Instead, learning unfolds through local gradient updates, noisy feedback, and implicit competition for representational capacity ( Saxe et al. 2019 ) . From this perspective, sparsity need not be the result of a rational optimization decision; it may instead arise as the outcome of selection dynamics acting over time. This observation motivates a fundamentally different question: how does sparsity emerge during training, even when no explicit pruning objective or discrete pruning mechanism is enforced? Addressing this question requires moving beyond static optimization viewpoints and toward a dynamical understanding of how model components persist, decline, or vanish over the course of learning. In this work, we propose an evolutionary perspective on neural network pruning. Rather than modeling neurons or parameter groups as rational agents that solve optimization problems, we model them as populations subject to selection pressures . Each parameter group competes for limited representational influence, and its prevalence evolves over time according to a fitness signal reflecting its contribution during training. Under this view, pruning corresponds to population extinction : components with persistently low fitness gradually lose influence and are eliminated, without requiring explicit pruning decisions, thresholding rules, or equilibrium computation. This evolutionary framing departs sharply from classical game-theoretic approaches. We do not assume rationality, best responses, or convergence to Nash equilibria. Instead, we focus on replicator-style selection dynamics that operate locally and incrementally ( Hofbauer and Sigmund 1998 ) . Sparsity emerges as a temporal phenomenon, driven by differential growth and decay under selection, rather than as a static solution to a global optimization problem. The evolutionary perspective offers several advantages. First, it aligns naturally with the stochastic and decentralized mechanisms underlying gradient-based training. Second, it explains why pruning often unfolds gradually and exhibits effective irreversibility, rather than occurring through abrupt, externally imposed thresholding. Third, it provides a principled foundation for understanding pruning as an emergent property of learning dynamics, rather than as an auxiliary objective imposed on a trained model. In this paper, we formalize neural network pruning as an evolutionary process over parameter populations. We derive selection dynamics governing the evolution of population mass (participation) for parameter groups and analyze their extinction behavior under mild fitness assumptions. Importantly, sparsity arises without explicit pruning schedules and under only mild decay pressure, emerging instead from evolutionary selection dynamics during training. 1.1 Contributions The main contributions of this paper are: 1. We introduce an evolutionary formulation of neural network pruning that does not rely on rationality, best-response dynamics, or equilibrium assumptions. 2. We model parameter groups as populations whose influence evolves under selection dynamics driven by local fitness signals. 3. We provide a theoretical characterization of sparsity emergence through population extinction under mild fitness heterogeneity. 4. We offer a dynamic, process-level explanation of pruning as an emergent outcome of learning. 1.2 Scope and Limitations This work focuses on establishing a principled formulation of neural network pruning as an evolutionary selection process and on characterizing its extinction behavior. While sparsity can emerge under sufficient fitness heterogeneity, the proposed framework is not designed to enforce aggressive pruning in highly symmetric or redundancy-dominated settings. Rather than optimizing for maximal compression, our goal is to understand when and why selection dynamics induce sparsity during training, thereby laying a foundation for future extensions that incorporate richer fitness signals. 2 Related Work The problem of neural network pruning has been studied extensively from multiple perspectives, ranging from classical sensitivity analysis to modern dynamic sparse training methods. This section surveys the main approaches to pruning and contextualizes our evolutionary framework within this broader landscape. The earliest principled approaches to pruning sought to identify which parameters could be removed with minimal impact on the loss function. Optimal Brain Damage (OBD) ( LeCun et al. 1990 ) introduced the idea of using second-order information to estimate the sensitivity of the loss to parameter removal, computing the diagonal of the Hessian to approximate the change in loss caused by setting individual weights to zero. Optimal Brain Surgeon (OBS) ( Hassibi and Stork 1993 ) extended this framework by using the full Hessian and considering the optimal weight adjustment after pruning, allowing for more accurate importance estimation. While these methods provided theoretical foundations for pruning, their reliance on expensive second-order computations limited their applicability to large modern architectures. Contemporary work has explored various approximations and scalable alternatives, but the fundamental tension between theoretical rigor and computational efficiency remains. Magnitude-based pruning represents a simpler and more scalable alternative, based on the intuition that small-magnitude parameters contribute less to network output. Han et al. ( Han et al. 2015 ) demonstrated that iterative magnitude-based pruning, combined with retraining, could achieve substantial compression rates while maintaining accuracy. This approach has become a standard baseline due to its simplicity and effectiveness. Regularization-based methods provide a complementary perspective by encouraging sparsity during training rather than imposing it post hoc. Classical \ell_{1} and \ell_{2} regularization ( Hanson and Pratt 1988 ) indirectly promote sparsity through weight decay

Read the complete text