Uncovering a Winning Lottery Ticket with Continuously Relaxed Bernoulli Gates
Itamar Tsayag, Ofir Lindenbaum · 2026
Read Uncovering a Winning Lottery Ticket with Continuously Relaxed Bernoulli Gates by Itamar Tsayag, Ofir Lindenbaum (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-08914.txt.
Introduction
Itamar Tsayag and Ofir Lindenbaum hunt strong lottery tickets with smooth Bernoulli gates. Over-parameterized nets cost too much memory and compute for tight deployments, and score-based ticket search cannot differentiate. Their method trains only gate parameters with weights frozen, optimizing a sparsity objective by gradient descent, and reaches up to 90 percent sparsity with minimal accuracy loss.
Catalog record
| CREATOR | Itamar Tsayag, Ofir Lindenbaum |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 20596 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.08914
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Uncovering a Winning Lottery Ticket with Continuously Relaxed Bernoulli Gates Itamar Tsayag, Ofir Lindenbaum Abstract Over-parameterized neural networks incur prohibitive memory and computational costs for resource-constrained deployment. The Strong Lottery Ticket (SLT) hypothesis suggests that randomly initialized networks contain sparse subnetworks achieving competitive accuracy without weight training . Existing SLT methods, notably edge-popup , rely on non-differentiable score-based selection, limiting optimization efficiency and scalability. We propose using continuously relaxed Bernoulli gates to discover SLTs through fully differentiable, end-to-end optimization—training only gating parameters while keeping all network weights frozen at their initialized values. Continuous relaxation enables direct gradient-based optimization of an e l l 0 \ ell_{0} -regularization objective, eliminating the need for non-differentiable gradient estimators or iterative pruning cycles. To our knowledge, this is the first fully differentiable approach for SLT discovery that avoids straight-through estimator approximations. Experiments across fully connected networks, CNNs (ResNet, Wide-ResNet), and Vision Transformers (ViT, Swin-T) demonstrate up to 90% sparsity with minimal accuracy loss—nearly double the sparsity achieved by edge-popup at comparable accuracy—establishing a scalable framework for pre-training network sparsification. I Introduction The increasing complexity and scale of modern deep learning models have significantly amplified their computational demands. While larger models generally achieve higher accuracy, their substantial resource requirements limit their training [ 25 , 26 ] and accessibility for many practitioners. A key factor contributing to these demands is the over-parameterization commonly observed in large neural networks [ 23 ] . Over-parameterized models often contain redundant components that do not substantially improve performance but still consume considerable computational resources [ 36 ] . To address this inefficiency, neural network compression techniques - such as pruning [ 12 , 7 , 38 , 5 ] , quantization [ 3 ] , knowledge distillation [ 2 , 1 ] , and low-rank factorization [ 27 , 8 ] - have been developed. These methods aim to reduce memory usage and computational costs while maintaining accuracy, making them essential for deploying models on resource-constrained devices. Among these techniques, pruning has gained significant attention for identifying and removing unnecessary parameters while preserving the network’s predictive capabilities. This aligns closely with the Lottery Ticket Hypothesis (LTH) [ 4 ] , which posits that within large, over-parameterized networks, there exist smaller subnetworks - termed “winning tickets” - that can achieve performance comparable to the original model. An intriguing discovery by [ 37 ] revealed that randomly selected subnetworks designated as winning tickets can exhibit non-trivial accuracy without any weight training . Building on this, [ 18 ] demonstrated that sufficiently over-parameterized networks inherently contain subnetworks capable of achieving near-target accuracy without additional training, effectively equating pruning to training; similar principles are further leveraged in [ 32 ] to enable efficient fine-tuning and compression via structured sparsity. This distinction gave rise to the terms weak lottery tickets, which require further training to achieve optimal performance, and strong lottery tickets (SLTs), which achieve competitive accuracy without modifying the original weights. The existence of strong lottery tickets was empirically validated by [ 24 ] , where a subnetwork in a randomly initialized Wide-ResNet50 matched the classification accuracy of a trained ResNet34 on ImageNet. Their edge-popup algorithm remains the primary approach for identifying SLTs. However, this method has notable limitations: it requires calculating “popup scores” for each weight and relies on a non-differentiable gradient estimator, which makes optimization inefficient and limits scalability to larger architectures. Our Contribution. We propose a fundamentally different approach to discovering strong lottery tickets. Unlike existing methods that rely on non-differentiable estimators or iterative prune-train cycles, we introduce continuously relaxed Bernoulli gates [ 34 , 35 , 9 ] that enable fully differentiable , end-to-end identification of sparse subnetworks with weights frozen at initialization. To our knowledge, this is the first approach to discover strong lottery tickets by continuously relaxing binary gates, eliminating the need for non-differentiable gradient estimation while achieving competitive sparsity-accuracy trade-offs across convolutional and transformer architectures. II Related Work II-A Lottery Ticket Hypothesis The Lottery Ticket Hypothesis (LTH), introduced by [ 4 ] , posits that within a randomly initialized dense neural network lies a sparse subnetwork - termed a winning ticket - that can be trained in isolation to match the accuracy of the original network. Winning tickets are typically identified through iterative magnitude-based pruning (MBP), where weights with the lowest magnitude are pruned, and the remaining weights are reset to their initial values. While effective, this process is computationally expensive due to repeated pruning and retraining cycles. II-B Strong Lottery Tickets Building on LTH, [ 24 ] introduced strong lottery tickets (SLTs) - subnetworks that achieve competitive accuracy without any weight training. Using the edge-popup algorithm, they demonstrated that a randomly initialized Wide ResNet-50 contains a subnetwork matching the performance of a trained ResNet-34 on ImageNet. Their work highlighted the critical role of initialization, with Scaled Kaiming Normal distribution yielding optimal results. The theoretical foundations for SLTs were established by [ 18 ] , who proved that sufficiently overparameterized networks contain subnetworks that approximate any target network solely through pruning. Pensia et al. [ 21 ] later relaxed these requirements, showing that any ReLU network with width n and depth l can be \epsilon -approximated by sparsifying a random network that is O(\log(nl)) times wider and twice as deep. II-C Neural Network Pruning Pruning methods can be categorized by when they operate. Post-training methods sparsify networks after training, including magnitude pruning [ 6 ] , movement pruning [ 28 ] , and structured pruning approaches. These methods typically require a trained network as input. During-training methods integrate pruning into the training process, often using \ell_{1} regularization to encourage sparsity. However, \ell_{1} regularization fails to achieve true sparsity during optimization and requires post-hoc thresholding [ 29 ] . Pre-training (or “at-initialization”) methods, including our approach, identify sparse subnetworks before any weight training occurs. The edge-popup algorithm [ 24 ] falls into this category but relies on non-differentiable score-based selection, limiting optimization efficiency. II-D Continuous Relaxations for Discrete Selection Gradient-based optimization over discrete variables (e.g., binary gates) requires continuous relaxations. The Concrete (or Gumbel-Softmax) distribution [ 17 , 10 ] provides a differentiable approximation to discrete random variables. [ 16 ] extended this with Hard-Concrete gates for \ell_{0} regularization in network sparsification. However, these logistic-based relaxations suffer from high gradient variance due to heavy-tailed distributions. [ 34 ] proposed Stochastic Gates (STG) using Gaussian-based relaxations, which achieve lower variance and more stable optimization compared to logistic alternatives. Originally developed for feature selection, STG has not previously been applied to discovering strong lottery tickets. Our work bridges this gap by leveraging STG’s favorable optimization properties to pre-train neural networks for sparsification. III Proposed Solution We present a novel approach for uncovering strong lottery tickets (SLTs) by leveraging continuously relaxed Bernoulli variables as differentiable gating mechanisms [ 34 ] . These gates enable unstructured weight pruning while maintaining end-to-end differentiability. Importantly, the original network weights W^{(i)} remain frozen throughout training; only the gating parameters are optimized, similar in spirit to Edge-Popup [ 24 ] . The gating network selectively identifies important weights by applying learnable gating variables at the weight level, facilitating efficient sparsification while preserving model expressiveness. The gating variable for a weight between neurons i