Sign-In to the Lottery: Reparameterizing Sparse Training From Scratch
Advait Gadhikar, Tom Jacobs, Chao Zhou, Rebekka Burkholz · 2025
Read Sign-In to the Lottery: Reparameterizing Sparse Training From Scratch by Advait Gadhikar, Tom Jacobs, Chao Zhou, Rebekka Burkholz (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-12801.txt.
Introduction
Advait Gadhikar, Tom Jacobs, Chao Zhou, and Rebekka Burkholz tackle sparse network training in 2025. Training a sparse network from scratch lags behind pruning a full one. A known theory says the right starting signs for the weights are the key to closing that gap.
Their method, Sign-In, reshapes the weights during training to force sign flips. Those flips complement what pruning methods can do, making the approach an add-on rather than a rival. Theory and tests both point to better from-scratch results. The authors also name the main open challenge that keeps a gap to dense-to-sparse training.
Catalog record
| CREATOR | Advait Gadhikar, Tom Jacobs, Chao Zhou, Rebekka Burkholz |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-12801
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Sign-In to the Lottery: Reparameterizing Sparse Training From Scratch Advait Gadhikar, Tom Jacobs, Chao Zhou, Rebekka Burkholz Abstract The performance gap between training sparse neural networks from scratch (PaI) and dense-to-sparse training presents a major roadblock for efficient deep learning. According to the Lottery Ticket Hypothesis, PaI hinges on finding a problem specific parameter initialization. As we show, to this end, determining correct parameter signs is sufficient. Yet, they remain elusive to PaI. To address this issue, we propose Sign-In, which employs a dynamic reparameterization that provably induces sign flips. Such sign flips are complementary to the ones that dense-to-sparse training can accomplish, rendering Sign-In as an orthogonal method. While our experiments and theory suggest performance improvements of PaI, they also carve out the main open challenge to close the gap between PaI and dense-to-sparse training. † † affiliationnotice: Equal contribution 1 Introduction As neural network sizes scale to billions of parameters ( Hoffmann et al. 2022 ; Kaplan et al. 2020 ) , building efficient training pipelines is of paramount importance. Neural network sparsification plays a pivotal role in enabling such efficient pipelines ( Frantar & Alistarh 2023 ; Jaiswal et al. 2024 ) . State-of-the-art sparsification methods rely on eliminating parameters during training ( Peste et al. 2021 ; Kuznedelev et al. 2024 ; Kusupati et al. 2020 ) or by exploring multiple sparse masks dynamically ( Evci et al. 2020 ; Lasby et al. 2023 ; Chen et al. 2021 ) to find high-performing sparse networks. The success of these methods can be attributed to finding an optimal combination of parameters and sparse masks ( Paul et al. 2023 ) . However, identifying the optimal combination requires training the dense network before it can be sparsified such that information from the dense network can be inherited by the subsequent sparse network ( Gadhikar & Burkholz 2024a ) . It is then no surprise that the most successful sparsification methods rely on dense training phases. Dense-to-sparse methods like gradual pruning ( Han et al. 2015 ) or continuous sparsification ( Kusupati et al. 2020 ) explicitly start from a dense network. Dynamic sparse training methods like RiGL ( Evci et al. 2020 ) utilize dense gradient information. However, this comes at the cost of high computational and memory demands during the dense training phase. Moreover, the benefits of sparsification become mostly available towards the end of training or during inference. Is there a way to train sparse networks from the start? The Lottery Ticket (LT) Hypothesis ( Frankle & Carbin 2019 ) posits that there exist sparse networks at random initialization which enjoy optimal parameter-mask coupling and can achieve the same performance as their dense counterpart. Finding such sparse networks at initialization would unlock the benefits of sparse training from scratch. A plethora of pruning at initialization (PaI) methods have attempted to find such sparse networks at initialization ( Lee et al. 2019 ; Tanaka et al. 2020 ; Wang et al. 2020 ; Pham et al. 2023 ; Liu et al. 2021 ) , however, they struggle to match the generalization performance of dense-to-sparse training methods. Understanding and overcoming the limitations of PaI remains an open research problem and is the focus of our work. The main challenge for PaI methods is to find highly performant parameter-mask combinations by leveraging task-specific information efficiently ( Kumar et al. 2024 ; Frankle et al. 2021 ) . The problem is thus comprised of two parts: a) learning a mask and b) optimizing the weights corresponding to the mask, which is hard if the mask is extremely sparse. Here we are less concered about learning a mask, but instead focus on the hard problem of optimizing parameter-mask combinations, given a mask at initialization. At the center of our investigation is the question: What hinders training a given sparse network and can the gap between dense-to-sparse training and PaI be closed? The key to addressing this question lies in promoting parameter sign flips . Echoing the analysis of learning rate rewinding (LRR) by Gadhikar & Burkholz 2024b , through a broader empirical investigation of state-of-the-art dense-to-sparse training methods, we identify a critical phenomenon responsible for finding optimal parameter-mask combinations: the ability to flip parameter signs during training. We term this phenomenon sign alignment, as the signs encode all relevant information additionally to the mask (and data) to enable effective training from scratch. Moreover, as we show, sign alignment occurs early in the dense-to-sparse training process, promoting convergence toward flatter minima. In contrast, PaI faces challenges in stabilizing sign flips. In line with this observation, it was shown in a theoretically tractable setting of a two-layer network consisting of one neuron that training a sparse network from scratch fails to flip signs from an unfavorable initialization ( Gadhikar & Burkholz 2024b ) . Motivated by this insight, we propose a way to flip and learn correct signs during sparse training from scratch in a higher number of cases (see Figure 2 ). We achieve this via Sign-In , a reparameterization m\odot w of the non-masked parameters, providing them with an inner degree of freedom (where m and w are continuous parameters). Note that this reparameterization, depending on the inner scaling, has a bias towards sparsity. This has been used by the continuous sparsification methods Spred ( Ziyin & Wang 2023 ) and PILoT ( Jacobs & Burkholz 2025 ) . In contrast, we do not use the reparameterization for sparsification but for improving the optimization geometry to enable sign alignment and thus seek to counteract its sparsity bias. We propose an inner scaling that promotes sign flips by rebalancing network layers and, correspondingly, the speed in which they learn. This provably recovers a different sign case than dense training and could thus be seen as complementary (see Figure 2 ). Moreover, in combination with a moderate degree of over-parameterization, which tends to be common in PaI, Sign-In fully solves the one-hidden-neuron problem. In line with this insight, we verify in experiments that existing dense-to-sparse methods can be improved by utilizing Sign-In . Most importantly, we demonstrate across multiple experiments that Sign-In significantly enhances the performance of sparse training from scratch, including those using random masks. We attribute this improvement to better sign alignment and the ability to discover flatter minima. However, as we show, Sign-In cannot replicate the same sign flipping mechanism as that of initial dense training, and is still not competitive with the accuracy of dense-to-sparse methods. This poses the question: Could there exist another unknown parameterization that could replace the sign flipping mechanism of dense training? Unfortunately, the answer is negative, as our proof of an impossibility statement reveals. While we can report progress in the right direction, it remains an open problem to completely close the gap between PaI and dense-to-sparse methods. Contributions 1) We empirically show that dense-to-sparse training methods identify trainable parameter-mask combinations via early sign alignment allowing them to find flatter minima and achieve state-of-the-art performance. 2) We provide evidence for the following hypothesis: Across dense-to-sparse training masks, the learned signs define an initialization that enables competitive sparse training from scratch. A fully random initialization (that is decoupled from the mask) performs on par or