The Multiple Ticket Hypothesis: Random Sparse Subnetworks Suffice for RLVR Israel Adewuyi, Solomon Okibe, Vladmir Ivanov Abstract The Lottery Ticket Hypothesis demonstrated that sparse subnetworks can match full-model performance, suggesting parameter redundancy. Meanwhile, in Reinforcement Learning with Verifiable Rewards (RLVR), recent work has shown that updates concentrate on a sparse subset of parameters, which further lends evidence to this underlying redundancy. We study the simplest possible way to exploit this redundancy: training only a randomly selected subset of parameters at extreme sparsities. Empirically, we find that training just 1% of parameters matches or exceeds full-parameter RLVR finetuning across 3 models and 2 task domains. Moreover, different random masks show minimal overlap ( ≤ 0.005 \leq 0.005 Jaccard similarity) and yet all succeed, suggesting pretrained models contain many viable sparse subnetworks rather than one privileged set. We term this the Multiple Ticket Hypothesis . We explain this phenomenon through the implicit per-step KL constraint in RLVR, which restricts updates to a low-dimensional subspace, enabling arbitrary sparse masks to succeed. 1 Introduction Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a powerful technique for post-training large language models (LLMs), enabling strong performance across domains like mathematics and code generation. Recent works revealed an intriguing property of RLVR: despite updating all model parameters during training, the optimization process naturally concentrates changes on a sparse subset of parameters. Mukherjee et al. ( 38 ) showed that RLVR effectively finetunes only 5-30% of parameters across different tasks, algorithms, and models, a finding validated by Zhu et al. ( 60 ) . Critically, this sparsity emerges intrinsically from the policy optimization objective itself rather than from explicit regularization (including KL penalty) or sparse constraints. These observations raise a natural question: if RLVR naturally updates only a small subset of parameters, what happens when we explicitly restrict training to a random sparse subset from the start? The answer depends on the domain. Chen et al. ( 9 ) found random sparse masks fail on vision tasks and careful iterative pruning was required to find winning tickets. However, Xu & Zhang ( 54 ) recently showed random masks at 0.001% density succeed for supervised fine-tuning (SFT) on NLP tasks, suggesting the regime matters. Does this extend to RLVR, which has fundamentally different dynamics: policy gradients, on-policy sampling, and implicit KL constraints ( 60 ) ? Motivated by the observed intrinsic sparsity of RLVR updates ( 38 ) and the distinct optimization geometry revealed by Zhu et al. ( 60 ) , we investigate whether random sparse training can succeed in the RLVR regime. In this work, we demonstrate that training a randomly selected subset ( \leq 1%) of parameters at \geq 99% sparsity matches or exceeds full-parameter RLVR finetuning performance. We validate this finding across three models (Qwen2.5 0.5B Base and Instruct and 1.5B models) and two distinct task domains (mathematical and logical reasoning).More surprisingly, we find that multiple independent random masks succeed - testing 20 different random 1% masks reveal that they all achieve comparable performance, despite sharing less than 0.5% of parameters in common (Jaccard similarity \approx 0.005 ). We term this phenomenon the Multiple Ticket Hypothesis : pretrained LLMs contain many sparse subnetworks capable of successful RLVR finetuning, and random sampling at sufficient density reliably finds one. This stands in contrast to the classical Lottery Ticket Hypothesis ( 18 ) , which posits that sparse subnetworks must be carefully identified through iterative pruning. Our findings suggest that for RLVR objective, the lottery has many winning tickets, and any random draw is likely to succeed. (a) GSM8K (20 seeds) (b) MATH-500 (20 seeds) Figure 1 : Multiple random parameter subsets match or exceed full finetuning at 99% sparsity on Qwen-2.5-1.5B. Performance of 20 random parameter subsets of Qwen-2.5-1.5B across 100 training steps for GSM8K and MATH-500. 0% sparsity means full parameter finetuning. 99% sparsity indicates that 1% of parameters were trained. To explain this parameter redundancy, we provide a plausible theoretical explanation grounded in the geometry of KL-constrained policy optimization. Building on prior work showing that RLVR updates satisfy implicit KL constraints ( 60 ) , we demonstrate that these constraints restrict policy updates to a low-dimensional subspace at each optimization step. This geometric restriction creates vast parameter redundancy: many different parameter subsets can span the same effective update space, making random selection effective. We formalize this intuition through the Fisher information geometry, showing that KL constraints induce a low-rank structure that explains why random sparse training succeeds. Beyond validating random sparse training as an effective baseline, our work offers practical and conceptual contributions. From a practical standpoint, training only 1% of parameters provides immediate computational benefits for RLVR research, reducing memory requirements and enabling larger batch sizes. Conceptually, our findings reveal fundamental properties about how RLVR interacts with pretrained representations: the optimization landscape contains extensive flat regions with many viable solutions, suggesting that pretrained models are vastly overparameterized relative to the dimensionality required for RLVR tasks. Our main contributions are: 1. We show that random sparse training at \geq 99% sparsity matches full RLVR finetuning across different models and tasks. 2. We further show that many random subset of parameters succeed and they share minimal parameter overlap, hence the Multiple Ticket Hypothesis. 3. We provide an explanation for the success of the sparse random subset through the geometry of KL-constrained optimization, showing how trust region constraints create low-dimensional update subspaces that many random parameter subsets can span. 2 Background and Notation 2.1 Reinforcement Learning with Verifiable Rewards Following 20 , we adopt Group Relative Policy Optimization (GRPO) ( 47 ) , an on-policy reinforcement learning algorithm that extends Proximal Policy Optimization (PPO) ( 46 ) while eliminating the need for a separate value critic. GRPO estimates advantages using relative rewards within a group of sampled responses. For each prompt x , the current policy \pi_{\theta} generates G candidate outputs \{y_{1},\dots,y_{G}\} . Their verifiable rewards \{R_{1},\dots,R_{G}\} are normalized to obtain group-relative advantages \hat{A}_{i}=(R_{i}-\mu)/\sigma , where \mu and \sigma are the group mean and standard deviation. The policy is updated by maximizing a clipped surrogate objective that favors higher-reward responses, regularized by a KL-divergence penalty against a reference policy \pi_{\text{ref}} : \displaystyle\mathcal{L}(\theta) \displaystyle=\mathbb{E}_{x,y_{i}}\Biggl[\min\!\Bigl(r_{i}(\theta)\hat{A}_{i},\;\text{clip }\!\bigl(r_{i}(\theta),1\!-\!\varepsilon,1\!+\!\varepsilon\bigr)\hat{A}_{i}\Bigr) (1) \displaystyle-\beta\,\text{KL}\bigl(\pi_{\theta}(\cdot|x)\parallel\pi_{\text{ref}}(\cdot|x)\bigr)\Biggr] with r_{i}(\theta)=\pi_{\theta}(y_{i}|x)/\pi_{\text{old}}(y_{i}|x) the importance ratio and \beta controlling regularization strength. Zero-RL Training. Following ( 20 ) , we perform ”zero-RL” training, starting directly from the pretrained base model without supervised fine-tuning. We also follow ( 56 ) in setting \beta=0 and using token-level policy gradients which removes explicit KL regularization. 2.2 Sampling Random Parameters To sample a random subset of parameters at x% sparsity, we iterate through all layers of the model and sample a random subset of the parameters with a fixed seed. Per-parameter-tensor masking. Let the model parameters be organized as a collection of tensors in different layers 1 1 1 In practice, the parameters are organized into tensors which are organized into layers. . \{\theta^{(l)}\}_{l=1}^{L}. For each parameter tensor \theta^{(l)} , we independently construct a binary mask m^{(l)}\in\{0,1\}^{\mathrm{shape}(\theta^{(l)})}. Given a target sparsity level s\in[0,1) with keep ratio p=1-s , we sample k^{(l)}=\lfloor p\cdot|\theta^{(l)}|\rfloor entries of \theta^{(l)} uniformly at random without replacement and set the corresponding k^{(l)} entries of m^{(l)} to one, with all remaining entries set to zero. This procedure results in approximately uniform sparsity across parameter tensors while the total number of active parameters is also approximately at a ratio of p . 2.3 Masked Training Masks are sampled once at initialization and held fixed throughout training. During training, gradients are computed densely for all parameters. The effective gradient used for optimization is given by \nabla_{\theta^{(l)}}^{\text{masked}}\mathcal{L}=m^{(l)}\odot\nabla_{\theta^{(l)}}\mathcal{L}, where \odot denotes elementwise multiplication. Parameters corresponding to zero entries in the mask receive zero gradient updates at all training steps and remain fixed at their initialization values. Optimizer states (e.g., momentum terms) are also maintained only for unmasked parameters. 3 Experimental Setup We validate our findings across multiple models and tasks to demonstrate that random sparse training generalizes beyond specific configurations. 3.1 Models and Datasets Models. We conduct experiments across three models of varying scales: Qwen2.5-0.5B (Base and Instruct) and Qwen2.5-1.5B. ( 43 ) . Tasks. We evaluate on two distinct domains: Mathematical Reasoning. We train Qwen2.5-1.5B and Qwen2.5-0.5B (Zero RL training ( 20 ; 57 ) ) on Hendrycks MATH ( 22 ) and evaluate on both MATH-500 and GSM8K ( 12 ) , reporting pass@1 accuracy. Logical Reasoning. We train Qwen2.5-0.5B-Instruct and evaluate on Alphabet Sort 2 2 2 https://huggingface.co/datasets/kalomaze/alphabetic-arxiv-authors-it1 , a multi-turn task that requires sorting an increasing list of names. Training and evaluation are limited to two turns of interaction. Table 1 summarizes the model-task combinations used in our experiments and we refer readers to Appendix A for the complete table of hyperparameters. Table 1 : Summary of models, training tasks, and evaluation tasks. Model Train set Eval set Qwen2.5-0.5B-It Alphabet Sort Alphabet Sort Qwen2.5-0.5B GSM8K GSM8K Qwen2.5-1.5B MATH GSM8K, MATH-500 3.2 Training Configuration Optimization. Unless otherwise stated, we use the AdamW optimizer ( 34 ) . Implementation. All experiments use a fork of the Prime-RL library ( 26 ) with Prime Environments for standardized task interfaces. (a) Alphabet sort (20 seeds) (b) GSM8K (20 seeds) Figure 2 : Multiple random parameter subsets match or exceed full finetuning at 99% sparsity on Qwen-2.5-0.5B. Performance of 20 random parameter subsets of Qwen-2.5-0.5B across 100 training steps for GSM8K and 150 steps for Alphabet sort. 3.3 Experimental Design Our experimental design tests whether random sparse training can match full-parameter RLVR finetuning across multiple random initializations. Multi-seed protocol. For each model-task-sparsity configuration, we train multiple independent models using different random masks, each requiring different learning rates. All other hyperparameters, including the training seed, remain fixed across runs. We report mean performance across the five masks, with error bars indicating standard deviation. This design allows us to assess both the effectiveness of random sparse training and the variance introduced by different random parameter selections. Sparsity levels and learning rates. Table 2 summarizes the sparsity levels tested, the corresponding number of active parameters, and the learning rates used. We provide complete hyperparameters and prompt templates in Appendix A . Figure 3 : Random sparse training matches full finetuning at different sparsities. Validation performance across sparsity levels for three tasks. Error bars show variation across five random masks. Horizontal dashed lines indicate full-parameter baselines. All results use best learning rate from sweep for each configuration. Table 2 : Sparsity levels, active parameter counts, and used learning rates. Sparsity (%) Active parameters Learning rate 0.5B 1.5B 0.5B 1.5B 0 \sim 0.5B \sim 1.5B 1\times 10^{-6} 1\times 10^{-6} 99 \sim 4.9M \sim 15M 1\times 10^{-4} 1\times 10^{-4} 99.5 \sim 2.4M \sim 7.7M 1\times 10^{-4} 1\times 10^{-4} 99.9 \sim 490K \sim 1.5M 1\times 10^{-3} 1\times 10^{-3} 99.95 \sim 247K \sim 770K 1\times 10^{-3} 1\times 10^{-3} 99.99 \sim 49K \sim 150K 1\times 10^{-3} 4\times 10^{-3} 99.999 \sim 4K \sim 15K 5\times 10^{-3} 1\times 10^{-2} 4 Results We show that pretrained language models contain a large number of disjoint sparse subnetworks that each suffice for effective reinforcement learning with verifiable rewards (RLVR) finetuning. We call this the Multiple Ticket Hypothesis (MTH). The evidence comes from training extremely sparse random subnetworks and observing consistent high performance despite negligible parameter overlap. 4.1 Multiple Ticket Hypothesis: Many Disjoint Subnetworks Succeed At 99% sparsity (1% trainable parameters), we trained 20 independent models using independently sampled random masks on Qwen2.5-1.5B (GSM8K and MATH-500) and Qwen2.5-0.5B-Instruct (Alphabet Sort). Figures 1 and 2 show that nearly all masks reach performance equal to or better than full-parameter finetuning. Average Jaccard similarity between any pair of successful masks is \approx 0.005 (Table 3 ), exactly as expected for random selection at this density. This near-zero overlap rules out the existence of a single privileged subnetwork. Instead, pretrained models appear to contain combinatorially many viable sparse tickets for RLVR and any sufficiently large random draw succeeds. With 490M (Qwen-2.5-0.5B) parameters and 99% sparsity, there are theoretically \binom{490M}{4.9M} possible masks. Our 20/20 success rate with \leq 0.5% overlap suggests the number of viable masks scales combinatorially, which is vastly more than the ”one winning ticket” paradigm. 4.2 Performance Matches Full Finetuning Down to Extreme Sparsity We next sweep sparsity using 5 random masks per level (seeds 0, 10, 42, 1002, 2001). Figure 3 reports mean validation pass@1 \pm std across masks. Key observations: • 99% to 99.95% sparsity (1% to 0.05% parameters) : Performance at these sparsities match, exceeds or slightly underperforms full finetuning on all tasks and models (GSM8K, MATH-500, Alphabet Sort; 0.5B and 1.5B scales). • 99.99% and beyond : Sharp degradation, with collapse below \sim 0.01–0.001% trainable parameters. The consistent transition point across tasks and scales suggests a task-agnostic lower bound on the effective trainable dimensionality required for RLVR, rather than a gradual degradation. 4.3 Comparison to Structured Sparsity Baselines At fixed budget (99% sparsity) (see Appendix C , a random mask outperforms structured alternatives (first-layer only, last-layer only) on all the tasks and model combinations we test in this work. No architectural bias or importance scoring is needed; random selection is sufficient. 4.4 Failure Cases Model collapse is a well established failure mode in RL training ( 28 ; 13 ; 14 ; 16 ) , but we also observe a lot more model collapse with the random masked training and we also attribute the high variance at higher sparsities in Figure 3 to this. 4.5 Summary Our experiments establish three main findings: 1. Training a random 1% of parameters is sufficient to match full RLVR finetuning across model scales and reasoning domains. 2. Successful subnetworks are highly non-overlapping (Jaccard \leq 0.005 ), supporting the Multiple Ticket Hypothesis : pretrained LLMs contain many, likely combinatorially many, viable sparse tickets for RLVR. 3. Performance depends primarily on the number of trainable parameters (effective dimensionality), not their specific identity. These results point to extreme functional redundancy in the parameter space of pretrained models when optimized under RLVR objectives. Table 3 : Jaccard similarity between pairs of successful masks. Values show mean across all mask pairs between the 5 masks for each model. Expected overlap for random masks are also shown. Configuration 99% 99.9% 99.99% Qwen2.5-0.5B 0.005 0.0005 0.000055 Qwen2.5-1.5B 0.005031 0.000498 0.0000528 Expected (random) 0.005 0.0005 0.00005 Figure 4 : Eigenspectrum analysis of the gradients. 5 Theoretical Explanation We explain the Multiple Ticket Hypothesis through the geometry of KL-constrained policy optimization. Our framework shows that per-step KL constraints restrict policy updates to a low-dimensional subspace, creating sufficient redundancy for arbitrary sparse masks to succeed. Setup and Assumptions. Let \pi_{\theta}(y|x) be a policy with parameters \theta\in\mathbb{R}^{d} , and let F(\theta) denote its Fisher information matrix with eigenvalues \lambda_{1}\geq\cdots\geq\lambda_{d} and corresponding orthonormal eigenvectors v_{1},\ldots,v_{d} . We assume: (1) Low effective rank : the top r\ll d eigenvalues capture nearly all Fisher variance, i.e., \sum_{i=1}^{r}\lambda_{i}/\sum_{i=1}^{d}\lambda_{i}\geq 1-\epsilon for small \epsilon ; (2) Delocalized eigenvectors : \|v_{i}\|_{\infty}\leq\mu/\sqrt{d} , meaning no single parameter dominates any eigenvector; (3) Small per-step updates : \|\Delta\|=O(\sqrt{K}) where K is the KL bound, ensuring second-order approximations hold. Proposition 5.1 (Low-Dimensional Policy Sensitivity) . Under assumptions (1) and (3), for any update \Delta satisfying D_{\mathrm{KL}}(\pi_{\theta+\Delta}\|\pi_{\theta})\leq K , the policy change depends only on the projection of \Delta onto the top- r eigenspace U=\mathrm{span}\{v_{1},\ldots,v_{r}\} . Components orthogonal to U have negligible impact. Proposition 5.2 (Sufficiency of Random Masks) . Under assumptions (1)–(3), let S\subset\{1,\ldots,d\} be a random subset of size k>r . With high probability, there exists an update \Delta_{S} supported on S that approximates any KL-feasible update in the top- r subspace: \|\Delta_{\parallel}-\Delta_{S}\|_{F}\leq\eta , where \|\cdot\|_{F} is the Fisher norm and \eta\to 0 as k increases. Complete theoretical outline appear in Appendix F . The key insight is that KL constraints create a trust region aligned with Fisher eigenvectors. Since only the top r directions matter for policy change (Proposition 5.1 ) and eigenvectors are delocalized across parameters, random sampling of k>r parameters reliably captures this subspace (Proposition 5.2 ). This explains both why random masks work and why multiple independent masks succeed despite minimal overlap—they each span the same low-dimensional policy-relevant subspace through different parameter combinations. Connection to Empirics. Our eigenspectrum analysis (Figure 4 ) reveals an effective rank r\approx 44 for Qwen2.5-0.5B on Alphabet Sort, representing an intrinsic dimensionality of \sim 0.0000089% of 490M parameters. This explains the observed sparsity threshold: performance degrades sharply below \sim 0.01% trainable parameters (Figure 3 ). The combinatorial number of valid masks, \binom{d}{k} choices each spanning the same r -dimensional subspace, directly yields the Multiple Ticket Hypothesis. Remark 5.3 . Our experiments use \beta=0 (no explicit KL penalty), yet Zhu et al. ( 60 ) established that on-policy RLVR satisfies implicit KL constraints through clipping and on-policy sampling. Thus \beta=0 represents a conservative test: success under weaker implicit constraints implies success under stronger explicit regularization. 6 Related Work 6.1 Post-Training of Large Language Models and RLVR Pretrained large language models ( 44 ; 8 ; 49 ; 5 ; 1 ; 10 ; 30 ) require post-training to achieve strong performance on downstream tasks. The main paradigms include supervised fine-tuning (SFT) ( 11 ; 52 ; 15 ; 24 ) and reinforcement learning (RL), often applied sequentially ( 40 ; 61 ; 20 ) . Recent progress in LLM reasoning ( 20 ) shows that Reinforcement Learning with Verifiable Rewards (RLVR) excels in domains with clear verification and correctness checks, such as mathematics, code generation, and logical reasoning. These methods typically build on policy optimization algorithms such as PPO ( 40 ) or the more memory-efficient GRPO ( 47 ) and variants that further improve them ( 56 ; 33 ; 32 ; 59 ) . RLVR’s optimization dynamics have drawn attention. Mukherjee et al. ( 38 ) showed that despite updating all parameters, RLVR concentrates changes on 5-30% of them. Zhu et al. ( 60 ) revealed implicit per-step KL constraints in RLVR even without explicit regularization ( \beta=0 ), distinguishing it from SFT. These observations inspired our explicit sparse training experiments, but unlike their focus on analyzing post-hoc subnetworks, we show that random sparse subnetworks at \geq 99% sparsity match or exceed full RLVR performance from the start, without prior training or identification. 6.2 Sparsity in Neural Network Training and Lottery Tickets The Lottery Ticket Hypothesis ( 18 ; 36 ) established that dense networks contain sparse subnetworks matching full-model performance, though identifying these “winning tickets” requires iterative pruning. Subsequent work extended LTH to various domains, including deep reinforcement learning ( 55 ; 50 ; 19 ) , and identified task-specific subnetworks in language models that can mitigate catastrophic forgetting ( 41 ; 42 ) . Chen et. al ( 9 ) showed that random sparse subnetworks underperformed winning tickets identified by iterative pruning on vision tasks. In contrast to LTH’s emphasis on a single privileged subnetwork identified via pruning, our Multiple Ticket Hypothesis reveals that pretrained LLMs contain many viable sparse subnetworks for RLVR and sampling a random subset of parameters at sufficient density reliably discovers one. 6.3 Parameter-Efficient Fine-Tuning (PEFT) PEFT methods cut costs for adapting / finetuning large models and numerous studies have established that they operate in constrained parameter subspaces ( 7 ; 25 ; 3 ; 6 ; 58 ; 53 ; 31 ; 23 ; 37 ) . Of most interest is LoRA ( 25 ) , which constrains update to low rank matrices i.e it learns the low-rank subspace during training via the adapter module. Schulman, John and Thinking Machines Lab ( 45 ) further showed that rank-1 LoRA matches full RL finetuning. In this work, we sample the subspace, rather than learn it and our focus is on RLVR, while most of the works in this domain have been focused on SFT. 6.4 Sparsity in RLVR Mukherjee et al. ( 38 ) observed RLVR’s intrinsic sparse updates (5-30%), conjecturing post-hoc subnetworks recover performance. Zhu et al. ( 60 ) linked sparsity to model-conditioned bias, emphasizing off-principal updates and spectral preservation. We align with this geometry but complement it: our framework, rooted in per-step KL constraints inducing low-dimensional subspaces, explains why arbitrary random masks at > 99% sparsity succeed without prior training. This shifts emphasis from describing sparsity to leveraging redundancy for efficient RLVR, supporting our Multiple Ticket Hypothesis over singular subnetworks. 6.5 Random Sparse Training for Fine-Tuning Most related, Xu & Zhang ( 54 ) demonstrated random masks at 0.001% trainable parameters match full SFT on NLP (language understanding and comprehension tasks), attributing flatter landscapes (smaller Hessians) and higher learning rates to overparameterization, analyzed via linear regression. Concurrently, Sampreeth et al. explored using expander graph masks instead of random masks for the initial subnetworks. We extend this to RLVR by showing that RLVR shows greater redundancy, with many independent masks succeeding. Mechanisms differ—SFT’s unconstrained updates vs. RLVR’s policy gradients, on-policy sampling, and implicit KL constraints ( 60 ) —leading to RLVR-specific low-rank Fisher structure from trust regions, not general flat landscapes. We provide empirical multiplicity evidence via Jaccard analysis and test on reasoning tasks with Qwen models, unlike their SFT classification task as RLVR success isn’t implied by SFT due to differing dynamics. 6.6 Fisher Information, Policy Optimization Geometry, and Intrinsic Dimensionality Fisher information aids understanding training dynamics, parameter importance for transfer, and generalization. In policy optimization, natural gradient methods use it for stable updates via metric tensors. We build on this by showing KL constraints in RLVR create low-rank gradient Fisher matrices, restricting updates to low-dimensional subspaces that enable random sparse training. This geometric view explains mask success and multiplicity. The success ties to intrinsic dimensionality: Aghajanyan et al. ( 2 ) showed fine-tuning needs few parameters despite billions total, aligning with overparameterization theory ( 4 ; 17 ) . Our framework applies this to RLVR, where KL induces policy-relevant low-dimensional subspaces, and delocalization lets random masks span them. The Multiple Ticket Hypothesis follows from trust-region methods in overparameterized networks. 7 Discussion and Conclusion Our investigation into random sparse training for Reinforcement Learning with Verifiable Rewards (RLVR) reveals a striking property of pretrained language models: the existence of combinatorially many viable subnetworks capable of matching full-parameter performance. This Multiple Ticket Hypothesis (MTH) fundamentally shifts our understanding of parameter redundancy in the RLVR regime. RLVR optimization We showed in Section 5 that the the Fisher is low-dim in RLVR. We conjecture that RLVR is locally optimizing a flat loss landscape. This intuition is further surported by preliminary experiments from Mukherjee et al., ( 39 ) where they showed that even simple optimizers like SDG also match and outperform optimizers such as Adam(W). These findings establish random sparse training as a strong baseline for parameter-efficient RLVR and suggest new directions for understanding how reinforcement learning interacts with pretrained language model representations. A potential point of contention is the fact that Mukherjee et al., ( 38 ) claim that the updates during RLVR are nearly full rank. We empirically show that the gradients are effectively low-rank. This distinction comes from the fact that first, Mukherjee et al., ( 38 ) estimate the rank from \Delta=\theta_{final}-\theta_{init} , while we estimate effective rank from the gradients (Appendix B ). It’s also possible that a matrix is nearly full rank, but has low effective rank. Practical Implications for Efficiency. Beyond its theoretical interest, the MTH offers immediate practical benefits for RLVR research. By training only 1% of parameters, researchers can significantly reduce the memory footprint of optimizer states and gradients, enabling the finetuning of larger models on consumer-grade hardware or the use of larger batch sizes. Unlike methods like LoRA, which require learning a low-rank adapter, random sparse training utilizes the model’s existing weights directly, acting as a highly efficient, unstructured Parameter-Efficient Fine-Tuning (PEFT) baseline. Redundancy and Pretraining. Crucially, our findings highlight that this redundancy is a byproduct of the pretraining process itself. The success of random masks when training from scratch suggests that pretraining “delocalizes” knowledge across the parameter space, creating the very landscape that RLVR subsequently navigates. 8 Limitations and Future Work While the Multiple Ticket Hypothesis provides a robust framework for understanding RLVR sparsity, several limitations remain: • Catastrophic Forgetting. Catastrophic forgetting has been explored in deep learning. Specifically in generative AI, skills acquired during pretraining are lost during subsequent finetuning stages ( 48 ; 27 ; 35 ; 21 ) . It’s been established that RLVR forgets less than SFT and we tie this to Zhu et al., ( 60 ) ’s work, as Zhu et al., ( 60 ) showed that naturally, RLVR chooses low principal weight directions. We conjecture that using random sparse mask for RLVR training is likely to lead to more catastrophic forgetting because random sampling doesn’t guarantee that principal weights aren’t selected and put under RLVR’s optimization pressure, which is likely to lead to catastrophic forgetting. We leave further exploration of this to future work. • Task Complexity and Sparsity Thresholds: We observed a consistent performance collapse when trainable parameters dropped below \sim 0.01%. While this threshold held across our reasoning tasks, more complex, multi-domain and longer horizon tasks (which is what the bulk of RLVR in practise is used for today) might require a higher “intrinsic dimensionality” and thus a lower maximum sparsity. • Model Scale: Our experiments were conducted on models up to 1.5B parameters. While the MTH appears to hold as model size increases, further validation on frontier-scale models (e.g., 70B+) is necessary to confirm if the ratio of “winning tickets” remains constant or grows with scale. We conjecture however that the MTH findings will hold for larger models as increasingly larger models are more overparameterized ( 51 ) and they would exhibit more parameter redundancy. • Stability and Model Collapse: We noted a higher frequency of model collapse at extreme sparsities. This suggests that while viable tickets exist at 99.9% sparsity, the optimization path to find them becomes increasingly narrow and sensitive to hyperparameter choices like learning rate and is deserving of more attention. Impact Statement This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here. Appendix A Complete Experimental Setup A.1 Hyperparameters Table 4 : Hyperparameters table Hyperparameter Training Setup 0.5B (Alphabet Sort) 0.5B (GSM8k) 1.5B (Maths) training steps 150 100 100 batch size 512 512 512 num rollouts 16 16 8 max tokens 128 1024 2048 sampling temperature 1.0 1.0 1.0 Optimizer AdamW AdanW AdamW clip ratio 0.2 0.2 0.2 KL constant 0 0 0 weight decay 0.01 0.01 0.01 max gradient norm 1.0 1.0 1.0 training seed 42 42 42 eval interval 10 10 10 eval sampling temperature 0.7 0.9 0.9 eval max tokens 128 1024 1024 eval metric pass@1 pass@1 pass@1 num eval samples 1024 1319 1319 eval seed 2001 - For alphabet sort, max number of turns and min number of turns are set to 2. A.2 Prompt Template Mathematical reasoning We use the following instruction template for all training and evaluation rollouts, with only the task-specific instruction changing): SYSTEM: You are a helpful mathematical AI assistant. Please reason step by step and put your final answer within \boxed{} USER: [TASK_DESCRIPTION] ASSISTANT: Alphabet sort We do not use any system prompt template for the Alphabet sort task. The dataset itself already contains instructions. SYSTEM: USER: Sort the following names in alphabetical order: [List of names] ASSISTANT: A.3 Additional details on Masks training The Qwen models have the output projection head tied to the embedding layer; we sample masks once for the embedding layer and reuse them during projection. Appendix B Eigenspectrum Analysis Methodology To compute the eigenspectrum of the gradient Fisher information matrix (Figure 4 in the main text), we used the following procedure: 1. Gradient collection: Using the Qwen2.5-0.5B model on the Alphabet Sort task, we ran 150 training steps and saved the complete gradient vector at each step. 2. Gradient matrix construction: We flattened each gradient tensor into a 1D vector of dimension d\approx 490{,}000{,}000 (corresponding to the total number of model parameters). Stacking all 150 gradient vectors produced a matrix G\in\mathbb{R}^{150\times 490M} . 3. Gram matrix computation: Rather than computing the full Fisher matrix F=G^{\top}G\in\mathbb{R}^{490M\times 490M} (which would be computationally infeasible), we computed the Gram matrix GG^{\top}\in\mathbb{R}^{150\times 150} . 4. Eigendecomposition: We performed eigenvalue decomposition on GG^{\top} to obtain the eigenspectrum. The non-zero eigenvalues of GG^{\top} are identical to those of G^{\top}G , allowing us to characterize the effective rank of the gradient space. This procedure reveals that the gradient updates lie in a low-dimensional subspace, as evidenced by the rapid eigenvalue decay shown in Figure 4 . The top few eigenvalues capture most of the variance, supporting Assumption 5.1 (low effective rank) in our theoretical framework. Appendix C Baselines We run structured sparsity baselines experiments (first and last layer) against full parameter finetune and random seed (seed = 0). Figure 5 : Comparison of random sparse training, full parameter finetuning and structured sparsity training on Qwen-2.5-1.5B. Figure 6 : Comparison of random sparse training, full parameter finetuning and structured sparsity training on Qwen-2.5-0.5B (Instruct / Base). Appendix D Learning Rate Puzzle As noted by Xu & Zhang, ( 54 ) and evident from Table 2 , we also observe that at increasing sparsities, higher learning rates are needed for the performance to match full RLVR finetuning. Figure 7 shows the learning rate sweep across multiple masks on Alphabet sort task, for Qwen2.5-0.5B-Instruct. Figure 7 : Learning-rate sweep on Alphabet Sort for Qwen2.5-0.5B-Instruct. Validation performance across sparsity levels and learning rates. All the masks for the various sparsities are seeded to 0. See the Appendix for the lr. sweep for the other seeded random masks. Appendix E Memory Savings Across Sparsities We update and track optimizer state for only the parameters being updated in a particular run. In the table below, we show the memory savings from tracking only the parameters being updated. Table 5 : Optimizer memory footprint during training (MiB). Full finetuning shown for reference. Model Full 99% 99.5% 99.9% 99.95% 99.99% 99.999% Qwen2.5-1.5B 11776 235.8 118 23.8 12 2.5 0.39 Qwen2.5-0.5B 3769.5 75.6 37.86 7.67 3.9 0.9 0.2 Appendix F Theoretical Proofs This appendix provides complete proofs for the theoretical results stated in Section 5 . F.1 Detailed Assumptions and Justifications We restate the assumptions from the main text with full justification. Assumption F.1 (Low Effective Rank) . There exists a small integer r\ll d and a small constant \epsilon>0 such that: \frac{\sum_{i=1}^{r}\lambda_{i}}{\sum_{i=1}^{d}\lambda_{i}}\geq 1-\epsilon. Justification. This assumption states that the top r eigenvectors capture nearly all the “energy” of the Fisher matrix. The empirical eigenvalue spectrum (Figure 4 ), showing rapid decay after the first few components, directly supports this assumption. Details of the eigenspectrum computation are provided in Appendix B . Assumption F.2 (Delocalization of Eigenvectors) . There exists a constant \mu>0 such that for each eigenvector v_{i} : \|v_{i}\|_{\infty}\leq\frac{\mu}{\sqrt{d}}. Justification. This condition states that no single parameter dominates an eigenvector; the eigenvector’s mass is spread across many coordinates. This is a common property in large random matrices and is empirically plausible for well-trained neural networks, where gradient information is typically distributed across many parameters rather than concentrated in a few. Assumption F.3 (Small-Step Regime) . The per-step update \Delta satisfies \|\Delta\|=O(\sqrt{K}) , where K is the KL bound. Justification. This ensures that second-order Taylor expansions are accurate and higher-order terms are negligible. In practice, the clipping mechanism in PPO/GRPO and the on-policy sampling procedure naturally enforce small per-step policy changes. F.2 Proof of Proposition 5.1 (Low-Dimensional Policy Sensitivity) Proposition F.4 (Restated) . Under Assumptions F.1 and F.3 , for any update \Delta satisfying the per-step KL constraint D_{\mathrm{KL}}(\pi_{\theta+\Delta}\|\pi_{\theta})\leq K , the second-order change in the policy depends only on the projection of \Delta onto the subspace U=\mathrm{span}\{v_{1},\ldots,v_{r}\} . Components orthogonal to U have negligible impact on the policy. Proof. We proceed in five steps. Step 1: KL Constraint in Quadratic Form. Using a second-order Taylor expansion and the definition of the Fisher matrix, the KL divergence can be approximated as: D_{\mathrm{KL}}(\pi_{\theta+\Delta}\|\pi_{\theta})=\frac{1}{2}\Delta^{\top}F(\theta)\Delta+O(\|\Delta\|^{3}). Under the small-step regime (Assumption F.3 ), the cubic term is negligible, so the constraint is essentially: \Delta^{\top}F(\theta)\Delta\leq 2K. ( \star ) Step 2: Decomposition of the Update. Decompose \Delta into two orthogonal components: \Delta=\Delta_{\parallel}+\Delta_{\perp}, where \Delta_{\parallel}\in U and \Delta_{\perp}\in U^{\perp} (the orthogonal complement, spanned by v_{r+1},\ldots,v_{d} ). Substituting into ( \star ): \Delta^{\top}F(\theta)\Delta=\Delta_{\parallel}^{\top}F(\theta)\Delta_{\parallel}+\Delta_{\perp}^{\top}F(\theta)\Delta_{\perp}\leq 2K. Step 3: Bounding the Contribution of \Delta_{\perp} . Since \Delta_{\perp} lies in the span of the tail eigenvectors: \Delta_{\perp}^{\top}F(\theta)\Delta_{\perp}=\sum_{i=r+1}^{d}\lambda_{i}\langle\Delta_{\perp},v_{i}\rangle^{2}\leq\lambda_{r+1}\|\Delta_{\perp}\|^{2}, where \lambda_{r+1} is the largest eigenvalue in the tail. By Assumption F.1 , \lambda_{r+1} is very small relative to the total variance. More precisely, let \Lambda_{\mathrm{tail}}=\sum_{i=r+1}^{d}\lambda_{i} . Then: \lambda_{r+1}\leq\frac{\Lambda_{\mathrm{tail}}}{d-r}\leq\frac{\epsilon\cdot\mathrm{Tr}(F(\theta))}{d-r}. Since \mathrm{Tr}(F(\theta)) is typically O(d) in neural networks, \lambda_{r+1}=O(\epsilon) . Therefore, even if \|\Delta_{\perp}\|^{2} is as large as O(K/\lambda_{\min}) (where \lambda_{\min} is the smallest eigenvalue), the product \lambda_{r+1}\|\Delta_{\perp}\|^{2} remains O(\epsilon K/\lambda_{\min}) . Given that \epsilon is small and \lambda_{\min} is not extremely small in practice, this term is negligible compared to the KL budget K . Step 4: Policy Change Depends Primarily on \Delta_{\parallel} . Consider the change in log-probability for a specific output y : \log\pi_{\theta+\Delta}(y|x)-\log\pi_{\theta}(y|x)=\Delta^{\top}g(y)+\frac{1}{2}\Delta^{\top}H(y)\Delta+O(\|\Delta\|^{3}), where g(y)=\nabla_{\theta}\log\pi_{\theta}(y|x) and H(y)=\nabla_{\theta}^{2}\log\pi_{\theta}(y|x) . The expected square of the linear term is \Delta^{\top}F(\theta)\Delta , which we have already bounded. The linear term decomposes as: \Delta^{\top}g(y)=\Delta_{\parallel}^{\top}g(y)+\Delta_{\perp}^{\top}g(y). The variance of the second term is: \mathbb{E}_{y\sim\pi_{\theta}}\bigl[(\Delta_{\perp}^{\top}g(y))^{2}\bigr]=\Delta_{\perp}^{\top}F(\theta)\Delta_{\perp}, which is negligible as argued above. Moreover, since g(y) lies in the span of the Fisher eigenvectors (by definition of F(\theta) ), the component \Delta_{\perp}^{\top}g(y) is only excited by tail eigenvectors, which have small eigenvalues and hence small typical magnitudes. Therefore, the change in log-probability—and thus the policy itself—is dominated by \Delta_{\parallel} . Step 5: Conclusion. To second order, the policy update depends only on \Delta_{\parallel} . The orthogonal component \Delta_{\perp} neither significantly affects the KL divergence nor the policy output. This establishes that the policy-relevant subspace is effectively low-dimensional. ∎ F.3 Proof of Proposition 5.2 (Sufficiency of Random Masks) Proposition F.5 (Restated) . Let S\subset\{1,\ldots,d\} be a random subset of indices of size k , chosen uniformly. Under Assumptions F.1 and F.2 , if k>r , then with high probability there exists an update \Delta_{S} supported on S (i.e., \Delta_{S,i}=0 for i\notin S ) such that: \|\Delta_{\parallel}-\Delta_{S}\|_{F}\leq\eta, where \|\cdot\|_{F} denotes the Fisher norm \|u\|_{F}=\sqrt{u^{\top}F(\theta)u} , and \eta is a small constant that decreases as k increases. Consequently, optimizing only over parameters in S can achieve policy improvement equivalent to full-parameter optimization within the KL-reachable region. Proof. We proceed in six steps. Step 1: Setup and Notation. Let V_{r}=[v_{1},\ldots,v_{r}]\in\mathbb{R}^{d\times r} be the matrix whose columns are the top r eigenvectors. Any vector in U can be written as V_{r}c for some coefficient vector c\in\mathbb{R}^{r} . Let P_{S} be the projection operator that zeros out coordinates not in S : (P_{S}u)_{i}=u_{i} if i\in S , and 0 otherwise. Step 2: Goal. We want to approximate a given \Delta_{\parallel}=V_{r}c by a vector \Delta_{S} supported on S . Equivalently, we want to find coefficients c^{\prime}\in\mathbb{R}^{r} such that \Delta_{S}=P_{S}(V_{r}c^{\prime}) is close to \Delta_{\parallel} in Fisher norm. Step 3: Delocalization and Random Masks. By Assumption F.2 , each eigenvector v_{i} has bounded infinity norm. This delocalization property implies that when we sample a random subset S of coordinates, the restricted vectors \tilde{v}_{i}=P_{S}(v_{i}) are likely to preserve the geometric structure of the original subspace. Formally, consider the matrix \tilde{V}_{r}=P_{S}(V_{r})\in\mathbb{R}^{d\times r} (which has zeros in rows outside S ). The product \tilde{V}_{r}^{\top}F(\theta)\tilde{V}_{r} measures how well the restricted eigenvectors capture the Fisher metric on the subspace. Because the eigenvectors are delocalized, each row of V_{r} has small norm. A standard concentration argument (see Lemma F.6 below) shows that with high probability, \left\|\frac{d}{k}\tilde{V}_{r}^{\top}F(\theta)\tilde{V}_{r}-V_{r}^{\top}F(\theta)V_{r}\right\|_{2}\leq\delta, where \delta decreases with k . Since V_{r}^{\top}F(\theta)V_{r}=\mathrm{diag}(\lambda_{1},\ldots,\lambda_{r}) is diagonal with large entries, the restricted Gram matrix is also well-conditioned when k is sufficiently larger than r . Step 4: Existence of a Good Approximation. Because the restricted Gram matrix is well-conditioned, the linear map c^{\prime}\mapsto P_{S}(V_{r}c^{\prime}) is injective on \mathbb{R}^{r} . Thus, for any desired \Delta_{\parallel}=V_{r}c , we can solve the least-squares problem: \min_{c^{\prime}\in\mathbb{R}^{r}}\|V_{r}c-P_{S}(V_{r}c^{\prime})\|_{F}. The solution satisfies: \|V_{r}c-P_{S}(V_{r}c^{\prime})\|_{F}\leq\kappa\cdot\|V_{r}c\|_{F}, where \kappa depends on the condition number of the restricted Gram matrix. As k increases, \kappa\to 0 . Setting \Delta_{S}=P_{S}(V_{r}c^{\prime}) yields the required approximation. Step 5: Connection to Policy Improvement. Since the policy change depends continuously on the update (as shown in Proposition 5.1 ), and since the Fisher norm dominates the change in log-probabilities, a small error \eta in Fisher norm translates to a small error in policy improvement. Therefore, optimizing over the mask S can achieve essentially the same policy improvement as full-parameter optimization, provided k>r . Step 6: Threshold Effect. The quality of approximation undergoes a phase transition: when kr , the error decreases as k increases. This explains the empirical observation that random masks work well above a certain sparsity threshold. ∎ F.4 Technical Lemmas Lemma F.6 (Concentration of Restricted Gram Matrix) . Let V_{r}\in\mathbb{R}^{d\times r} have orthonormal columns with \|v_{i}\|_{\infty}\leq\mu/\sqrt{d} . Let S be a random subset of size k . Then with probability at least 1-\delta , \left\|\frac{d}{k}\tilde{V}_{r}^{\top}\tilde{V}_{r}-I_{r}\right\|_{2}\leq C\mu^{2}r\sqrt{\frac{\log(1/\delta)}{k}}, where \tilde{V}_{r}=P_{S}(V_{r}) and C is an absolute constant. Proof Sketch. This follows from matrix Bernstein inequalities applied to the sum of independent random matrices X_{j}=\frac{d}{k}\mathbf{1}_{j\in S}(V_{r})_{j:}^{\top}(V_{r})_{j:} , where (V_{r})_{j:} is the j -th row of V_{r} . The delocalization assumption ensures each term has bounded norm \|X_{j}\|_{2}\leq\frac{d}{k}\cdot\frac{\mu^{2}}{d}=\frac{\mu^{2}}{k} . Applying matrix Bernstein with variance proxy \sigma^{2}=O(r/k) yields the stated bound. ∎ Remark on the Concentration Bound. The proof relies heavily on the delocalization assumption (Assumption F.2 ). While this is plausible for large neural networks, it is difficult to verify rigorously. However, empirical studies of eigenvectors in trained networks often show diffuse weight distributions, supporting this assumption. Additionally, the concentration bound requires k=\Omega(r\log r) , which is consistent with our empirically observed sparsity threshold. F.5 Synthesis: Why Random Masks Work Propositions 5.1 and 5.2 together explain the empirical success of random sparse fine-tuning in RLVR: 1. KL constraints create a low-dimensional trust region. The per-step KL bound restricts updates to a region defined by the Fisher matrix’s quadratic form. 2. The Fisher matrix has low effective rank. Due to Assumption F.1 , the policy-relevant subspace is only r -dimensional, where r\ll d . 3. Delocalization enables random sampling. Due to Assumption F.2 , a random mask of size k>r captures this subspace with high probability. 4. Multiple masks succeed. The combinatorial number of ways to choose k parameters from d —each capable of spanning the same r -dimensional subspace—directly yields the Multiple Ticket Hypothesis. F.6 Connection to Neural Tangent Kernel Theory Our theoretical framework connects to Neural Tangent Kernel (NTK) theory. In the infinite-width limit, neural networks operate in a “lazy training” regime where the kernel remains approximately constant. While our setting involves finite-width networks with potentially evolving representations, the low effective rank of the Fisher matrix suggests a similar phenomenon: the policy-relevant directions are determined early and remain stable, allowing arbitrary parameter subsets to navigate this low-dimensional landscape. The delocalization assumption (Assumption F.2 ) is particularly natural in the NTK regime, where eigenvectors of the kernel matrix tend to be spread across many input dimensions rather than localized. This provides theoretical grounding for our empirical observation that random masks work across different model architectures and scales. 29