RNM-TD3: N:M Semi-structured Sparse Reinforcement Learning From Scratch

Isam Vrce 1 , Andreas Kassler 1 and Gökçe Aydos 2 1 Deggendorf Institute of Technology, Dieter-Görlitz-Platz 1, Deggendorf, Germany 2 Department of Engineering Technology, Technical University of Denmark, Ballerup, Denmark { isam.vrce, andreas.kassler } @th-deg.de, [email protected] · 2026

Read RNM-TD3: N:M Semi-structured Sparse Reinforcement Learning From Scratch by Isam Vrce 1 , Andreas Kassler 1 and Gökçe Aydos 2 1 Deggendorf Institute of Technology, Dieter-Görlitz-Platz 1, Deggendorf, Germany 2 Department of Engineering Technology, Technical University of Denmark, Ballerup, Denmark { isam.vrce, andreas.kassler } @th-deg.de, [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-14578.txt.

Introduction

Isam Vrce and two co-authors study a 2026 arXiv sparsity method for deep reinforcement learning. Thinness squeezes deep nets with little score loss, and reward-learning nets with just 5 percent of old weights can still train nearly as well as dense forms. Yet most methods use messy fine-grained thinness, capping hardware speed-up from ragged compute patterns.

Ordered coarse-grained thinness enables speed-up but usually harms scores and tangles pruning. The paper offers, to its lore, the first study of N:M ordered thinness in reward learning, weighing squeeze, scores, and hardware thrift. Row-wise N:M thinness is forced through training for all nets in off-policy TD3, keeping fit with accelerators backing N:M sparse matrix acts. Tests on steady-control benchmarks show the thin agent beating its dense twin at 50 to 75 percent thinness.

Catalog record

CREATORIsam Vrce 1 , Andreas Kassler 1 and Gökçe Aydos 2 1 Deggendorf Institute of Technology, Dieter-Görlitz-Platz 1, Deggendorf, Germany 2 Department of Engineering Technology, Technical University of Denmark, Ballerup, Denmark { isam.vrce, andreas.kassler } @th-deg.de, [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-14578

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RNM-TD3: N:M Semi-structured Sparse Reinforcement Learning From Scratch Isam Vrce 1 , Andreas Kassler 1 and Gökçe Aydos 2 1 Deggendorf Institute of Technology, Dieter-Görlitz-Platz 1, Deggendorf, Germany 2 Department of Engineering Technology, Technical University of Denmark, Ballerup, Denmark { isam.vrce, andreas.kassler } @th-deg.de, [email protected] Abstract Sparsity is a well-studied technique for compressing deep neural networks (DNNs) without compromising performance. In deep reinforcement learning (DRL), neural networks with up to 5% of their original weights can still be trained with minimal performance loss compared to their dense counterparts. However, most existing methods rely on unstructured fine-grained sparsity, which limits hardware acceleration opportunities due to irregular computation patterns. Structured coarse-grained sparsity enables hardware acceleration, yet typically degrades performance and increases pruning complexity. In this work, we present, to the best of our knowledge, the first study on N : M N{:}M structured sparsity in RL, which balances compression, performance, and hardware efficiency. Our framework enforces row-wise N : M N{:}M sparsity throughout training for all networks in off-policy RL (TD3), maintaining compatibility with accelerators that support N : M N{:}M sparse matrix operations. Experiments on continuous-control benchmarks show that RNM-TD3, our N : M N{:}M sparse agent, outperforms its dense counterpart at 50%-75% sparsity (e.g., 2:4 and 1:4), achieving up to a 14% increase in performance at 2:4 sparsity on the Ant environment. RNM-TD3 remains competitive even at 87.5% sparsity (1:8), while enabling potential training speedups. 1 INTRODUCTION Figure 1: Illustration of a single cycle of the RNM-TD3 algorithm, demonstrated with an N:M sparsity pattern where N=2 and M=4 . The weight matrices depicted have dimensions R\times C (rows \times columns). (Left) A dense matrix W and an N:M sparse mask E are combined to create the sparse matrix \tilde{W} . This sparse matrix is trained for K steps, during which only the active weights update the underlying dense matrix. (Right) After K training steps, the weights in the original matrix W change in magnitude (only at the positions of active weights), resulting in a different mask E . The updated matrix W and the new mask E create a new sparse matrix \tilde{W} . This cycle repeats until the end of training. Modern deep learning (DL) largely relies on over-parameterized networks, whose performance scales with data, model size, and compute [ 12 , 14 ] . Yet, practical deployment demands low latency, low energy consumption, and small memory footprints. Model compression techniques, such as pruning/quantization [ 10 , 8 ] , distillation [ 13 ] , and sparse training [ 6 , 4 ] , address these challenges by reducing model size while preserving accuracy. In this work, we focus on sparse training in DRL. A neural network is considered sparse when a large fraction of its parameters are zero. Based on the distribution of zero-valued parameters, sparsity can be classified into three categories: Unstructured sparsity zeroes individual weights distributed across the network without any structure, often achieving high compression with minimal accuracy loss [ 39 , 23 , 7 ] . However, the irregular pattern limits hardware speedups. Structured sparsity removes entire channels, filters, blocks. This process shrinks tensors, which reduces the number of multiply and accumulate (MAC) operations. However, coarser granularity often yields lower accuracy than unstructured sparse networks at the same sparsity level [ 3 ] , with some mitigation via dependency-aware pruning at an additional computational cost [ 5 ] . Semi-structured N{:}M sparsity enforces a constraint whereby each contiguous group of M weights contains no more than N nonzero weights [ 38 ] . When supported in hardware (e.g., 2:4 on NVIDIA Ampere Tensor Cores), this regularity enables accelerated matrix multiplications during the forward pass [ 20 ] . However, accelerating backpropagation requires an N{:}M sparsity pattern along the columns of the original weight matrix W [ 37 , 35 ] . Beyond the 2:4 sparsity pattern, finer sparsity patterns such as 1:3 and 1:4 have also been explored [ 28 ] , enabling greater flexibility. In contrast to supervised learning, DRL involves training on non-stationary data induced by a continually evolving policy, using bootstrapping with delayed rewards and moving targets [ 27 , 21 ] . These characteristics make DRL training inherently unstable, and pruning can further exacerbate this instability [ 18 ] . Existing sparse DRL approaches either prune pre-trained dense models [ 36 ] or employ topology-evolution strategies that involve neuron pruning and regrowth [ 26 , 29 ] . While effective for compression, both strategies limit training-time acceleration, as post-training pruning does not exploit sparse matrices during training. In this work, we introduce an end-to-end N{:}M structured sparse DRL framework that maintains a hardware-aware N{:}M sparsity pattern throughout training for both actor and critic networks of the state-of-the-art DRL algorithm Twin Delayed Deep Deterministic Policy Gradient (TD3). We demonstrate that the period of mask updates correlates with Sparse Architecture Divergence (SAD) [ 38 ] , providing new insights into the dynamics of sparsity during DRL training. Results indicate that training an N{:}M sparse agent can match the performance of dense and unstructured baselines. We make the following contributions: • Framework for N{:}M -sparse DRL training. We maintain strict N{:}M constraints throughout training so that both actor and critic are compatible with hardware-aware sparse matrix-matrix multiplications. (e.g., 2:4 on NVIDIA Ampere Tensor Cores) [ 20 ] . • SAD mask dynamics study. We relate the period of mask update to the SAD between successive masks. We find that DRL favors less frequent mask updates compared to supervised learning settings. Moreover, we show that maintaining a stable, non-zero SAD, which is achieved naturally through training, is correlated with improved cumulative reward. • Empirical validation. We demonstrate that on MuJoCo continuous-control benchmarks, our RNM-TD3 agent outperforms dense baselines on average return and rivals unstructured sparse baselines, while offering theoretical acceleration on hardware that supports N{:}M sparse computation. The rest of the paper is structured as follows. Section 2 reviews related work. Section 3 details our methodology and introduces our sparse training strategy. We evaluate our approach in Section 4 and conclude the paper in Section 5. 2 RELATED WORK Early work on compressing neural networks in RL focused on knowledge distillation. A trained agent served as the source model, and its behavior was distilled into a compact student network, resulting in a model size reduction of up to 15 times with no performance loss [ 24 ] . Building on this, [ 25 ] introduced an unconstrained teacher-student paradigm, reporting a 42% performance gain while requiring \sim 10\times fewer training steps for the student. The first attempts to induce sparsity in DRL focused on post-training pruning: (i) train a dense model, (ii) remove small-magnitude weights, (iii) fine-tune the sparse model. This strategy closely mirrors iterative magnitude pruning techniques developed in deep supervised learning [ 11 ] , and was later adapted to DRL [ 18 ] . Works inspired by the lottery ticket hypothesis (LTH) [ 6 ] applied in RL show that sparse sub-networks, when trained from scratch, can match or even surpass their dense counterparts [ 36 , 32 ] . In offline RL, single-shot pruning methods (i.e., sparsity is defined at initialization and remains fixed throughout training) such as SNIP [

Read the complete text