Hierarchical Reinforcement Learning for Neural Network Compression (HiReLC): Pruning and Quantization

Kamar Hibatallah Baghdadi, Kawther Guoual Belhamidi, Sara Belhadj, Aissa Boulmerka, Nadir Farhi · 2026

Read Hierarchical Reinforcement Learning for Neural Network Compression (HiReLC): Pruning and Quantization by Kamar Hibatallah Baghdadi, Kawther Guoual Belhamidi, Sara Belhadj, Aissa Boulmerka, Nadir Farhi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-26002.txt.

Introduction

Kamar Hibatallah Baghdadi leads a team that shrinks neural networks with reinforcement learning. Compression mixes pruning, which cuts weights, and quantization, which lowers numeric precision. Searching both together by hand is slow. HiReLC splits the search into two levels. Low-level agents tune each block of the network on its own. They pick, per kernel, the bitwidth, how much pruning keeps, the quantization type, and its granularity.

High-level agents coordinate the whole budget. Ensemble voting guides them, informed by Fisher Information sensitivity scores. Policy checks are expensive, so a surrogate model steps in. A small neural surrogate predicts outcomes during an active learning loop. Fine-tuning after compression still gives the final verdict. On vision transformer and convolutional benchmarks, storage shrinks roughly sixfold. One setting even gains accuracy after compression. Other settings lose between about half a percent and five percent, a trade the team judges practical.

Catalog record

CREATORKamar Hibatallah Baghdadi, Kawther Guoual Belhamidi, Sara Belhadj, Aissa Boulmerka, Nadir Farhi
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE85 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.26002

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Hierarchical Reinforcement Learning for Neural Network Compression (HiReLC): Pruning and Quantization Kamar Hibatallah Baghdadi, Kawther Guoual Belhamidi, Sara Belhadj, Aissa Boulmerka, Nadir Farhi Abstract We present HiReLC , a hierarchical ensemble-reinforcement learning framework for automated joint quantization and structured pruning of deep neural networks. The framework decomposes the compression search across two levels of abstraction: low-level agents (LLAs) operate independently per block, selecting per-kernel configurations over a multi-discrete action space spanning bitwidth, pruning keep-ratio, quantization type, and granularity, while high-level agents (HLAs) coordinate global budget allocation via ensemble voting guided by Fisher Information–based sensitivity estimates. To mitigate the computational cost of policy evaluation, an iterative active learning loop interleaves surrogate-guided RL optimization with post-compression fine-tuning, using a lightweight MLP surrogate to amortize expensive evaluations and a logit-MSE proxy during cold-start. The surrogate is used for reward shaping rather than as a replacement for final post-compression evaluation. The controller is architecture-agnostic by design, with a modular layer abstraction decoupling the RL environment from the underlying network topology. Experiments across Vision Transformer and CNN benchmarks demonstrate effective parameter-storage compression ratios of 5.99–6.72 × \times with a 3.83% gain in one setting and 0.55–5.62% accuracy drops elsewhere, supporting hierarchical policy decomposition and sensitivity-aware guidance as practical design choices for joint neural network compression. 1 Introduction Deep neural networks have achieved remarkable performance across visual recognition tasks, yet their deployment on resource-constrained platforms remains challenged by high parameter counts, memory footprints, and inference latencies. Vision Transformers (ViTs) exemplify this challenge in modern vision systems ( 6 ; 15 ) , and the efficiency gap is pervasive across architectural families, motivating the need for general-purpose compression frameworks. Existing compression approaches, whether based on static heuristics, one-shot mixed-precision assignment, or single-objective optimization, fail to capture cross-layer dependencies and lack adaptive refinement during fine-tuning. Model compression techniques spanning pruning and quantization have been explored broadly ( 13 ; 34 ; 4 ) , but reinforcement learning methods still face high variance in large action spaces, coarse granularity, and limited sensitivity-aware guidance. We note a related but orthogonal line of work that compresses reinforcement-learning agents themselves during training ( 9 ; 22 ; 32 ) ; HiReLC instead compresses pre-trained target networks as a post-training procedure. To address these limitations, we propose HiReLC , an architecture-agnostic hierarchical ensemble-reinforcement learning framework for joint quantization and structured pruning. Low-level agents (LLAs) operate per block over a multi-discrete action space, while high-level agents (HLAs) coordinate global budget allocation via Fisher Information–based sensitivity estimates. An iterative active learning loop interleaves surrogate-guided RL optimization with post-compression fine-tuning across an ensemble of heterogeneous RL policies. The main contributions of this work are summarized as follows: • We propose HiReLC , an architecture-agnostic hierarchical ensemble-RL framework for joint structured pruning and mixed-precision quantization. The framework decomposes the search across two tiers: a High-Level Agent ( HLA ) ensemble responsible for global budget allocation, and per-block Low-Level Agent ( LLA ) ensembles operating over a four-dimensional action space (bitwidth, keep-ratio, quantization type, granularity). Unlike single-level controllers targeting a single compression axis ( 13 ; 34 ) , HiReLC achieves 5.99–6.72 \times effective parameter-storage compression across four architectures, with 0.55–5.62% drops in loss cases and a 3.83% gain in one setting. • We integrate Fisher Information sensitivity scores ( 30 ) into both hierarchy levels: initializing LLA observations, contributing to the sensitivity penalty P_{\mathrm{sens}} , and driving HLA budget corrections, ensuring critical blocks receive lighter compression without additional search-time evaluations. Prior RL methods rely on magnitude-based heuristics. Our sensitivity-aware design propagates gradient-based block importance through both reward and action-correction, improving trade-offs across ViTs ( 6 ) and CNNs ( 12 ; 27 ) . • We introduce a surrogate-augmented active learning loop interleaving RL optimization with post-compression fine-tuning. A lightweight MLP surrogate amortizes expensive evaluations once a minimum number of configurations have been evaluated (see Section 3.6 ); a logit-MSE proxy covers the cold-start phase, while reported surrogate MAEs of 3.57–15.20% quantify when the learned proxy is reliable and when it remains noisy. Unlike Hessian-based methods ( 5 ) that incur high computation costs, our surrogate loop reduces the number of full model evaluations required during search, enabling a practical framework-level study within a small number of cycles. 2 Related Work Vision Transformers have achieved state-of-the-art performance across vision tasks, but their computational and memory demands pose significant deployment challenges ( 6 ; 15 ) . Early efficiency efforts focused on token reduction or dynamic computation skipping ( 24 ; 26 ) , though these do not address parameter-level compression. Compressing ViTs via pruning or quantization is particularly challenging due to the sensitivity of self-attention mechanisms ( 18 ) , motivating adaptive, structure-aware strategies. Post-training quantization often degrades Transformer performance at low bit-widths ( 4 ) , while quantization-aware training methods such as LSQ ( 7 ) better adapt to low-precision constraints. Structured pruning (removing attention heads ( 19 ) , feed-forward neurons ( 35 ) , or entire blocks) has become the dominant paradigm, as unstructured sparsity yields limited real-world speedups ( 8 ) . Reinforcement learning has been explored for automating compression decisions. AMC ( 13 ) and HAQ ( 34 ) formulate compression as sequential decision-making, using a DDPG agent to select layer-wise pruning ratios or mixed-precision assignments, demonstrating effectiveness on CNNs such as MobileNet and ResNet. However, both employ single-level controllers on a single compression axis. HAWQ-V2 ( 5 ) uses Hessian trace information to guide mixed-precision quantization, achieving strong compression ratios at minimal accuracy cost, though at significant computational expense. I-ViT ( 16 ) demonstrates robust integer-only quantization for ViTs via QAT but does not address structural pruning. DeepCompress-ViT ( 1 ) is orthogonal to structured pruning: it combines an encoder–decoder weight representation with low-bit quantization and unified compression training to target ViT storage and edge efficiency, rather than learning a structured channel-selection policy. In contrast, our work introduces a hierarchical ensemble-RL framework that jointly optimizes quantization and structured pruning, incorporating sensitivity-aware priors, ensemble-based agents, and an active learning compression loop to overcome the limitations of static policies and unstable training under aggressive compression targets. The term “ensemble” is used deliberately: HiReLC trains independent PPO/A2C policies and combines their proposals by voting, rather than modeling cooperative or competitive multi-agent interaction with communication, centralized critics, or value factorization as in MARL methods such as MADDPG and QMIX ( 17 ; 25 ) . Table 1 : Notation summary for recurring symbols and table abbreviations. Retained-size fractions are smaller when compression is stronger; compression ratios are larger when compression is stronger. Symbol / term Meaning \mathcal{M} , \boldsymbol{\theta} , \boldsymbol{\theta}_{0} , \boldsymbol{\theta}^{\prime} , \boldsymbol{\theta}^{*} Model, original, fine-tuned baseline, compressed, and best parameters \mathbf{c} , c_{i,k} Full and per-kernel compression configuration \rho_{i,k} Keep-ratio: fraction of output channels retained \tau_{i,k} Quantization type \mu_{i,k} Granularity \nu_{i,k} Kernel retained-size fraction (b_{i,k}/b_{0})\rho_{i,k} \bar{\nu}_{i} Block retained-size fractions \bar{\nu} Global retained-size fractions \bar{\nu}^{*} Equivalent retained-size target CR 1/\bar{\nu} \mathcal{R}(\mathbf{c}) Effective parameter-storage compression ratio R_{\mathrm{target}} Target CR lower bound \ell_{i} HLA compression tier \sigma_{i} decoded strategy p_{i} scalar strategy code \mathcal{B}_{i} HLA LayerBudget assigned to block B_{i} \Delta_{\mathrm{acc}} , \Delta\mathcal{A}^{(t)} Allowed and measured accuracy drops r_{t}^{\mathrm{LLA}} , r_{t}^{\mathrm{HLA}} Tier-specific RL rewards \mathcal{P} , \mathcal{S} , \mathcal{U} , \mathcal{T} , \mathcal{W} MDP tuple,

Read the complete text