UnIT : Scalable Un structured I nference- T ime Pruning for MAC-efficient Neural Inference on MCUs DOI: XXXXXXX.XXXXXXX
Ashe Neth, Sawinder kaur, Mohammad Nur Hossain Khan, Subrata Biswas, Asif Salekin, Bashima Islam · 2025
Read UnIT : Scalable Un structured I nference- T ime Pruning for MAC-efficient Neural Inference on MCUs DOI: XXXXXXX.XXXXXXX by Ashe Neth, Sawinder kaur, Mohammad Nur Hossain Khan, Subrata Biswas, Asif Salekin, Bashima Islam (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-07885.txt.
Introduction
Ashe Neth and co-authors prune neural networks while they run, on the tiniest hardware. Their 2025 paper introduces UnIT, a method for microcontrollers that skips multiplications the current input does not need, instead of cutting the network down ahead of time.
Classic pruning happens in training and removes whole blocks, wasting fine-grained savings on simple chips. UnIT swaps most multiplications for cheap threshold checks, reuses those checks across links, and tunes pruning per layer. On a common microcontroller it cut multiply-accumulate work by up to eighty-two percent, ran up to eighty-four percent faster, and used far less energy, with small accuracy costs. Under shifted data it matched or beat retrained models while doing less work.
Catalog record
| CREATOR | Ashe Neth, Sawinder kaur, Mohammad Nur Hossain Khan, Subrata Biswas, Asif Salekin, Bashima Islam |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-07885
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
UnIT : Scalable Un structured I nference- T ime Pruning for MAC-efficient Neural Inference on MCUs DOI: XXXXXXX.XXXXXXX Ashe Neth, Sawinder kaur, Mohammad Nur Hossain Khan, Subrata Biswas, Asif Salekin, Bashima Islam Abstract . Existing pruning methods are typically applied during training or compile time and often rely on structured sparsity. While compatible with low-power microcontrollers (MCUs), structured pruning underutilizes the opportunity for fine-grained efficiency on devices without SIMD support or parallel compute. To address these limitations, we introduce UnIT ( Un structured I nference- T ime pruning), a lightweight method that dynamically identifies and skips unnecessary multiply-accumulate (MAC) operations during inference, guided by input-specific activation patterns. Unlike structured pruning, UnIT embraces irregular sparsity and does not require retraining or hardware specialization. It transforms pruning decisions into lightweight comparisons, replacing multiplications with threshold checks and approximated divisions. UnIT further optimizes compute by reusing threshold computations across multiple connections and applying layer- and group-specific pruning sensitivity. We present three fast, hardware-friendly division approximations tailored to the capabilities of common embedded platforms. Demonstrated on the MSP430 microcontroller, UnIT achieves 11.02% to 82.03% MAC reduction, 27.30% to 84.19% faster inference, and 27.33% to 84.38% lower energy consumption compared to training-time pruned models, while maintaining accuracy with 0.48-7%. Under domain shift, UnIT matches or exceeds the accuracy of retrained models while requiring significantly fewer MACs. These results establish unstructured inference-time pruning as a viable and practical solution for efficient, retraining-free deployment of deep neural networks on MCUs. 1. Introduction Deploying deep neural networks (DNNs) on low-power microcontroller units (MCUs) is essential for enabling edge intelligence in domains such as healthcare monitoring ( Diab and Rodriguez-Villegas 2022 ) , environmental sensing ( Gorlatova et al. 2010 ; Kansal and Srivastava 2003 ) , and industrial automation ( Thramboulidis et al. 2007 ) . These systems must operate under tight constraints on energy, memory, and compute. In batteryless deployments ( Islam and Nirjon 2019 ; Montanari et al. 2020 ; Farina et al. 2024 ; Bakar et al. 2022 ; Gobieski et al. 2019 ) , the compute budget may be limited to just a few operations per inference cycle. Reducing multiply-accumulate (MAC) operations is therefore critical, as multiplications can be orders of magnitude more expensive than additions. For instance, on the MSP430 MCU, a multiplication takes approximately 77 clock cycles ( MSP 2018 ) , while an addition takes only 6 ( MSP 2024 ) . Traditional methods for reducing inference cost include quantization ( Biswas et al. 2025 ; Gobieski et al. 2019 ) and pruning during training or compile time ( Shen et al. 2022 ; Anwar et al. 2017 ; Kwon et al. 2020 ; Bakar et al. 2022 ) . However, these techniques produce static computation graphs that cannot adapt to input-specific variation or energy fluctuations at runtime. While structured pruning is widely used in high-end accelerators due to its alignment with SIMD and thread-parallel execution models ( Yang et al. 2018 ; Cheng et al. 2017 ) , it removes entire filters or channels and thus fails to leverage the finer control opportunities offered by single-threaded MCUs with no SIMD support. In contrast, MCUs are more amenable to unstructured pruning, which retains or skips individual connections. Unlike structured approaches that operate at the granularity of full channels or layers, unstructured pruning provides fine-grained control over which MACs are executed. This can lead to better energy efficiency and accuracy trade-offs, particularly when operating under tight resource budgets. However, implementing unstructured pruning at inference time is challenging due to the overhead of evaluating pruning conditions for each connection and the lack of hardware support for irregular computation. In addition to hardware constraints, MCUs often operate in dynamic environments where inputs deviate from the training distribution due to user variability, environmental noise, or temporal drift. Pruning strategies designed during training struggle in such settings, as they are optimized for the source domain and lack the flexibility to adapt at inference. Inference-time pruning, by contrast, makes decisions based on the input seen at runtime, allowing the model to skip unnecessary computations while preserving critical ones for out-of-distribution inputs. Prior work on input-aware early exit ( Islam and Nirjon 2019 ; Farina et al. 2024 ; Montanari et al. 2020 ) shows that runtime adaptation improves robustness and efficiency on low-power MCUs. However, early exits operate at the layer level—terminating entire inference paths—which limits their granularity and can prematurely discard useful computations. To achieve finer control, we propose input-aware, unstructured pruning at the connection level, enabling the model to retain or remove individual MAC operations based on their runtime utility. This allows deeper layers to be retained when needed, without sacrificing energy efficiency. To address these challenges, we present UnIT , a novel unstructured inference-time pruning method tailored to low-power MCUs. Instead of pruning weights or activations, UnIT dynamically skips individual MAC operations by checking the significance of each activation-weight pair based on the current input. Each connection corresponds to a scalar MAC, and pruning decisions are made without computing the full multiplication. This is enabled by a key hardware observation: on many MCUs, including the MSP430FR5994, conditional branching requires only 2 to 4 clock cycles, while a multiplication takes approximately 77 ( MSP 2018 ) . This gap makes connection-level pruning both feasible and efficient. UnIT introduces a reuse-aware thresholding strategy that avoids computing the full MAC. Rather than multiplying the activation and weight and then comparing the result to a threshold, it pre-divides the layer-wise threshold by either the activation or the weight and compares the remaining term to this result. The choice of reuse direction is based on the layer type: input activations are reused in fully connected layers, so thresholds are computed relative to them; weights are reused in convolutional layers, so they become the basis for comparison. This strategy enables a single threshold division to be reused across multiple MACs, reducing computation and memory overhead without modifying the model architecture. Even with this optimization, division operations remain a bottleneck since they are nearly as expensive as multiplications on MCUs. Thus, UnIT introduces three lightweight, hardware-specific approximations for the division operation: bit shifting, binary tree search, and bit masking. Each technique is selected based on the numerical format and capabilities of the target device. These estimators allow UnIT to retain its fine-grained pruning capability while ensuring that runtime costs remain negligible, even on resource-constrained embedded platforms. UnIT is designed for resource-constrained MCUs and integrates seamlessly into existing model compression pipelines. It complements training-time techniques by adding a runtime pruning layer that further improves the trade-off between efficiency and accuracy. Its lightweight, general-purpose logic is especially effective in edge scenarios that require real-time, energy-efficient inference, including deployment on intermittently powered or battery-free devices. Unlike prior inference-time methods, UnIT introduces runtime adaptivity without modifying the model architecture or requiring retraining. This compatibility allows it to layer on top of quantized or training-time pruned models, delivering additional reductions in MAC operations and energy usage with minimal implementation overhead. Our contributions are as follows: • MAC-Free, Input-Aware Pruning at Inference Time: UnIT introduces a new inference-time pruning method that skips individual MAC operations based on the runtime significance of input-weight pairs. Unlike existing approaches that