Benchmarking Ultra-Low-Power μ \mu NPUs

Josh Millar, Yushan Huang, Sarab Sethi, Hamed Haddadi, Anil Madhavapeddy · 2025

Read Benchmarking Ultra-Low-Power μ \mu NPUs by Josh Millar, Yushan Huang, Sarab Sethi, Hamed Haddadi, Anil Madhavapeddy (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-22567.txt.

Introduction

Josh Millar and four co-authors present the first head-to-head benchmarks of micro neural processing units, tiny chips that run neural networks on very little power, in a 2025 paper. Running AI on the device itself cuts delay, protects privacy, and lowers cost compared with the cloud. Several commercial chips now target this niche, but independent comparisons were missing.

The team built and released an open compilation pipeline so the same trained models could run fairly across very different hardware. Their tests confirmed some expected trends but also exposed surprises: some chips scaled oddly as models grew more complex, and spec sheets did not always predict real performance. The authors offer guidance for both chip and software designers.

Catalog record

CREATORJosh Millar, Yushan Huang, Sarab Sethi, Hamed Haddadi, Anil Madhavapeddy
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-22567

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Benchmarking Ultra-Low-Power μ \mu NPUs Josh Millar, Yushan Huang, Sarab Sethi, Hamed Haddadi, Anil Madhavapeddy Abstract . Efficient on-device neural network (NN) inference offers predictable latency, improved privacy and reliability, and lower operating costs for vendors than cloud-based inference. This has sparked recent development of microcontroller-scale NN accelerators, also known as neural processing units ( μ \mu NPUs), designed specifically for ultra-low-power applications. We present the first comparative evaluation of a number of commercially-available μ \mu NPUs, including the first independent benchmarks for multiple platforms. To ensure fairness, we develop and open-source a model compilation pipeline supporting consistent benchmarking of quantized models across diverse microcontroller hardware. Our resulting analysis uncovers both expected performance trends as well as surprising disparities between hardware specifications and actual performance, including certain μ \mu NPUs exhibiting unexpected scaling behaviors with model complexity. This work provides a foundation for ongoing evaluation of μ \mu NPU platforms, alongside offering practical insights for both hardware and software developers in this rapidly evolving space. Conference: The 31st Annual International Conference on Mobile Computing and Networking; November 4–8, 2025; Hong Kong, China The 31st Annual International Conference on Mobile Computing and Networking (ACM MOBICOM ’25), November 4–8, 2025, Hong Kong, China DOI: 10.1145/3680207.3765264 ISBN: 979-8-4007-1129-9/2025/11 CCS: Hardware Power and energy CCS: Computing methodologies Machine learning CCS: Computer systems organization Embedded and cyber-physical systems † † cc-license: by 1. INTRODUCTION Performing neural network (NN) inference on constrained devices has applications across numerous domains, including wearable health monitoring ( 1 ) , smart agriculture ( 2 ) , real-time earable audio processing ( 3 ) , and predictive maintenance ( 4 ) . Embedding inference also offers improved privacy over cloud-based alternatives, by eliminating the need to transmit sensitive data, alongside improved latency for time-critical applications, reduced operating costs for vendors, and improved reliability by removing dependence on network connectivity. Given their unique form factor and low power consumption, microcontrollers (MCUs) are widely used in resource-constrained environments. However, their performance is also often constrained by limitations in memory capacity, throughput, and compute. The computational demands of modern neural networks (NNs) have catalyzed the development of specialized hardware accelerators across the computing spectrum, from high- performance data centers to ultra-low-power and embedded devices. At the resource-constrained end of the spectrum, microcontroller-scale neural processing units ( \mu NPUs) have recently emerged, designed to operate within extremely tight power envelopes – in the milliwatt or sub-milliwatt range – while still providing latency sufficient to support real-time inference. These platforms represent a new class of accelerator, combining the power efficiency of MCUs with the cognitive capabilities previously exclusive to more powerful computing platforms. The core advantage of \mu NPUs stems from their ability to exploit the inherent parallelism of neural networks with dedicated multiply-accumulate (MAC) arrays alongside specialized memory structures for weight storage. This architectural specialization enables \mu NPUs to achieve orders of magnitude improvement in latency compared to general-purpose MCUs executing equivalent workloads. Despite the growing availability of \mu NPU platforms, the field lacks a standardized and comprehensive evaluation or benchmark suite. Existing benchmarks focus solely on Analog Devices’ MAX78000 ( 5 , 6 , 7 ) , lacking side-by-side comparison with other platforms. Naturally, hardware vendors themselves provide performance metrics, but these are usually based on proprietary evaluation frameworks, using disparate NN models, quantization strategies, and other varying optimizations. This heterogeneity across evaluation methods, and absence of independent verification of vendor-provided performance claims, creates uncertainty for hardware designers and embedded software developers in selecting the most suitable platform for their application’s constraints. The lack of standardized benchmarks also hampers research by obscuring the relationship between architectural design and inference performance. Given the rapid pace of development and increasing diversity of available \mu NPU platforms, establishing reliable comparative benchmarks has become an urgent need. To this end, we make the following contributions: • Side-by-Side Benchmark of \mu NPU Platforms: We conduct the first comparative evaluation of commercially-available \mu NPU platforms, enabling direct performance comparisons across diverse hardware architectures under consistent workloads and measurement conditions. • Independent Platform Benchmarks: We also provide the first fine-grained and independent performance benchmarks for several \mu NPU platforms that have not previously been subject to third-party evaluation, offering unbiased verification of vendor performance claims. • Open-Source Model Compilation Toolchain: We develop and release 1 1 1 https://github.com/j0shmillar/uNPU-Bench an open-source toolchain to support consistent and simplified transplanting of NN models across our \mu NPU platforms, reducing the engineering overhead associated with cross-platform evaluation. • Recommendations for Developers: Informed by our benchmark results, we provide actionable recommendations to developers regarding platform selection, key focus areas for model optimization, and trade-offs for various applications and constraints. In developing a unified compilation and benchmarking pipeline, we standardize model representations across the various \mu NPU platforms, enabling direct comparison of latency, memory, and energy performance. The evaluation also includes fine-grained analysis of the various stages of model execution, from NPU initialization and memory input/output overheads to CPU pre/post-processing – aspects that can significantly impact end-to-end performance but are often not addressed in technical evaluations. The resulting analysis uncovers both expected performance trends as well as surprising disparities between hardware specifications and actual performance, including certain \mu NPUs exhibiting unexpected scaling behaviors with increasing model complexity. We hope our findings provide valuable insights to both developers and hardware architects. 2. BACKGROUND & MOTIVATION 2.1. Constrained Neural Computing The shift from cloud-based to on-device neural computing has numerous advantages for real-time data processing, especially with increasing concerns regarding data privacy and security ( 8 ) . Unlike cloud-based solutions, local inference mitigates security risks by processing sensitive data locally, which is particularly advantageous in domains such as medical diagnostics and surveillance ( 9 , 10 ) . Additionally, local processing reduces end-to-end latency alongside operating costs for model vendors. However, traditional NN accelerators, such as GPUs and TPUs, are ill-suited to resource-constrained environments given their power consumption and large form factors ( 11 , 12 ) . MCUs are compact, low-power computing platforms, often reliant on a single CPU and shared memory bus ( 13 ) . While MCUs are commonly adopted for resource-constrained IoT applications ( 14 , 15 , 16 ) , they generally lack the computational resources for efficient NN inference. Specifically, the computational capability of typical MCUs is often limited to a few million MAC operations per second, far below the tens of billions MACs/s required for real-time NN inference. Their absence of dedicated hardware acceleration results in large latency overheads and elevated power consumption during NN processing. Limited SRAM and flash memory also often poses challenges for efficiently managing the large weight matrices required for NN inference. Given the various shortcomings of traditional MCUs, microcontroller-scale \mu NPUs are emerging as a response. These specialized NN accelerators offer dedicated neural processing hardware, providing higher throughput for NN workloads, meeting the stringent requirements of real-time NN inference ( 17 , 18 , 19 ) while maintaining low-power operation. Collectively, \mu NPUs position themselves as a key solution for real-time NN processing in low-power environments. 2.2. \mu NPU Hardware Design \mu NPU hardware design is optimized for efficient tensor operations via specialized MAC units and parallelizable memory hierarchies ( 20 , 21 ) . Fig.

Read the complete text