ReGate: Enabling Power Gating in Neural Processing Units Conference: The 58th IEEE/ACM International Symposium on Microarchitecture; October 18–22, 2025; Seoul, Korea ISBN: 978-X-XXXX-XXXX-X/XX/XX

Yuqi Xue, Jian Huang · 2025

Read ReGate: Enabling Power Gating in Neural Processing Units Conference: The 58th IEEE/ACM International Symposium on Microarchitecture; October 18–22, 2025; Seoul, Korea ISBN: 978-X-XXXX-XXXX-X/XX/XX by Yuqi Xue, Jian Huang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-02536.txt.

Introduction

Yuqi Xue and Jian Huang worked on saving power in chips built for neural networks, in a 2025 paper. Such chips sit at the heart of data centers, so their energy use matters. The pair found that in several chip generations, idle power that leaks away made up a large share of all energy used.

Their system, ReGate, turns off parts of the chip when they sit idle, at a fine scale. Some parts follow set patterns and can be switched by hardware in step with the work. Others rest for long or varied spells, so they need either a small watcher circuit or control by software. In tests the system cut chip energy by about a sixth on average, with little loss of speed.

Catalog record

CREATORYuqi Xue, Jian Huang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE80 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-02536

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ReGate: Enabling Power Gating in Neural Processing Units Conference: The 58th IEEE/ACM International Symposium on Microarchitecture; October 18–22, 2025; Seoul, Korea ISBN: 978-X-XXXX-XXXX-X/XX/XX Yuqi Xue, Jian Huang Abstract . The energy efficiency of neural processing units (NPU) is playing a critical role in developing sustainable data centers. Our study with different generations of NPU chips reveals that 30%–72% of their energy consumption is contributed by static power dissipation, due to the lack of power management support in modern NPU chips. In this paper, we present ReGate, which enables fine-grained power-gating of each hardware component in NPU chips with hardware/software co-design. Unlike conventional power-gating techniques for generic processors, enabling power-gating in NPUs faces unique challenges due to the fundamental difference in hardware architecture and program execution model. To address these challenges, we carefully investigate the power-gating opportunities in each component of NPU chips and decide the best-fit power management scheme (i.e., hardware- vs. software-managed power gating). Specifically, for systolic arrays (SAs) that have deterministic execution patterns, ReGate enables cycle-level power gating at the granularity of processing elements (PEs) following the inherent dataflow execution in SAs. For inter-chip interconnect (ICI) and HBM controllers that have long idle intervals, ReGate employs a lightweight hardware-based idle-detection mechanism. For vector units and SRAM whose idle periods vary significantly depending on workload patterns, ReGate extends the NPU ISA and allows software like compilers to manage the power gating. With implementation on a production-level NPU simulator, we show that ReGate can reduce the energy consumption of NPU chips by up to 32.8% (15.5% on average), with negligible impact on AI workload performance. The hardware implementation of power-gating logic introduces less than 3.3% overhead in NPU chips. 1. Introduction Hardware accelerators for AI workloads, such as neural processing units (NPUs), are pervasive in modern cloud platforms ( Google 2022 ; AWS 2023 ; Kennedy 2024 ) . For instance, Google has produced seven generations of TPUs (a typical example of NPU chips) for its cloud in the past six years to accommodate the growing AI demands ( RUN:AI 2022 ; Jouppi et al. 2023 ; Robinson 2024 ) . This surge in AI has become the major source of the increasing energy consumption of data centers, leading to a significant increase in data center greenhouse gas emissions over the past five years ( reu 2025 ; tim 2025 ; wsj 2024 ; bas 2024 ; goo 2024 ) . The energy efficiency of NPU chips is playing a critical role in the sustainable development of data centers. Unfortunately, NPU chips today suffer from severe energy inefficiency. Taking the typical TPU architecture as an example, we conduct a characterization study of its energy efficiency across different generations of chips by running various ML workloads, including large language models (LLMs), recommendation models, and stable diffusion models. We report the static and dynamic energy consumption of core hardware components of the chips in \lx@sectionsign 3 , including systolic arrays (SAs), vector units (VUs), on-chip SRAM, HBM controller & PHY (physical layer), inter-chip interconnect (ICI) controller & PHY, and others. Our study reveals that despite the energy efficiency improvements of newer NPU generations, there still exists a significant energy waste. Given a typical 60% utilization (duty cycle) for the NPU chips ( Wu et al. 2022 ) and a datacenter power usage efficiency (PUE) of 1.1 ( Google 2025 ) , 17%–32% of energy is wasted due to chip idleness (e.g., while the chip is powered on but not running any jobs). When the chips are busy, 30%–72% of energy comes from static power dissipation, due to the lack of system and architectural support for flexible power management in NPU chips. As most machine learning (ML) workloads are either compute-, memory-, or network-bound, they cannot always fully utilize all components on an NPU chip. Those underutilized components dissipate a significant amount of static power, as the current NPU design cannot dynamically turn them off or put them into low-power mode (i.e., power gating). Since static power dissipation will be (relatively) higher as the feature size continues to shrink in the future, supporting flexible power management is highly desirable for a sustainable NPU chip design. A promising approach is to reduce their static power consumption by power-gating idle hardware components at fine granularity. However, given the following unique challenges and opportunities, enabling power-gating in NPU chips is not straightforward. First, the NPU architecture has unique components such as large systolic arrays. These large components will have a much longer power-gate and wake-up latency, rendering the conventional simple idle-detection or no-access policies (developed for generic CPU processors and GPU architectures) ( Abdel-Majeed et al. 2013 ; Flautner et al. 2002 ) less effective. The new components require new techniques for power gating. Second, the NPU architecture adopts an in-order pipeline without dynamic scheduling to simplify hardware design, and it relies on the compiler to exploit the instruction-level parallelism (ILP) of ML programs. This renders many sophisticated power-gating techniques on CPUs/GPUs infeasible, such as using branch prediction and cache hit/miss as hints to decide when to power on/off the ALUs and caches ( Hu et al. 2004 ; Sadrosadati et al. 2019 ; Ravi and Lipasti 2017 ) , and changing the instruction scheduling policy to create longer idle periods of ALUs for creating more power-gating opportunities ( Abdel-Majeed et al. 2013 ; Kumar et al. 2014 ; Aghilinasab et al. 2016 ) . Third, NPUs are domain-specific accelerators for ML programs, which are known to be deterministic and predictable. This creates unique opportunities for software to manage the power states of individual components on an NPU chip in an accurate manner. We present ReGate, a hardware-software co-design approach that enables fine-grained power gating of each component in an NPU chip. To identify the power-gating opportunities in NPU chips, we first analyze the temporal and spatial utilization patterns of each component of an NPU chip, based on which we decide its best-fit power management scheme (i.e., hardware- vs. software-managed power gating). We list our key insights and contributions as follows. Hardware-managed power gating is the most effective approach for the SA, the ICI, and the HBM controllers. The SA can be both temporally and spatially underutilized, and a naïve idle-detection mechanism that controls the entire SA is insufficient for exploiting the idleness of the processing elements (PEs) in the SA. And, we also cannot simply add an idle-detection state machine to each PE due to high hardware cost. Our key insight is that the SA has a deterministic systolic dataflow pattern across its PEs. We can follow the dataflow to propagate the power-on/off signal through the PEs in the same way as the input vector propagates. This enables cycle-level power gating for each PE with minimal hardware overhead. The ICI is only active in collective operators (e.g., AllReduce, AllToAll, ReduceScatter, and AllGather), and it can be completely power-gated in other operators that do not perform inter-chip communications. As an operator is typically long enough (at least a few \mu s), it suffices to employ an idle-detection mechanism for power gating the ICI controller & PHY. Similarly, HBM is usually accessed at large granularity. For ML workloads that favor a large tile size for improved arithmetic

Read the complete text