How to keep pushing ML accelerator performance? Know your rooflines!

Marian Verhelst, Luca Benini, and Naveen Verma · 2025

Read How to keep pushing ML accelerator performance? Know your rooflines! by Marian Verhelst, Luca Benini, and Naveen Verma (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-16346.txt.

Introduction

Marian Verhelst, Luca Benini, and Naveen Verma survey machine learning accelerator design in this 2025 paper. Models keep growing, and their energy cost for inference grows with them. Special chips aim to raise speed per area while cutting energy per task.

The authors offer an enhanced roofline model as a shared lens for the field. A roofline plot shows whether a design is capped by compute or by memory flow. Their version maps where real accelerators sit against those bounds. Examples from top recent designs show how compute and memory choices interact. The paper closes with open problems that could push accelerator performance further.

Catalog record

CREATORMarian Verhelst, Luca Benini, and Naveen Verma
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE84 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-16346

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

How to keep pushing ML accelerator performance? Know your rooflines! Marian Verhelst, Luca Benini, and Naveen Verma Abstract The rapidly growing importance of Machine Learning (ML) applications, coupled with their ever-increasing model size and inference energy footprint, has created a strong need for specialized ML hardware architectures. Numerous ML accelerators have been explored and implemented, primarily to increase task-level throughput per unit area and reduce task-level energy consumption. This paper surveys key trends toward these objectives for more efficient ML accelerators and provides a unifying framework to understand how compute and memory technologies/architectures interact to enhance system-level efficiency and performance. To achieve this, the paper introduces an enhanced version of the roofline model and applies it to ML accelerators as an effective tool for understanding where various execution regimes fall within roofline bounds and how to maximize performance and efficiency under the rooline. Key concepts are illustrated with examples from state-of-the-art designs, with a view towards open research opportunities to further advance accelerator performance. PubID: pubid: 0000–0000/00$00.00 © 2021 IEEE I Introduction and Motivation The size of machine learning models has been growing rapidly across the last decade, as illustrated in Figure 1 . This growth led to an explosion in the required number of operations per ML training or inference tasks, as well as growing memory footprints. The growth rate in complexity of these models substantially outpaces Moore’s law (Figure 1 ). As a result, tracking the requirements in energy efficiency and throughput requires agility in design innovation and clarity around design trade-offs. This sustained innovation pressure has led to a myriad of dedicated hardware accelerators, exploiting specialized efficiency-boosting techniques, tailored for the ML domain. This push towards specialized architectures is challenged by a growing diversity in workloads. The rapid evolution of ML models, in terms of their topologies (layer dimensions, connectivity), precision requirements, modalities and embedding/de-embedding approaches, etc., require specialization yet without giving up on programmability, necessitating careful balance between flexibility and efficiency. Industrial and academic research have responded to the challenging needs, achieving impressive nearly 100 \times performance improvement every 24 months, maintained over the past 8 years as seen in Figure 1 . Despite the wide variety of ML accelerator architectures in the NPU’s (Neural Processing Units) of commercial products and published in the literature, a large part of their performance improvements can, however, be attributed to different variants of a common list of effective optimizations, including: a.) maximizing parallelism; b.) exploiting (spatial and temporal) data reuse; c.) quantization; d.) sparsity; and e.) near- or in-memory compute. This paper surveys these techniques, and illustrates their key insights and trade-offs with various examples from state-of-the-art (SOTA) designs. Fig. 1: Evolution of ML model size, GPU performance and Moore’s law across the last decade. Data from [ 1 ] and [ 2 ] . As these techniques are deployed in widely diverging platforms and at different scale, from tiny/extreme-edge devices, to edge processors, to cloud systems, it is difficult to assess them relative to each other. We require a unifying framework to assess their impact and to gain structured understanding of what sets efficiency and performance, as well as what the associated trade-offs are. This paper attempts to do this, leveraging and enhancing the roofline model for throughput and energy efficiency. The goal is to clearly visualize the impact of the different architectural techniques through the roofline representation, thereby enhancing contextual/comparative understanding. Finally, this will lead to insights into fundamental trade-off between the different techniques impacting performance, ultimately leading to open questions and directions for future research. This paper is organized as follows: Section II outlines the components contributing to the energy and latency of ML accelerator execution, leading to a visualization comprising two different rooflines. The most impactful efficiency enhancement techniques from recent research are subsequently surveyed and assessed in light of the roofline models in Section III , illustrated by several examples from recent SOTA. Next, section IV dives deeper into some fundamental trade-offs between the presented techniques, and how they have a conflicting impact on the performance and efficiency rooflines. This will finally lead to an outlook on the future, and directions for further research in Section V. II The roofline models for throughput and energy efficiency II-A Where does the energy and latency go? Fig. 2: Typical architecture of an ML accelerator. Note that the amount of PE’s in the datapath, the number of memory levels in the memory hierarchy and type of memory at each level (registers, SRAM, DRAM) can vary from system to system. Illustrative bandwidth and energy numbers are from a 22 nm technology. Almost all ML accelerators, regardless of whether standalone units, data-path extensions to a processor, or tensor cores integrate in a GPU, can be characterized by a common baseline architecture, depicted in Figure 2 . A regular array of processing elements are each capable of one or more multiply-accumulate (MAC) operations. The data for these operations are fed from an on-chip memory system, typically consisting of a hierarchy of registers and one or more levels of SRAM scratch pads. Additionally, the memory system typically couples to off-chip memory and/or storage subsystems. To optimize ML accelerators for throughput and energy efficiency, one has to understand where the energy and latency are spent in such systems. While one could focus on the efficiency of the MAC operation itself, this often only contributes to a portion of the total system energy and latency. Specifically, the energy per inference task E_{task} and latency per inference task L_{task} are heavily impacted by the necessary data movement operations to bring the required data on chip, and to move the data between the different on-chip memory levels. For a given number of bytes N_{Li} that must be accessed in each level Li of the memory hierarchy in order to perform a given number of multiply and/or accumulate operations N_{op} per inference, the total energy per inference can be computed as: E_{task}=N_{op}.E_{op}+\sum_{i}N_{Li}.E_{Li} (1) with E_{Li} being the energy per byte of memory access at memory level Li , and E_{op} being the energy per multiply or accumulate operation. Latency cannot be derived with a similar sum of the latency impact of the different memory and compute components: As ML accelerators are typically designed to perform compute and data-movement operations in parallel, compute operations can be hidden behind memory transfers or vice versa. As a result, the total latency per inference task is typically obtained by: L_{task}=\frac{1}{f_{clk}}.max{(\frac{N_{Ln}}{B_{Ln}},...,\frac{N_{L1}}{B_{L1}},\frac{N_{op}}{A_{op}})} (2) with B_{Li} being the bandwidth (bytes/cycle) of memory level Li , A_{op} being the amount of arithmetic operators physically on the chip, and f_{clk} being the chip’s operating frequency. To determine which contribution dominates the total energy and latency of an ML accelerator, one can fill in energy and memory bandwidth values of a target technology, as given in Figure 2 for a 22 nm CMOS. Doing so, it quickly becomes apparent that in both the energy and latency models, the ratios between the operation counts N_{op} 1 1 1 Throughout the complete paper, only addition and multiply operations are considered towards the number of operations (ops), where a MAC (multiply accumulate) counts for 2 operations. and the different memory access counts N_{Li} strongly determine the relative contribution of

Read the complete text