Pai : Fast, Accurate, and Full Benchmark Performance Projection with AI

Avery Johnson, Mohammad Majharul Islam, Riad Akram, Abdullah Muzahid · 2026

Read Pai : Fast, Accurate, and Full Benchmark Performance Projection with AI by Avery Johnson, Mohammad Majharul Islam, Riad Akram, Abdullah Muzahid (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-19330.txt.

Introduction

Johnson and co-authors forecast chip benchmark scores with a learned model instead of slow simulators. Cycle-level simulators take too long to run full suites and are hard to keep up. Their PAI reads a trace of program traits that do not depend on chip details. A layered LSTM model then predicts the speed metrics for the run. On the SPEC CPU 2017 suite, mean error in cycles per instruction is about 9 percent. The full suite takes under three minutes to project, far faster than prior methods.

Catalog record

CREATORAvery Johnson, Mohammad Majharul Islam, Riad Akram, Abdullah Muzahid
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE19501
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.19330

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Pai : Fast, Accurate, and Full Benchmark Performance Projection with AI Avery Johnson, Mohammad Majharul Islam, Riad Akram, Abdullah Muzahid Abstract The exponential increase in complex IPs within modern SoCs, driven by Moore’s Law, has created a pressing need for fast and accurate hardware-software power-performance analysis. Traditional performance simulators (such as cycle accurate simulators) are often too slow to simulate full benchmarks within a reasonable timeframe; require considerable effort for development, maintenance, and extensions; and are prone to errors, making pre-silicon performance projections and competitive analysis increasingly challenging. Prior attempts in addressing this challenge using machine learning fall short as they are either slow, inaccurate or unable to predict the performance of full benchmarks. To address these limitations, we present Pai , the first technique to accurately predict full benchmark performance without relying on detailed simulation or instruction-wise encoding. At the heart of Pai is a hierarchical Long Short Term Memory (LSTM)-based model that takes a trace of microarchitecture independent features from a program execution and predicts performance metrics. We present the detailed design, implementation and evaluation of Pai . Our initial experiments showed that Pai can achieve an average IPC prediction error of 9.35% for SPEC CPU 2017 benchmark suite while taking only 2 min 57 sec for the entire suite. This prediction error is comparable to prior state-of-the-art techniques while requiring 3 orders of magnitude less time. I Introduction Emergence of Internet-of-Things and smart devices are making System-on-Chips (SoCs) a popular design option for their size, cost, and performance efficiency. Modern SoCs are sophisticated, incorporating a wide range of Intellectual Property (IP) blocks such as CPUs, GPUs, memory subsystems, various hardware accelerators, communication interfaces, sensors, etc. Furthermore, the IPs themselves are becoming more and more complex. This necessitates any tool that can make designing SoCs faster and easier to do. Performance modeling is an integral part for designing the next generations SoCs. It is a way to determine what design choices are better and therefore, should be considered at different stages of the design cycle. However, the complexity of modern SoCs make it a challenging task to model or predict performance. Traditionally, performance modeling is done through an architectural performance simulator such as Z-sim [ 15 ] , Gem5 [ 3 ] , ChampSim [ 5 ] , etc. These simulators simulate varying degrees of architectural events in a cycle-by-cycle fashion, eventually leading to a set of desired metrics such as Instruction Per Cycle (IPC), cache misses, branch mispredictions, power consumption, etc. The simulators, although quite accurate, are extremely slow. They operate at Kilo Instructions per Second (KIPS) speed and can take from few hours to days or weeks to simulate a full benchmark. As an example, it takes 8 hours to simulate a single core for one second of execution in Zsim [ 15 ] . This exorbitant runtime puts a serious hurdle to the performance estimation step of the SoC design process. As every iteration of the design cycle needs to simulate many benchmarks to evaluate and eventually, improve the design, these simulators impose a significant development cost and delay. Furthermore, even state-of-the-art simulators aren’t perfect; with an average error of 9.7% and 13% respectively for Z-sim and Gem5 [ 11 ] . What is needed is a performance modeling/prediction technique that is both fast and accurate . So, we ask ourselves the following question: Can this problem be solved with emergent machine learning models? Given that machine learning models are fast at inference (compared to the time for cycle-accurate simulation), it’s no surprise that it has already been applied in earlier research [ 10 , 14 , 13 , 2 ] . Most recently, Li et al. [ 10 ] propose to combine two deep learning models - one to capture program representations by encoding individual instructions and one to capture microarachitectural specifications. The third model combines the output of those two models and predicts performance of a program. As the model needs to encode individual instructions, it’s practicality is limited by how many instructions can be processed before the model starts to exhibit poor prediction accuracy. TAO [ 14 ] uses a dataset of microarchitecture independent features from a detailed simulator. This dataset is used to train a multi-head attention network in order to predict IPC, branch mispredictions, etc. TAO requires a detailed simulator for the the dataset to begin with. Therefore, it is inherently limited by the speed and accuracy of detailed simulator. Munigala et al. [ 13 ] use a multilayer perceptron to predict the bandwidth and latency of a benchmark based on seven architectural parameters (core count, clock frequency, Last Level Cache (LLC) size, etc.). However, like TAO, the performance metrics are obtained from a detailed simulator and hence, it faces similar issues. Barboza et al. [ 2 ] use many performance counters across multiple benchmarks to train a classifier to predict performance bottlenecks. The goal is to find performance degrading components of a microarchitecture. Like earlier works, it also relies on detailed simulation and considers only a single architecture. Besides these techniques, there are earlier attempts to use machine learning for performance prediction [ 8 , 9 , 1 ] . However, they are program and/or architecture specific, and therefore, lacking generality. To overcome the limitations of prior work and propose a general approach capable of full benchmark performance prediction, we start by setting three design goals - (i) detailed simulation independence, (ii) microarchitecture independence, and (iii) full benchmark prediction capability . With these design goals in mind, we propose Pai . Pai is the first technique to accurately predict full benchmark performance for seen and unseen programs and architectures without relying on detailed simulation or instruction-wise encoding. At the heart of Pai is a deep learning model based on hierarchical LSTM [ 6 ] . The model takes a trace of microarchitecture independent features from a program execution. The features are collected either from real machines or instruction set emulators (such as Simics [ 12 ] that is orders of magnitude faster than detailed simulation). Pai augments those features with high level microarchitectural specifications and provides performance prediction of the full benchmark both for seen and unseen programs and architectures. We provide a detailed design of Pai ’s deep learning model as well as the approach to generate necessary dataset for training the model. Our initial experiments showed that Pai can achieve an average IPC prediction error of 9.35% for SPEC CPU 2017 benchmark suite while taking only 2 min 57 sec for the entire suite. This prediction error is comparable to prior state-of-the-art techniques [ 14 , 10 ] while requiring 3 orders of magnitude less time. II Main Idea: PAI II-A Overview of Pai At a high level, Pai works by augmenting the traditional design cycle with an additional deep learning-based predictor (as shown in Figure 1 ) that provides a first-order approximation of performance metrics. The intuition is that by approximating the performance potential of different design choices for seen and unseen programs, Pai enables hardware designers to disregard many of those choices and focus on only a few good ones. To achieve this goal, Pai works in two phases. In the training phase, a dataset

Read the complete text