SCALE-Sim TPU: Validating and Extending SCALE-Sim for TPUs CCS: Computer systems organization Systolic architectures CCS: Computer systems organization Neural networks CCS: Software and its engineering Compilers
Jingtian Dang, Ritik Raj, Changhai Man, Jianming Tong, Tushar Krishna · 2026
Read SCALE-Sim TPU: Validating and Extending SCALE-Sim for TPUs CCS: Computer systems organization Systolic architectures CCS: Computer systems organization Neural networks CCS: Software and its engineering Compilers by Jingtian Dang, Ritik Raj, Changhai Man, Jianming Tong, Tushar Krishna (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-22535.txt.
Introduction
Dang and co-authors validate and extend SCALE-Sim, a cycle-level simulator for TPU-style accelerators. Simulators often drift from real hardware, so the team checks its core matrix model against a Google TPU v4. Simulated cycles track measured latency closely enough for a simple conversion. They add small learned models for non-matrix operations, cutting end-to-end timing errors sharply. A new frontend based on StableHLO lets workloads flow in straight from JAX and PyTorch. The result is a more faithful tool for whole-model timing studies.
Catalog record
| CREATOR | Jingtian Dang, Ritik Raj, Changhai Man, Jianming Tong, Tushar Krishna |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29192 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.22535
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SCALE-Sim TPU: Validating and Extending SCALE-Sim for TPUs CCS: Computer systems organization Systolic architectures CCS: Computer systems organization Neural networks CCS: Software and its engineering Compilers Jingtian Dang, Ritik Raj, Changhai Man, Jianming Tong, Tushar Krishna Abstract . Cycle-accurate simulators are widely used to study systolic accelerators, yet their accuracy and usability are often limited by weak validation against real hardware and poor integration with modern ML compiler stacks. This paper presents SCALE-Sim TPU , a validated and extended version of SCALE-Sim v3 for TPU-style accelerators. Specifically, we make three contributions: (1) We validate SCALE-Sim’s systolic GEMM model against measurements on Google TPU v4 and show that simulated cycle counts exhibit a strong linear correlation with hardware latency, enabling a simple cycle-to-latency mapping. (2) We introduce lightweight learned latency models for non-systolic elementwise operations, achieving median relative errors below 3% using only tensor size and shape, substantially improving end-to-end latency estimation. (3) We integrate a StableHLO-based frontend that allows workloads from modern ML frameworks such as JAX and PyTorch to be simulated directly via a unified compiler IR. Together, these contributions improve the fidelity, coverage, and practicality of cycle-accurate simulation for whole-model performance analysis on TPUs. Figure 1 . Overview of the SCALE-Sim TPU Workflow and Contributions. ML programs written in JAX or PyTorch are compiled into StableHLO, which serves as a unified input interface. The StableHLO parser extracts operator metadata and routes systolic-array operations (e.g., GEMM and convolution) to SCALE-Sim v3, while non-systolic operations are handled by lightweight analytic or learned latency models. Circles denote data artifacts or files, and rectangles denote processing components. The green region represents the complete SCALE-Sim TPU toolchain. The blue dashed region highlights legacy components inherited from prior work (SCALE-Sim v3), while the orange dashed region highlights the components introduced or extended in this paper. 1. Introduction Systolic-array accelerators such as Google’s Tensor Processing Units (TPUs) are a dominant substrate for modern machine learning workloads. Cycle-accurate simulators are therefore widely used to analyze performance bottlenecks, guide architectural exploration, and support benchmarking. However, in practice, two key limitations reduce the effectiveness of existing systolic-array simulators. First, their predictive validity is often unclear due to limited validation against real, production-scale hardware. Second, most simulators focus primarily on systolic kernels (e.g., GEMM and convolution), while ignoring or oversimplifying non-systolic operations, and require bespoke operator descriptions that are poorly aligned with modern ML compilation pipelines. This paper addresses these gaps by validating and extending SCALE-Sim v3 ( 9 ) , a modular cycle-accurate simulator for systolic accelerators. Our goal is to enable accurate, whole-model latency estimation for TPU-style accelerators directly from modern compiler IR, while maintaining lightweight modeling complexity. Our contributions are: \bullet TPU-grounded validation of SCALE-Sim’s systolic array model. We perform a TPU-grounded validation of SCALE-Sim’s systolic-array performance model by comparing simulated cycle counts against measured kernel latencies on Google TPU v4 across a wide range of GEMM shapes. We show that a simple analytical cycle model, when combined with a linear cycle-to-time mapping, can effectively translate SCALE-Sim cycle counts into TPU execution time, while also highlighting regimes where this approximation becomes less accurate. \bullet Learnable latency modeling for elementwise operations. To better account for the contribution of non-systolic operators to end-to-end model latency, we extend SCALE-Sim with lightweight, learnable latency models for elementwise operations. Using histogram-based gradient boosting regression, we model elementwise operator latency as a function of tensor size and shape. The resulting models achieve high predictive accuracy on TPU hardware, with median relative errors below 3% for representative operators such as addition and ReLU, substantially improving whole-model latency estimation beyond systolic kernels alone. \bullet StableHLO-based, framework-agnostic user interface for accelerator simulation. We introduce a StableHLO-based frontend that enables framework-agnostic simulation from modern ML stacks, including JAX and PyTorch. The frontend parses StableHLO to extract operator metadata and identifies operations supported by SCALE-Sim TPU, eliminating ad-hoc configuration formats and decoupling the simulator from framework-specific IRs. Supported systolic operations are routed to validated analytical models, while supported non-systolic operations are handled by learned latency models, enabling end-to-end latency estimation for the covered subset of compiler-emitted programs. Together, these contributions improve the fidelity , coverage , and usability of cycle-accurate systolic-array simulation. SCALE-Sim TPU provides a practical toolchain for evaluating real ML workloads on TPUs, grounded in hardware measurements and compatible with modern compiler infrastructures. Figure 1 summarizes the end-to-end workflow and highlights the components introduced in this paper. 2. Background and Motivation 2.1. SCALE-Sim V3 SCALE-Sim v3 ( 9 ) is a modular, cycle-accurate simulator designed for performance analysis of systolic-array–based accelerators. It extends earlier versions of SCALE-Sim by supporting multi-core architectures with spatio-temporal partitioning, sparse matrix multiplication, detailed DRAM modeling via Ramulator, on-chip data layout effects, and energy estimation through Accelergy. These enhancements allow SCALE-Sim v3 to capture interactions between compute, memory, and data movement more faithfully, enabling detailed exploration of architectural trade-offs across latency, bandwidth, and energy for modern AI workloads. 2.2. TPU v4 TPU v4 ( 3 ) is Google’s fourth-generation Tensor Processing Unit, designed to accelerate large-scale machine learning workloads using systolic-array–based matrix multiplication. It features significantly increased compute density, higher-bandwidth on-chip and off-chip memory, and improved interconnects compared to earlier TPU generations, enabling efficient execution of large GEMM- and convolution-dominated workloads. TPU v4 serves as a production-scale accelerator platform and provides a representative target for validating cycle-level performance models against real hardware behavior. 2.3. Motivation Why validate against real hardware? While simulator-based studies are common, many systolic-array simulators are rarely validated against production accelerators. To ground SCALE-Sim’s modeling assumptions, we use state-of-the-art systolic-array accelerator Google TPU as a representative platform and analyze when SCALE-Sim correctly reflects performance behavior. Why model elementwise operations? End-to-end ML model latency is not determined solely by systolic-array kernels such as GEMM and convolution. Modern ML workloads contain a large number of non-GEMM operators, including elementwise additions, activations, normalization, and other memory-bound operations, whose cumulative latency can be significant. Recent workload characterization shows that non-GEMM operations can account for 11.3%–73.6% of total inference time across models and platforms, and still contribute 15%–48% of end-to-end latency even after operator fusion ( 4 ) . As GEMM and convolution are increasingly accelerated by specialized hardware, these non-systolic operators become a dominant performance bottleneck. Prior work on inference latency prediction therefore emphasizes the need for operation-level modeling, including elementwise operators, to achieve accurate end-to-end latency estimation across diverse hardware and software stacks ( 6 ) . Consequently, to enable faithful whole-model latency estimation, it is necessary to model non-systolic elementwise operations alongside GEMM and convolution. Why StableHLO? Modern ML workloads are authored in frameworks such as JAX and PyTorch and increasingly flow through compiler IRs. StableHLO provides a convenient, unified representation for extracting operator shapes and attributes, avoiding per-operator hand-written configuration formats and simplifying integration ( 7 ) . 3. Related Works Prior work on accelerator performance modeling has largely focused on either cycle-level simulation or analytical modeling of systolic architectures. TimeLoop ( 8 ) offers a flexible analytical approach for mapping DNN layers to accelerator architectures; however, it relies on custom configuration files and validates results analytically (or against lower-level models) rather than through direct measurements on production hardware. COCOSSim ( 1 ) performs cycle-accurate simulation of both systolic