SISA: A Scale-In Systolic Array for GEMM Acceleration
Luigi Altamura, Alessio Cicero, Mateo Vázquez Maceiras, Mohammad Ali Maleki, Pedro Trancoso · 2026
Read SISA: A Scale-In Systolic Array for GEMM Acceleration by Luigi Altamura, Alessio Cicero, Mateo Vázquez Maceiras, Mohammad Ali Maleki, Pedro Trancoso (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-29913.txt.
Introduction
Altamura and co-authors present SISA, a chip layout that speeds up matrix multiplication. Large language models lean on such operations, but their skewed matrices waste square systolic arrays. SISA splits the square array into horizontal rectangular slabs. Slabs can be scheduled on their own for small or skewed shapes. The full array still runs as one for large jobs. Tests show up to 8.52 times speedup. Energy-delay product falls by 93 percent.
Catalog record
| CREATOR | Luigi Altamura, Alessio Cicero, Mateo Vázquez Maceiras, Mohammad Ali Maleki, Pedro Trancoso |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29506 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.29913
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SISA: A Scale-In Systolic Array for GEMM Acceleration Luigi Altamura, Alessio Cicero, Mateo Vázquez Maceiras, Mohammad Ali Maleki, Pedro Trancoso Abstract The currently dominant AI/ML workloads, such as LLM , rely on the efficient execution of GEMM ( GEMM ) operations. Thus, most systems are equipped with dedicated matrix hardware accelerators based on square SA of PE . While this organization was effective for traditional Deep Neural Networks (DNNs), LLM introduce input-dependent and highly skewed matrices, leading to underutilized SA resources. To address this challenge, we propose SISA (Scale-In Systolic Array), a novel SA architecture that partitions the traditional square array into horizontal rectangular slabs. With minimal overhead, SISA exposes parallelism through independently scheduled slabs for efficient execution of small or skewed matrix shapes, while retaining full-array operation for large GEMM . SISA achieves up to 8.52 × \times speedup and 93% energy-delay-product (EDP) reduction for representative LLM compared to a state-of-the-art monolithic SA with the same number of PE . 1 Introduction In recent years, the use of AI-based applications has increased rapidly. LLM ( LLM ), which form the backbone of widely deployed interactive systems such as ChatGPT, are quite demanding on the computing systems. LLM inference consists of two distinct phases: prefill and decode , each with fundamentally different characteristics. During prefill, the model processes the entire input prompt using dense GEMM ( GEMM ), which are compute-bound. In contrast, decode generates tokens sequentially using GEMV ( GEMV ), making it predominantly memory-bandwidth bound. In addition to GEMV , the decode also repeatedly accesses the key–value (KV) cache, which increases in size at each decode step. These characteristics naturally map the phases to different architectural primitives. Prefill benefits from SA ( SA )-based accelerators that maximize throughput for large GEMM , whereas decode is better suited to VPU that efficiently handle GEMV . However, for LLM systems supporting multiple concurrent requests, batching may be exploited to increase arithmetic intensity and amortize memory traffic [ 1 , 2 ] . This leads to having GEMM operations in the decode phase as well, making a SA -based accelerator the primary architecture for LLM . As a result, many modern AI accelerators incorporate SA . The most well-known example is the Google TPU, which integrates a large square SA (e.g., 128×128 and 256×256 PE ) [ 11 ] that is highly efficient for executing the large GEMM common in LLM workloads. (a) (b) Figure 1 : (a) Prompt length derived from interactive chatbot requests and (b) alternative SA designs. Nevertheless, when matrices are highly-skewed or smaller than the SA size, a significant fraction of PE remain idle. Previous works have shown that such SA -matrix shape mismatches lead to considerable under-utilization [ 4 ] , and these mismatches are common in practice. Figure 1 a shows that real-world chatbot prompts are typically short (mean 41.8, median 12 tokens) [ 17 ] , leading to skewed GEMM s during the LLM prefill phase. Consequently, the LLM prefill phase often involves skewed matrices (e.g., (12,8192) for a 12-token prompt in Llama3.2-3B). In addition, the number of tokens per prompt varies across requests. When multi-batching is not possible or when latency-sensitive prefill is required, matrix shapes remain highly skewed, since increasing batching may degrade Quality-of-Service by increasing the TTFT ( TTFT ). When decode requests are batched to amortize memory access costs in a GEMM -based LLM decode phase, the resulting matrix shapes depend on the batch size, with small batches yielding highly skewed matrices. However, two factors prevent aggressively large batching (e.g., decode batch sizes beyond 32–64). First, larger batch sizes increase response time, so batching is kept moderate to balance QoS and efficiency. Second, the decode phase accesses the KV cache, which can become large depending on the request. Since each request maintains its own KV cache, batching requires multiple such structures to be stored simultaneously in memory, further limiting the batch size. In order to address these and other mismatch issues between SA and GEMM shapes of matrices, researchers have proposed different architectures that can adapt the arrangement of available PE to better match the shapes of the matrices [ 5 , 7 , 9 , 10 , 13 , 15 , 20 , 21 ] . In Figure 1 b we show the tradeoff between flexibility/efficiency and overhead/complexity. Monolithic SA accelerators serves as the baseline, whereas flexible accelerators such as the multi-pod SA offers more flexibility at the cost of increased buffer and interconnection network overheads. To address this challenge, we propose SISA, a new SA architecture that extends the traditional n\times n square design with a scale-in mechanism that partitions the array into horizontal smaller rectangular sub-arrays, referred to as slabs, capable of parallel, independently scheduled execution. This slab-level parallelism enables SISA to efficiently process highly skewed matrix shapes by increasing effective parallelism relative to a monolithic array. When full-array execution is advantageous, SISA seamlessly operates as a conventional systolic array, preserving peak GEMM performance while incurring only minimal additional on-chip buffering overhead. This design point is represented in Figure 1 b. The main contributions of this work are: • A novel SA architecture that operates either as a unified array or partitioned into slabs that better adapt to skewed matrix shapes. • A scheduling and tiling mechanism that exploits parallelism across slabs. • An instance of SISA with 128\times 128 PE and its evaluation using state-of-the-art (SOTA) LLM models. We have evaluated SISA with GEMM operations from LLM , and observed a speedup of up to 8.52\times and a reduction in energy-delay-product (EDP) up to 93% when compared to a TPU with the same amount of compute and memory resources. In the worst case, SISA achieves comparable performance to the TPU with only an 8.47% higher EDP. When compared to alternative SOTA flexible SA architectures, SISA performs better in most cases, achieving speedups up to 2.61\times while also exhibiting lower area overhead. 2 Related Work Table 1: Comparison of SAs across key design requirements. For relative requirements, like the area overhead, we take as reference an equivalent monolithic SA. Architecture Fine-Grained Array/Pods Reconfigurability Low Area Overhead Low Data Movement TPU [ 11 ] X ✓ ✓ Gemmini [ 6 ] X ✓ ✓ SOSA [ 21 ] ✓ X X SAVector [ 5 ] ✓ ✓ X Planaria [ 7 ] X ✓ ✓ FlexSA [ 13 ] ✓ X ✓ HALO [ 20 ] ✓ X ✓ DyNNamic [ 10 ] ✓ X ✓ SARA [ 15 ] ✓ X ✓ ReDas [ 9 ] ✓ X ✓ SISA (this work) ✓ ✓ ✓ To meet the computational requirements of LLM , SA appear as a fitting architecture to build ML accelerators. SA are architectures composed by a set of interconnected PE . Thanks to their regular structure and communication patterns, SA can offer high computational capabilities for GEMM while keeping energy consumption efficient compared to other architectures: in SA , results are fed directly to the next computational unit without intermediate storage, following a systolic wavefront [ 12 ] . However, while monolithic SA are a matching architecture for computing GEMM , multiple variations have been presented in recent works. These variations of SA aim