Optimizing GEMM for Energy and Performance on Versal ACAP Architectures

Ilias Papalamprou, Dimosthenis Masouros, Ioannis Loudaros, Francky Catthoor, Dimitrios Soudris · 2025

Read Optimizing GEMM for Energy and Performance on Versal ACAP Architectures by Ilias Papalamprou, Dimosthenis Masouros, Ioannis Loudaros, Francky Catthoor, Dimitrios Soudris (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-06907.txt.

Introduction

Ilias Papalamprou and four co-authors optimize matrix multiplication, the workhorse of scientific computing and deep learning, for AMD's Versal ACAP chips. These chips mix AI engines, programmable logic, and processors, which suits edge computing but makes mapping the computation across them complex. Earlier work largely ignored the trade-off between speed and energy.

Their automated system searches for mappings tuned either for performance or for energy efficiency. Instead of relying on analytical models, it trains a machine learning model on roughly six thousand measured on-board tests to guide the search. On the Versal VCK190 board, the resulting designs improved throughput by about 1.23 times on average and energy efficiency by about 1.25 times over leading systems, with larger gains in the best cases.

Catalog record

CREATORIlias Papalamprou, Dimosthenis Masouros, Ioannis Loudaros, Francky Catthoor, Dimitrios Soudris
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-06907

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Optimizing GEMM for Energy and Performance on Versal ACAP Architectures Ilias Papalamprou, Dimosthenis Masouros, Ioannis Loudaros, Francky Catthoor, Dimitrios Soudris Abstract General Matrix Multiplication (GEMM) is a fundamental operation in many scientific workloads, signal processing and particularly deep learning. It is often a bottleneck for performance and energy efficiency, especially in edge environments with tight resource and power constraints. AMD’s Versal ACAP offers heterogeneous components (AIEs, PL, PS) that can address these challenges, but mapping GEMM across them is complex, with prior works largely overlooking energy-performance tradeoffs. In this paper, we propose an automated framework for Versal ACAP that generates GEMM mappings optimized for either performance or energy efficiency. Unlike prior analytical approaches, our method leverages a Machine Learning (ML) model, trained on ≈ \approx 6000 on-board experiments of different GEMM mappings, to guide Design Space Exploration, yielding more efficient designs. Evaluation on the Versal VCK190 shows geomean improvements of 1.23 × {\mkern-2.0mu\times\mkern-2.0mu} (up to 2.5 × {\mkern-2.0mu\times\mkern-2.0mu} ) in throughput and 1.25 × {\mkern-2.0mu\times\mkern-2.0mu} (up to 2.7 × {\mkern-2.0mu\times\mkern-2.0mu} ) in energy efficiency over state-of-the-art frameworks. I Introduction General Matrix Multiplication (GEMM) is a fundamental kernel in scientific computing, signal processing, and especially deep learning (DL), where it can account for up to 99\% of total FLOPs [ 15 ] , in classical Multilayer Perceptrons (MLPs) [ 24 ] and Large Language Models (LLMs) [ 3 ] . As the primary computational bottleneck, GEMM efficiency directly determines both performance and energy consumption. This challenge is further amplified in embedded systems deployed in edge computing environments [ 12 , 23 ] , where strict power budgets, limited cooling, and constrained on-chip resources prevent sustaining peak throughput at the cost of excessive power draw [ 29 , 11 , 19 ] . In these settings, energy efficiency (e.g., FLOPs per Watt) becomes the decisive metric, requiring co-optimization of performance and power to sustain high efficiency under resource and power constraints [ 25 ] . AMD’s Versal Adaptive Compute Acceleration Platform (ACAP) [ 9 ] is a compelling target for achieving both high-performance and energy efficient execution. It’s architecture integrates three distinct components: i) AI Engines (AIEs) with VLIW, SIMD and local scratchpad memory, ii) programmable logic (PL) for custom functions and flexible datapaths, and iii) embedded CPU as the processing system (PS) for orchestration. These components are interconnected via a Network-on-Chip (NoC), enabling fine-grained customization of compute-memory pipelines. This heterogeneity offers substantial opportunities for energy-efficient GEMM acceleration, but also in the same time it presents three main challenges: 1. Complex hardware mapping: Executing GEMM workloads across heterogeneous components (AIEs, PL and PS) produces a vast design space with thousands of possible mapping options ( > 6000 for typical GEMM operations), making manual exploration impractical. Automated frameworks are essential to perform design space exploration (DSE) and identify efficient mappings. 2. Restricted on-chip memory: Limited local memory in the AIEs and low DDR bandwidth further complicate workload mapping. Tiling strategies are often employed, which split computation into smaller tiles, improving data reuse and alleviating bandwidth pressure (see Sec. III-A ). Yet, finding optimal tiling factors is non-trivial, as they directly determine the data transferred to and from DDR and the number of AIEs that can be effectively utilized, thereby affecting performance and power consumption. 3. Unreliable power modeling: Diverse operations across heterogeneous components, such as vectorized instructions, NoC transfers and memory accesses in AIEs, PL and DDR, make system power consumption challenging to predict. Vendor-provided tools often diverge substantially from real hardware measurements [ 17 , 30 ] , hindering energy-aware optimization. Accurate power modeling is required to efficiently guide the DSE process. (a) Throughput and energy efficiency. (b) Power of best designs. Fig. 1 : Impact of tiling on GEMM performance and power. Several prior works explore GEMM acceleration on Versal ACAP. However, they focus on maximizing performance either through high throughput AIE kernels [ 36 ] , multiple units for diverse GEMM workloads [ 31 , 32 , 30 ] or by improving data reuse on the AIEs [ 26 , 27 , 22 ] . To guide DSE, they typically rely on analytical models to estimate performance and select tiling parameters, which in turn determine AIE allocation and buffer sizes. While effective for throughput optimization, this strategy introduces two key limitations. First, power is largely overlooked . An implicit assumption in these works is that maximizing throughput naturally yields optimal energy efficiency, but this is not always the case. Fig. 1(a) shows the throughput and energy efficiency of different tiling configurations for a GEMM workload. We observe that the highest-throughput design (purple cross) is 22.4 \% less energy-efficient than the most energy-efficient one (teal star). This is caused by higher AIE utilization, which results in a \approx 11 Watt increase in power consumption (Fig. 1b). Second, analytical models are prone to inaccuracies . While the DSE process can tolerate small estimation errors when only relative differences matter, large deviations risk overlooking optimal mappings. As shown in Fig. 1(a) , analytical-model based mappings (yellow square) can reduce throughput by 17% compared to the actual highest-throughput design. Moreover, such models also neglect configurations that are superior not only in performance but also in energy efficiency (gray circles). To address the above limitations, we introduce an automatic framework that optimizes GEMM mappings on Versal ACAP architectures. Given the GEMM dimensions and a target objective (throughput or energy efficiency) our framework performs DSE driven by a machine learning (ML) model. Trained on a custom dataset from thousands of on-board experiments, the model predicts latency, power, and resource utilization with high accuracy. This enables efficient exploration of the design space and identification of Pareto-optimal mappings along the energy-performance frontier, yielding implementations tuned for either high throughput or energy-efficient operation. To enhance robustness across unseen workloads, custom-crafted features are integrated into the ML model. To the best of our knowledge, this is the first work to optimize GEMM mappings for both performance and energy efficiency on Versal ACAP. The novel contributions of this work are as follows: • We create an open-source dataset of \approx 6000 different GEMM mappings with experiments on Versal VCK190 1 1 1 Will be released open source upon acceptance. . • We develop an ML-model enhanced with custom-crafted features, that predicts throughput, energy and allocated resources of GEMM designs, achieving 51 \% higher accuracy compared to analytical-based modeling approaches [ 33 ] . • We introduce an automatic framework that performs DSE to identify Pareto-optimal hardware mappings for a given workload, enabling configurations optimized for either throughput or energy efficiency. We evaluate our framework on diverse GEMM workloads and compare it against state-of-the-art approaches [ 31 , 33 ] on Versal ACAP and NVIDIA embedded GPUs. Our experimental results showcase geomean improvements of 1.23 \times and up to 2.52 \times in throughput, and geomean 1.25 \times and up to 2.69 \times in energy efficiency over the state-of-the-art. II Related Work TABLE I : Comparison with Related Work Prior Works Performance Modeling Considered Components Predicted Metrics † Energy-Perf. Tuning [ 6 ] ML-based AIE \mathcal{T} ✗ [ 31 , 32 , 36 , 33 , 8 , 34 , 30 ] Analytical AIE, PL, DDR \mathcal{T},\mathcal{R} ✗ [ 26 , 27 ,

Read the complete text