VEXP: A Low-Cost RISC-V ISA Extension for Accelerated Softmax Computation in Transformers This work was supported by the NeuroSoC project, funded under the European Union’s Horizon Europe research and innovation programme (Grant Agreement No. 101070634).

Run Wang 1 , Gamze Islamoglu 1 , Andrea Belano 2 , Viviane Potocnik 1, Francesco Conti 2 , Angelo Garofalo 2 , Luca Benini 1 2 · 2025

Read VEXP: A Low-Cost RISC-V ISA Extension for Accelerated Softmax Computation in Transformers This work was supported by the NeuroSoC project, funded under the European Union’s Horizon Europe research and innovation programme (Grant Agreement No. 101070634). by Run Wang 1 , Gamze Islamoglu 1 , Andrea Belano 2 , Viviane Potocnik 1, Francesco Conti 2 , Angelo Garofalo 2 , Luca Benini 1 2 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-11227.txt.

Introduction

Run Wang and six co-authors add a tiny hardware block to RISC-V chips that makes the softmax step of transformers far cheaper, in this 2025 arXiv paper. As matrix math got faster, softmax became the bottleneck, and its exponentiation is the costly core. Their extension, VEXP, approximates that step inside the floating-point unit, taking about one percent extra chip area.

With tuned software kernels, softmax latency fell more than a hundredfold and its energy use by over seventy times against the baseline cluster. A key attention kernel sped up eightfold. Whole-model runs, including GPT-2, GPT-3, and a vision transformer, cut latency up to about sixfold with negligible accuracy loss and no retraining.

Catalog record

CREATORRun Wang 1 , Gamze Islamoglu 1 , Andrea Belano 2 , Viviane Potocnik 1, Francesco Conti 2 , Angelo Garofalo 2 , Luca Benini 1 2
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-11227

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

VEXP: A Low-Cost RISC-V ISA Extension for Accelerated Softmax Computation in Transformers This work was supported by the NeuroSoC project, funded under the European Union’s Horizon Europe research and innovation programme (Grant Agreement No. 101070634). Run Wang 1 , Gamze Islamoglu 1 , Andrea Belano 2 , Viviane Potocnik 1, Francesco Conti 2 , Angelo Garofalo 2 , Luca Benini 1 2 Abstract While Transformers are dominated by Floating-Point (FP) Matrix-Multiplications, their aggressive acceleration through dedicated hardware or many-core programmable systems has shifted the performance bottleneck to non-linear functions like Softmax. Accelerating Softmax is challenging due to its non-pointwise, non-linear nature, with exponentiation as the most demanding step. To address this, we design a custom arithmetic block for Bfloat16 exponentiation leveraging a novel approximation algorithm based on Schraudolph’s method, and we integrate it into the Floating-Point Unit (FPU) of the RISC-V cores [ 1 ] of a compute cluster, through custom Instruction Set Architecture (ISA) extensions, with a negligible area overhead of 1 % 1\text{\,}\mathrm{\%} . By optimizing the software kernels to leverage the extension, we execute Softmax with 162.7 × \times less latency and 74.3 × \times less energy compared to the baseline cluster, achieving an 8.2 × \times performance improvement and 4.1 × \times higher energy efficiency for the FlashAttention-2 kernel in GPT-2 configuration. Moreover, the proposed approach enables a multi-cluster system to efficiently execute end-to-end inference of pre-trained Transformer models, such as GPT-2, GPT-3 and ViT, achieving up to 5.8 × \times and 3.6 × \times reduction in latency and energy consumption, respectively, without requiring re-training and with negligible accuracy loss. I Introduction Transformer-based models such as the GPT family [ 2 ] and the LLaMa family [ 3 ] , have emerged as a cornerstone of machine learning, demonstrating state-of-the-art performance in diverse domains, including natural language processing (NLP), computer vision, and audio processing. These models leverage pre-trained representations on large-scale unlabeled datasets, enabling remarkable accuracy improvements in fine-tuned downstream tasks such as sentence classification and question answering. At the core of their success is the Transformer architecture [ 4 ] , which utilizes the self-attention mechanism to model complex relationships within input sequences. Despite the interest in deploying Transformer-based models on mobile and edge devices, their substantial computational and memory requirements present challenges in meeting the resource and energy constraints of these devices. In encoders and the prefill stage of decoders, the computational complexity of attention layers scales quadratically with the input sequence length, leading to memory and computational overheads that necessitate mitigation by means of dedicated acceleration. Although many architectures utilize General Matrix-Matrix Multiplication (GEMM) acceleration to alleviate the computational burden, performance bottlenecks are increasingly shifting toward non-linear operations, especially the Softmax function within the attention layers. Fig. 1: Runtime breakdown for GPT-3 on a RISC-V multi-cluster platform [ 5 ] . For each sequence length, the left bar shows unoptimized GEMM results, while the right bar reflects optimized GEMM results. Accelerating Softmax poses challenges due to its non-linear, non-pointwise nature and its reliance on a transcendental function, i.e. the exponentiation. The low arithmetic intensity of Softmax constrains parallelism and processing efficiency, a limitation that becomes more pronounced as GEMM latency decreases with acceleration. For example, the runtime breakdown for BERT on Volta GPU from Steven et al. [ 6 ] shows that Softmax contributes more than 30% for long sequences. Moreover, as shown in Figure 1 , deployment of GPT3-XL on the RISC-V multi-cluster platform [ 5 ] reveals that Softmax contributes to 30% of the runtime prior to GEMM operator acceleration, and 70% afterwards for sequence length of 2048. The use of large accelerators is justified for GEMMs , which constitute the majority of a Transformer’s workload in terms of number of operations. However, allocating considerable silicon area for Softmax acceleration is sub-optimal, as it represents only a small portion of the overall computational workload. Hence, addressing these challenges necessitates innovative solutions that optimize the Softmax function with minimal area costs while preserving accuracy. Existing software-level optimizations [ 7 ] often fall short of delivering the accuracy, performance and efficiency improvements necessary for large-scale or low-power deployments. Although hardware accelerators improve performance and energy efficiency, they typically lack flexibility due to their dependence on fixed-function datapath, and often lead to considerable integration area overhead in existing systems. Moreover, achieving accuracy parity often necessitates retraining [ 6 ] [ 8 ] —a technique that is undesirable, and often impractical for Large Language Models (LLMs) . In contrast, the growing adoption of the open and extensible RISC-V ISA to design domain-specialized programmable compute units offers a promising approach for addressing these limitations. In this work, we identify the exponential function as the primary computational bottleneck in Softmax computation, and we accelerate it on RISC-V-based systems, with low hardware overhead, through specialized ISA extensions. We demonstrate significant performance and energy efficiency gains in accelerating Transformer inference at native precision (Bfloat16), without compromising model accuracy, and without jeopardizing area and power consumption of the compute system. The contributions of this paper are: • We design a custom arithmetic block for accelerating the exponential function on Brain Floating-Point (Bfloat16 or BF16) data, integrate it with ultra-low overhead in the FPU of programmable RISC-V processors of an octa-core compute cluster, and extend their ISA with a custom EXP instruction; • We implement the cluster using GlobalFoundries 12\text{\,}\mathrm{nm} technology down to silicon-ready design and demonstrate that our solution incurs only a 1.0% area overhead at the cluster level and a negligible power overhead of 1.8% on workload with peak FPU utilization that do not exploit EXP, while the energy required to execute exponential operation is reduced by two orders of magnitude; • By exploiting the proposed ISA extensions, we optimize the execution of Softmax in software, demonstrating 162.7 \times latency reduction and 74.3 \times less energy consumption compared to a non-optimized kernel on the baseline cluster and achieving 1.4 \times better area efficiency and 7.4 \times lower power consumption compared to state-of-the-art Softmax accelerators, as detailed in Section VI . Moreover, we integrate our fully optimized Softmax kernel into FlashAttention-2, showing 8.2 \times performance and 4.1 \times energy efficiency improvements; • We scale up the proposed cluster to a 16-cluster system to evaluate the acceleration capabilities of our solution on end-to-end execution of pre-trained, un-tuned Transformers. We benchmark models such as GPT-2, GPT-3, and Vision Transformers (ViT), achieving up to 5.8 \times latency reduction and up to 3.6 \times less energy consumption compared to a baseline system without the proposed optimizations. Notably, these gains are achieved without re-training and with an accuracy loss of less than 0.1%. II Related Work Optimization techniques for Softmax can be broadly divided into two categories: workflow scheduling and computational approximations. Workflow scheduling techniques such as FlashAttention [ 9 ] and FlashAttention-2 [ 10 ] enhance data reuse through a tiling technique, thereby improving both memory efficiency and parallelism. While these techniques do not directly target optimizing the Softmax computation kernel, they are complementary to the kind of optimizations explored in this work, which focus specifically on improving the efficiency of the computation kernel itself. Computational approximations target the core Softmax operations: exponentiation and division. Full-precision exponentiation units based on iterative methods like Taylor

Read the complete text