Flex-PE : Flexible and SIMD Multi-Precision Processing Element for AI Workloads

Mukul Lokhande, Gopal Raut, Santosh Kumar Vishvakarma · 2024

Read Flex-PE : Flexible and SIMD Multi-Precision Processing Element for AI Workloads by Mukul Lokhande, Gopal Raut, Santosh Kumar Vishvakarma (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-11702.txt.

Introduction

Mukul Lokhande, Gopal Raut, and Santosh Kumar Vishvakarma built a chip part for AI math that bends to many precisions. Their paper, from twenty twenty-four, deals with processing elements for deep learning and like loads. Models now shift fast, and gear must switch number size and curve shapes on the fly. Past parts do one or the other, not both. Their part, Flex-PE, does both in one cell.

It runs in lanes that split one wide job into many slim ones. It holds the main curve shapes used in neural nets plus the core sum of products step. Throughput scales with how slim the numbers are. Readers who build AI chips get a cell that trades width for speed as the job asks.

Catalog record

CREATORMukul Lokhande, Gopal Raut, Santosh Kumar Vishvakarma
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-11702

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Flex-PE : Flexible and SIMD Multi-Precision Processing Element for AI Workloads Mukul Lokhande, Gopal Raut, Santosh Kumar Vishvakarma Abstract The rapid adaptation of data-driven AI models, such as deep learning inference, training, Vision Transformers (ViTs), and other HPC applications, drives a strong need for run-time precision configurable different non-linear activation functions (AF) hardware support. Existing solutions support diverse precision or run-time AF reconfigurability but fail to address both simultaneously. This work proposes a flexible and SIMD multi-precision processing element (Flex-PE), which supports diverse run-time configurable AFs, including sigmoid, tanh, ReLU and softmax, in addition to MAC operation. The proposed design achieves an improved throughput of up to 16 × \times FxP4, 8 × \times FxP8, 4 × \times FxP16 and 1 × \times FxP32 in pipeline mode with 100% time-multiplexed hardware. This work proposes an area-efficient multi-precision iterative mode in the SIMD systolic arrays for edge-AI use cases. The design delivers superior performance with up to 62× and 371× reductions in DMA reads for input feature maps and weight filters in VGG-16, with an energy efficiency of 8.42 GOPS / W within the accuracy loss of 2%. The proposed architecture supports emerging 4-bit computations for DL inference while enhancing throughput in FxP8/16 modes for transformers and other HPC applications. The proposed approach enables a way for future energy-efficient AI accelerators in both edge and cloud environments. I Introduction (a) (b) Fig. 1: (a) Typical Deep Neural Network (DNN) model showcasing various layers including Conv, Pooling, and FC. (b) AI SoC featuring a RISC-V-enabled Systolic Array with detailed PE architecture. Artifical Intelligence (AI) has grown far ahead of human capabilities and is a major data-driven decision maker. However, the demand for computing resources has grown rapidly from deep neural networks (DNNs) to the recent large-language models (LLMs). This highlights the need for hardware accelerators. We must not only focus on increasing the peak compute power of hardware but also keep in mind memory usage and intra/inter-chip communication bottlenecks while serving AI models [ 34 ] . In the past 20 years, TOPS performance has been enhanced by 60,000 \times , while DRAM bandwidth and interconnect bandwidth have been improved by only 30 \times and 100 \times respectively, creating a disparity known as the memory wall problem [ 34 ] . The memory wall problem is characterized by limitations in memory capacity, data transfer bandwidth, and access latency, which majorly affect overall system performance. The bottlenecks have not been only for edge inference evaluation on real-time data but also for cloud training with collected data. Thus, extensive parallel processing with quantised precision has become the need of the hour, considering the rapid growth in AI use cases to exploit throughput gains. AI applications, from real-time edge inference to large-scale cloud training, have forced the necessity for hardware accelerators that can effectively trade between performance, flexibility, and efficiency. The rapidly varying diversity activation functions (AFs) and precision support introduce an additional layer of complexity in the design for such accelerators. The breakdown of the total execution of different AI workloads, including DNN (LeNet), RNNs (LSTM, BI-LSTM, and GRU) and the transformer (BERT), Fig. 2 highlights the significant need for AF optimization. Edge-AI devices require rapid event-triggered response, superior operational efficiency, and dense computational throughput. In contrast, high-performance computing (HPC) devices focus on providing enhanced large-scale performance within available memory bandwidth and interconnect at the cloud node. The general DNN hardware architecture is shown in Fig. 1(a), and RISC-V enabled AI SoC is depicted in Fig. 1(b) with Systolic array architecture. Previous works have explored quantization and Single Instruction Multiple Data (SIMD) vectorization techniques to provide efficient trade-offs between arithmetic intensity and data feed rates [ 33 ] . Precision-reduction techniques like quantization enable faster inference with minor accuracy compromise, while high-precision training burdens computational and memory resources, forcing bandwidth bottlenecks. Prior works have demonstrated consumer-satisfactory DNN and Transformer inference within 4 or 8-bit dynamic fixed-point (FxP) precision. However, higher precision is necessary for handling error accumulation, precise gradient calculations, critical data dependencies, attention fidelity and convertible backpropagation in DNN training and HPC applications such as RNNs, LSTMs, and Transformers to achieve satisfactory performance [ 42 , 43 , 44 ] . Pre-processing blocks can easily handle the conversion between various intermediate precision or integer to fixed-point conversions [ 1 ] without contributing any overhead. TABLE I: State-of-the-art methodologies and comparison of corresponding features in AF units for AI applications. Comparison AF supported Configurability SIMD supported Supported Precision Datatypes Throughput Hardware architecture Design Overhead (Area/Delay) Applications TAI-24 [ 8 ] Sigmoid, Tanh Yes No 24 Fixed 1/1/1/1/1/1 Pipelined Area RNN/LSTM TCSVT’24 [ 7 ] Softmax No No 16/32 Fixed 1/1 pipeline Area Transformers ISQED’24 [ 9 ] Sigmoid, Tanh, Softmax No No 12/16/32 Bf/Tf/Float/Posit 1/1/1 Iterative/Parallel/Pipelined Delay/Area edge inference ISQED’24 [ 32 ] Sigmoid, Tanh Yes Yes 8/16 Fixed 2/1 Pipelined Area DNN inference TCAS-I’23 [ 10 ] Softmax No No 32 Fixed 1 Pipelined Area DNN Training TCAS-II’23 [ 11 ] Sigmoid, Tanh, Swish, ReLU Yes No 12 Fixed 1 Taylor-Series + NRDiv NA DNN inference IEEE Access’23 [ 12 ] Softmax No No 8/16/32 Fixed 1/1/1 Pipelined Area DNN inference TC’23 [ 13 ] Sigmoid, Tanh Yes No 16 Fixed 1 Pipelined Area LSTM ISVLSI’20 [ 14 ] Sigmoid, Tanh Yes No 8/12/16 Fixed 1/1/1 Iterative Delay ANN TCAD’19 [ 15 ] Tanh, SELU No No 7/8 Fixed 1/1 LUT-combo NA ANN Proposed Sigmoid, Tanh, Softmax, ReLU Yes Yes 4/8/16/32, 12/24* Fixed 16/8/4/1 Flexible Runtime tradeoff DNN, RNN/LSTM, Transformers Training and Inference Note \ast : The work also supports 2 \times FxP12, 4 \times FxP4 or 1 \times FxP24 heterogeneous precision operations. Fig. 2: Workload analysis [ 8 ] emphasizing on the growing demand for performance-enhanced non-linear activation functions. AI compute hardware primarily consists of general matrix-matrix multiplication (GEMM), matrix-vector multiplication (MVM), and multiply-and-accumulate (MAC) operations to execute diverse DL workloads-such as convolution, recurrent and fully connected (FC) layers and Transformer workloads, involving encoder and decoder layers. Existing accelerators are effective for specific tasks but lack the flexibility to support dynamically varying precisions and activation function requirements, limiting the execution of edge-AI and HPC workloads simultaneously. Prior works [ 35 ] highlight that significant matrix processing cores are necessary for highly parallelizable workloads, but the performance is bottlenecked by challenges associated with thread scheduling, data moment, resource utilisation and the energy efficiency of the varying proportion of scalar, non-MAC operations. The CORDIC-based methodology has been widely used in the previous methodologies [ 23 , 14 ] . AI algorithm-specific operand dependency-aware tiling overcomes these limitations by enhancing compute utilization, system performance, and power efficiency [ 41 ] . However, the efficient SIMD-enabled hardware is crucial to fully leverage these optimizations and execute these operations in parallel. The approach empowers low latency at the edge node while providing flexibility at the cloud node for workload mapping, optimizing compute density and reducing utilisation of memory bandwidth and interconnect usage. A key drawback in state-of-the-art (SoTA) works [ 14 , 7 , 8 , 9 , 32 , 10 , 11 , 12 , 13 , 15 ] , which can be observed from Table I , is the absence of SIMD-configurable multi-precision activation function

Read the complete text