Design and Implementation of an FPGA-Based Tiled Matrix Multiplication Accelerator for Transformer Self-Attention on the Xilinx KV260 SoM CCS: Hardware Field-programmable gate arrays CCS: Hardware Data-flow architectures

Richie Li, Sicheng Chen · 2025

Read Design and Implementation of an FPGA-Based Tiled Matrix Multiplication Accelerator for Transformer Self-Attention on the Xilinx KV260 SoM CCS: Hardware Field-programmable gate arrays CCS: Hardware Data-flow architectures by Richie Li, Sicheng Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-16731.txt.

Introduction

Richie Li and Sicheng Chen built a hardware accelerator for the matrix math at the heart of transformer language models, in a 2025 paper. The attention step in these models leans on heavy matrix products, which form a key bottleneck. Their chip design runs on a small, resource-tight FPGA board.

The design keeps data on chip, reuses it through two levels of tiling, and unrolls its compute engine like a systolic array. Paired with a compact transformer model for its key projections, it ran seven times faster than a phone-class processor baseline. Against a naive software version the gain was about two hundredfold. The authors pitch it for efficient inference at the edge.

Catalog record

CREATORRichie Li, Sicheng Chen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-16731

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Design and Implementation of an FPGA-Based Tiled Matrix Multiplication Accelerator for Transformer Self-Attention on the Xilinx KV260 SoM CCS: Hardware Field-programmable gate arrays CCS: Hardware Data-flow architectures Richie Li, Sicheng Chen Abstract . Transformer-based large language models (LLMs) rely heavily on intensive matrix multiplications for attention and feed-forward layers, with the Q, K, and V linear projections in the Multi-Head Self-Attention (MHA) module constituting a decisive performance bottleneck. In this work, we introduce a highly optimized tiled matrix multiplication accelerator on a resource-constrained Xilinx KV260 FPGA that not only addresses this challenge but sets a new standard for efficiency and performance. Our design exploits persistent on-chip storage, a robust two-level tiling strategy for maximal data reuse, and a systolic-like unrolled compute engine that together deliver unparalleled speed and energy efficiency. Integrated with DistilBERT for Q, K, and V projections, our accelerator achieves an unequivocal 7× speedup over ARM CPU implementations (PyTorch) and an extraordinary 200× improvement over naive NumPy, reaching a throughput of up to 3.1 GFLOPs for matrix multiplications on (64,768) × (768,3072) matrices while operating at a conservative 100 MHz. These results decisively demonstrate the transformative potential of FPGA-based acceleration for critical Transformer operations, paving the way for scalable and energy-efficient deep learning inference on edge devices. 1. Introduction Large Language Models (LLMs) like Transformers rely on intensive matrix multiplications for their self-attention and feed-forward blocks. As shown in the Figure 1 , computing the Q, K, V, and output linear projections during Multi-Head Self-Attention in each Transformer layer entails multiplying large matrices (input activation \times Wq/Wk/Wv/Wo). DistilBERT ( distilbert ) , a compact Transformer variant ( vaswani2017attention ) , still performs billions of multiply-accumulate operations per inference. These operations are memory-bandwidth heavy on general processors, motivating specialized hardware acceleration. FPGAs offer custom parallelism and energy efficiency for matrix math, but deploying LLM acceleration on edge FPGAs presents challenges due to limited resources and power. Diagram showing the multi-head self-attention pipeline in a Transformer model. Figure 1. MHA Pipeline in transformer Diagram showing the multi-head self-attention pipeline in a Transformer model. Motivated by the observation that the Q, K, and V projections form a significant computational bottleneck, we strategically concentrated our efforts on accelerating these key components. This targeted approach allowed us to fully optimize our design for the resource constraints of the Xilinx Kria KV260 Vision AI Kit (housing a Zynq UltraScale+ FPGA), while still delivering substantial performance improvements. Our approach exploits data reuse and parallelism to speed up matrix multiplication operations within the tight DSP and BRAM budget of an edge FPGA. We adopt a block/tiled matrix multiply approach: one input matrix (“A”) remains in on-chip memory, while the other (“B”) is processed in smaller column blocks to maximize reuse. Unlike prior LLM accelerators that apply model-wide optimizations (e.g., pruning, specialized sparse formats), our work directly accelerates dense GEMM (General Matrix Multiply) cores within the Q, K, V projection operations. To encourage reproducibility and further research, we provide our FPGA design, software interface, and benchmarking scripts in an open-source repository at GitHub . 2. Contributions In this work, we present a specialized FPGA-based accelerator for key matrix multiplication operations in Transformer self-attention. While elements of our contributions are woven throughout the Introduction and Discussion sections, here we explicitly summarize the main novel aspects and potential benefits for future research: • Targeted Focus on Self-Attention Bottleneck: We pinpoint the Q, K, and V projections within Multi-Head Self-Attention as critical performance bottlenecks. Unlike many existing FPGA accelerators that aim to cover a broad range of model operations (often at the cost of higher resource demands), our design is tightly focused on these fundamental matrix multiplications, making it more suitable for edge FPGAs with constrained resources. • Tiled and Data-Centric Design: By employing two-level tiling and on-chip data persistence, we minimize off-chip memory transfers and fully leverage data reuse. Our systolic-like unrolled compute engine—sized specifically to fit KV260 resources—enables high throughput (up to 3.12 GFLOPs) while staying within strict DSP and BRAM budgets. This data-centric approach is amenable to analytic frameworks such as MAESTRO ( MAESTRO ) and other DNN dataflow studies ( DNNDataflows ) . • High-Level Synthesis (HLS) and Easy Integration: Implementing the accelerator in C++ HLS facilitates rapid design space exploration and more accessible customization. The accelerator is integrated via AXI4 and PYNQ overlays, offering a user-friendly interface that can be extended to other matrix-multiply-based workloads with minimal effort. • Quantized DistilBERT Integration: We replace standard PyTorch Q, K, and V linear layers with our FPGAQuantizedLinear module and validate end-to-end functionality on a quantized DistilBERT. This demonstrates near-lossless accuracy while providing up to a 7 \times speedup over PyTorch CPU baselines for these layers. Our open-source release includes scripts for quantization, FPGA invocation, and reproducible benchmarking, potentially lowering the barrier for researchers to adopt FPGA acceleration in transformer applications. • Roadmap for Future Scaling: Although we focus on moderate matrix sizes (e.g., 64 \times 768, 768 \times 3072) representative of DistilBERT, our tiled architecture offers a blueprint for scaling to larger transformer variants. The clear separation of on-chip buffering, tiling logic, and computation modules can be extended to handle more extensive hidden dimensions and different attention heads with minimal redesign. By addressing a critical compute bottleneck within Transformers and providing an open-source HLS-based implementation, we aim to advance both the practical adoption of FPGAs in edge-deployed large language models and future studies on optimized dataflows for deep learning inference. 3. Related Work FPGA Accelerators for Transformers Recent research has produced FPGA-based LLM accelerators such as FlightLLM ( FlightLLM ) and SSR ( SSR ) . FlightLLM (FPGA’24) maps entire LLM inference flows onto FPGAs, leveraging sparsity and mixed-precision; implemented on a high-end Alveo U280, it achieves 6× higher energy efficiency than an NVIDIA V100 GPU ( FlightLLM ) by using sparse DSP chains and always-on-chip decoding. SSR (FPGA’24) explores launching multiple accelerators in parallel versus sequentially to balance latency and throughput. On a Versal ACAP VCK190, SSR attains up to 2.5× throughput versus an NVIDIA A10G GPU, with 8.5× energy efficiency ( SSR ) . FAMOUS focuses on the Transformer’s attention mechanism: a flexible multi-head attention core on Alveo U55C that sustains 328 GOPS and runs 2.6× faster than an NVIDIA V100 GPU and 1.3× faster than prior FPGA designs. Earlier, Qi et al. (GLSVLSI’21) combined model compression (pruning) with FPGA optimization to fit Transformers onto FPGAs. These works, however, target large FPGAs or datacenter contexts and often integrate advanced techniques (sparsity, multi-engine concurrency) less applicable to small-edge devices. In addition to large-scale accelerators, recent research has advanced data-centric analytical frameworks to understand data reuse and mapping strategies in DNN accelerators. For instance, MAESTRO ( MAESTRO ) and the approach presented in “Understanding Reuse, Performance, and Hardware Cost of DNN Dataflows: A Data-Centric Approach” ( DNNDataflows ) provide comprehensive models for quantifying data movement, reuse, and the associated hardware costs. Although these frameworks target a broader range of DNN workloads, they share foundational principles with our work—particularly the emphasis on maximizing both temporal and spatial data reuse—which informs our targeted accelerator design for Transformer self-attention on edge FPGAs. Our Approach vs Prior Art Unlike FlightLLM or SSR, which assume abundant resources (HBM memory, >9000 DSPs) and

Read the complete text