TerEffic: Highly Efficient Ternary LLM Inference on FPGA

Chenyang Yin 1 , Zhenyu Bai 2 , Pranav Venkatram 2 , Shivam Aggarval 2 , Zhaoying Li 2 , and Tulika Mitra 2 · 2025

Read TerEffic: Highly Efficient Ternary LLM Inference on FPGA by Chenyang Yin 1 , Zhenyu Bai 2 , Pranav Venkatram 2 , Shivam Aggarval 2 , Zhaoying Li 2 , and Tulika Mitra 2 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-16473.txt.

Introduction

Chenyang Yin and co-authors present TerEffic in 2025, hardware that runs large language models with ternary weights on a reconfigurable chip. Edge devices lack the memory and power budget for standard models. Ternary quantization, where each weight takes one of three values, slashes memory while keeping much of the accuracy.

General processors waste that advantage because they still move and multiply data the wide way. Their FPGA design is built around ternary arithmetic from the start. The result is fast, low-power inference suited to phones, robots, and offline devices. Efficient local models matter for privacy and for places with poor connectivity, two growing limits on cloud-based AI.

Catalog record

CREATORChenyang Yin 1 , Zhenyu Bai 2 , Pranav Venkatram 2 , Shivam Aggarval 2 , Zhaoying Li 2 , and Tulika Mitra 2
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-16473

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TerEffic: Highly Efficient Ternary LLM Inference on FPGA Chenyang Yin 1 , Zhenyu Bai 2 , Pranav Venkatram 2 , Shivam Aggarval 2 , Zhaoying Li 2 , and Tulika Mitra 2 Abstract Deploying Large Language Models (LLMs) efficiently on edge devices is often constrained by limited memory capacity and high power consumption. Low-bit quantization methods, particularly ternary quantization, have demonstrated significant potential in preserving model accuracy while substantially decreasing memory footprint and computational costs. However, existing general-purpose architectures and accelerators have not fully exploited the advantages of low-bit quantization due to insufficient specialized hardware support. We introduce TerEffic, an FPGA-based architecture tailored for ternary-quantized LLM inference. The proposed system offers flexibility through reconfigurable hardware to meet various system requirements. We evaluated two representative configurations: a fully on-chip design that stores all weights within on-chip memories, scaling out using multiple FPGAs, and an HBM-assisted design capable of accommodating larger models on a single FPGA board. Experimental results demonstrate significant performance and energy efficiency improvements. For single-batch inference on a 370 M-parameter model, our fully on-chip architecture achieves 16,300 tokens/second, delivering a throughput 192× higher than NVIDIA Jetson Orin Nano with a power efficiency of 455 tokens/second/W, marking a 19× improvement. The HBM-assisted architecture processes 727 tokens/second for a larger 2.7B-parameter model—3× the throughput of NVIDIA A100—while consuming only 46W, resulting in a power efficiency of 16 tokens/second/W, an 8× improvement over the A100. I Introduction Large Language Models (LLMs) have revolutionized AI by delivering unprecedented performance across diverse applications. State-of-the-art LLMs such as ChatGPT [ 1 ] , Claude [ 2 ] , and Google Gemini [ 3 ] continue scaling upwards in parameter count to enhance their capabilities. However, a concurrent and significant trend has emerged toward more efficient inference through the deployment of smaller yet highly effective models. This efficiency is primarily achieved through advanced techniques like model distillation [ 4 , 5 ] and low-bit quantization [ 6 , 7 , 8 , 9 ] . For example, Meta’s Llama3.2 [ 10 ] provides lightweight models with 1B and 3B parameters specifically optimized for high-performance edge tasks, while GPT-4o mini [ 11 ] demonstrates superior capabilities to GPT-4 at significantly reduced inference costs. Nevertheless, existing AI inference accelerators—such as GPUs, NPUs, and TPUs—do not fully align with this emerging trend. Firstly, these accelerators lack specialized hardware tailored to efficiently handle the arithmetic operations and data movement patterns specific to low-bit quantized models, particularly binary and ternary LLMs. Secondly, current accelerators are predominantly DRAM-centric, relying heavily on off-chip High Bandwidth Memory (HBM). Such DRAM-based designs incur considerable energy overhead and suffer from bandwidth bottlenecks due to the memory wall [ 12 ] . However, aggressive model distillation combined with low-bit quantization (e.g., 1-bit LLMs) significantly reduces memory requirements, practically enabling fully SRAM-based on-chip inference. As illustrated in Figure 1 , low-bit quantization allows larger models to fit entirely within on-chip SRAM, making architectures with distributed SRAM—such as FPGAs—particularly advantageous. By leveraging high on-chip memory bandwidth, FPGAs can substantially improve throughput and energy efficiency compared to conventional GPU-based architectures. Fig. 1 : On-chip vs. Off-chip Memory for Inference: Throughput and Energy Trends with Increasing Model Parameters Therefore, we propose TerEffic , a specialized FPGA-based accelerator architecture optimized explicitly for efficient ternary LLM inference. TerEffic provides tailored hardware units designed for efficient ternary arithmetic with optimized memory design and datapaths to unlock the potential of low-bit quantized LLM inference. In addition, TerEffic features versatility for ternary LLMs with different sizes and activation functions. To make our design practical under different system requirements, we propose two architecture variants: (a) a fully on-chip design that stores all weights within on-chip memories and can scale out using multiple FPGAs through the GTY/QSFP interface for FPGA-FPGA direct data transfer. (b) an HBM-assisted design that stores weights off-chip, capable of accommodating larger models on a single FPGA board. The HBM-assisted variant can be deemed as a specialized design that leverages the potential of ternary quantization while the fully on-chip design provides further improvement with full-SRAM inference. Both designs demonstrate remarkable advantages over GPUs in terms of throughput and power efficiency. For a 370M-parameter model, our fully on-chip architecture achieves a single-batch throughput of 16,300 tokens/second ( 192\times higher) and a power efficiency of 455 tokens/second/W ( 19\times better) compared to NVIDIA’s Jetson Orin Nano. For larger models that far exceed the on-chip memory capacity of the FPGA, our HBM-assisted architecture also shows high performance. For a 2.7B-parameter model, it provides a single-batch throughput of 727 tokens/second and a power efficiency of 16 tokens/second/W, which is 3\times and 8\times better compared to NVIDIA’s A100, respectively. II Background & Related work II-A Binary and Ternary Quantizations Quantization has been a key strategy to accelerate inference as neural networks grow rapidly in parameter count. Among the various quantization methods, binary and ternary quantizations feature extremely compressed weights that require little memory capacity and bandwidth. They were initially validated in CNN models such as BNN [ 13 ] , XNOR-Net [ 14 ] , and TWN [ 15 ] , achieving high efficiency with minimal accuracy loss. Entering the era of LLMs, larger model sizes and limited hardware resources further necessitate low-bit quantization, especially in the context of edge devices. LLMs utilizing binary or ternary quantization are also referred to as ’1-bit’ LLMs, as each weight theoretically takes less than 2 bits for storage. As a pioneer of 1-bit LLM, BitNet [ 8 ] employed binary quantization and introduced the BitLinear layer as a replacement for linear projections. As its variant and improvement, BitNet 1.58b [ 9 ] utilized ternary quantization to match the performance of full-precision (fp16 or bf16) Transformers. Later, TernaryLLM [ 16 ] leveraged knowledge distillation in its training and surpassed previous low-bit quantization methods. However, as the Q, K, V matrices in the self-attention mechanism remain in high precision, the high-precision matrix multiplications were not eliminated. To address this issue and realize a fully ’MatMul-free’ model, [ 17 ] used an RNN variant to replace the traditional self-attention in their ternary model and achieved excellent performance. Furthermore, this 1-bit quantization trend was extended to the Vision Transformer (ViT) domain [ 18 ] and a scaling law [ 19 ] was established, providing a theoretical basis for the impressive performance of these 1-bit models. Advances in post-training reparameterization [ 20 ] also paved the way for low-cost training of 1-bit LLMs. More recently, Microsoft released an official CPU inference framework for BitNet, bitnet.cpp [ 21 ] . However, this framework focused on the software level, while ternary models still have significant potential for improvements from custom hardware. II-B FPGA-based Transformer Accelerators Field-programmable gate arrays (FPGAs) are widely used for accelerating various tasks. In the context of deep learning, several works have contributed toward accelerating Transformer inference using FPGAs. Many early attempts [ 22 , 23 , 24 , 25 , 26 , 27 , 28 ] adopted overlay architectures that utilized an instruction set and universal computing engine for all layers. While overlay designs demonstrate flexibility for different models, they always suffer from frequent data movements of intermediate results. Later, some spatial architectures have been proposed, featuring specialization of distinct PEs for specific operations. [ 29

Read the complete text