ResBench: Benchmarking LLM-Generated FPGA Designs with Resource Awareness

Ce Guo, Tong Zhao · 2025

Read ResBench: Benchmarking LLM-Generated FPGA Designs with Resource Awareness by Ce Guo, Tong Zhao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-08823.txt.

Introduction

Ce Guo and Tong Zhao introduce a benchmark that judges AI-written chip designs not just on whether they work, but on how much hardware they waste. Language models can now write the code that configures programmable chips, yet earlier tests checked only correctness. A correct design that hogs the chip is a poor design in practice.

Their benchmark, ResBench, holds fifty-six problems in twelve categories, from simple state machines to financial computing. An open framework feeds each problem to a model, checks the generated code, and measures resource use, focusing on lookup tables, the basic building blocks of these chips. Results showed wide gaps between models: some produced lean designs while others burned far more resources for the same task. The authors say resource awareness must become a standard axis in judging code models for hardware.

Catalog record

CREATORCe Guo, Tong Zhao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-08823

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ResBench: Benchmarking LLM-Generated FPGA Designs with Resource Awareness Ce Guo, Tong Zhao Abstract . Field-Programmable Gate Arrays (FPGAs) are widely used in modern hardware design, yet writing Hardware Description Language (HDL) code for FPGA implementation remains a complex and time-consuming task. Large Language Models (LLMs) have emerged as a promising tool for HDL generation, but existing benchmarks for LLM-based code generation primarily focus on functional correctness while overlooking hardware resource usage. Furthermore, current benchmarks offer limited diversity and do not fully represent the wide range of real-world FPGA applications. To address these shortcomings, we introduce ResBench, the first resource-focused benchmark explicitly designed to distinguish between resource-optimized and inefficient LLM-generated HDL code. ResBench consists of 56 problems across 12 categories, covering applications from finite state machines to financial computing. Our open-source evaluation framework automatically tests LLMs by generating Verilog code, verifying correctness, and measuring resource usage. The experiments, which primarily analyze Lookup Table (LUT) usage, reveal significant differences among LLMs, demonstrating ResBench’s capability to identify models that generate more resource-optimized FPGA designs. DOI: XXXXXXX.XXXXXXX Conference: International Symposium on Highly Efficient Accelerators and Reconfigurable Technologies; May 26–28, 2025; Kumamoto, Japan ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Hardware Board- and system-level test CCS: Software and its engineering Source code generation CCS: Hardware Reconfigurable logic applications CCS: Hardware Functional verification CCS: Hardware Physical verification 1. Introduction Field-Programmable Gate Arrays (FPGAs) are widely used in reconfigurable computing, providing flexible and high-performance hardware implementations for applications such as artificial intelligence acceleration, financial computing, and embedded systems. However, FPGA development traditionally requires manual coding in hardware description languages (HDLs), a process that is both time-consuming and prone to errors. Large Language Models (LLMs) have recently shown potential in automating HDL generation, offering a way to improve FPGA design productivity. While studies such as VeriGen ( 32 ) and RTLLM ( 23 ) have examined the feasibility of generating Verilog with LLMs, most existing benchmarks focus primarily on functional correctness while paying little attention to resource constraints, which is an essential consideration in FPGA design. FPGA designs are subject to strict hardware resource constraints. Even when HDL code passes functional correctness tests, its efficiency in utilizing these resources can vary significantly depending on how logic is optimized and mapped to FPGA hardware. In reconfigurable computing, resource-aware optimizations play a crucial role in determining whether a design can be practically deployed and the degree of parallelism achievable. However, existing benchmarks for LLM-generated hardware designs typically evaluate the generated code based only on syntax and functional correctness, overlooking resource-related issues. To address this limitation, we introduce ResBench , the first FPGA-resource-focused benchmark specifically designed to evaluate LLM-generated designs based on resource usage. Unlike previous benchmarks that focus primarily on syntax and functional correctness, ResBench highlights how well LLMs generate Verilog code optimized for FPGA resource utilization. Our key contributions are: • A resource-focused benchmark featuring 56 problems across 12 categories, covering real-world FPGA workloads such as combinational logic, state machines, AI accelerators, and financial computing applications. (Section 3 ) • An open-source automated evaluation framework that performs LLM querying, functional correctness testing, FPGA synthesis, and resource measurement 1 1 1 Code repository: https://github.com/jultrishyyy/ResBench . The framework automatically generates Verilog code using LLMs and evaluates its correctness and resource usage. (Section 4 ) • A detailed study of nine LLMs, comparing their performance in functional correctness and FPGA resource usage. The results reveal substantial differences in how various models generate resource-conscious designs. (Section 5 ) By integrating FPGA resource awareness into benchmarking, ResBench provides a practical evaluation of LLM-generated HDL. This benchmark establishes a foundation for advancing AI-driven FPGA design, encouraging the development of more resource-efficient models optimized for FPGAs. 2. Background This section explores the evolution of large language models (LLMs) from general code generation to hardware design generation using hardware description languages (HDLs). We examine their capabilities and limitations, as well as existing benchmarks for LLM-generated hardware design. Table 1 . Benchmarks for LLM HDL Generation Benchmark Size PL Type Features VerilogEval ( 21 ) 156 Verilog Verilog code generation tasks Covers a wide range of tasks from simple combinational circuits to finite state machines; includes automatic functional correctness testing. HDLEval ( 17 ) 100 Multiple Language-agnostic HDL evaluation Evaluates LLMs across multiple HDLs using standardized testbenches and formal verification; categorizes problems into combinational and pipelined tests. PyHDL-Eval ( 4 ) 168 Python-embedded DSLs Specification-to-RTL tasks Focuses on Python-embedded DSLs for hardware design; includes Verilog reference solutions and testbenches; evaluates LLMs’ ability to handle specification-to-RTL translations. RTLLM ( 22 ) 50 Verilog, VHDL, Chisel Design RTL generation Supports evaluation across multiple HDL formats; spans various design complexities and scales; includes an automated evaluation framework. VHDL-Eval ( 37 ) 202 VHDL VHDL code generation tasks Aggregates translated Verilog problems and publicly available VHDL problems; utilizes self-verifying testbenches for functional correctness validation. GenBen ( 38 ) 351 Verilog Fundamental hardware design and debugging tasks Evaluates synthesizability, power consumption, area utilization, and timing performance to ensure real-world applicability. 2.1. Code-specialized and HDL-specialized LLMs Language models have seen significant advancements, particularly with the introduction of Transformers ( 36 ) . Large-scale pre-trained models such as BERT ( 8 ) , the GPT series ( 28 , 5 , 44 ) , and PaLM 2 ( 7 ) have expanded their capabilities across various tasks, including code generation. Code-specialized LLMs are models designed to generate computer programs based on human-language prompts. Surveys such as ( 16 , 45 , 39 ) review the latest techniques and model architectures developed for this purpose. In general, a code-specialized LLM can be created using two different approaches. • The first approach trains models from scratch on large-scale open-source code datasets spanning multiple programming languages, as demonstrated by CodeGen ( 26 ) , InCoder ( 9 ) , and StarCoder ( 19 ) . These code-first LLMs excel at code completion and multi-language generation. However, this method demands extensive computational resources and high-quality datasets. Additionally, these models may struggle with instruction-driven tasks due to their limited ability to process human language instructions. • The second approach fine-tunes general-purpose LLMs for coding tasks, as seen in Codex ( 6 ) , which is derived from GPT-3, and Code Llama ( 30 ) , which builds on Meta’s Llama. Fine-tuning incorporates the linguistic knowledge of general LLMs, allowing them to handle natural language prompts while maintaining strong coding capabilities. This method is more computationally efficient than training from scratch, but it may lack the precision of code-first models in understanding programming language syntax. While significant research has explored the use of language models for general software code generation, the application of LLMs to HDL code has not received comparable attention. Existing work on HDL code generation primarily focuses on improving general-purpose or software-code-specialized LLMs. In particular, these studies aim to improve these LLMs’ understanding of hardware description tasks by training them on HDL datasets and benchmarking frameworks. Notable efforts in this area include VeriGen, MEV-LLM, and AutoVCoder. • VeriGen ( 33 ) fine-tunes CodeGen (2B, 6B, 16B) on a Verilog dataset collected from GitHub repositories and textbooks. It employs supervised fine-tuning with problem-code pairs, validating functional correctness using a benchmark problem set and problems from HDLBits tutorials ( 31 ) . • MEV-LLM

Read the complete text