TuRTLe: A Unified Evaluation of LLMs for RTL Generation

Dario Garcia-Gasulla, Gokcen Kestor, Emanuele Parisi, Miquel Albertí-Binimelis, Cristian Gutierrez, Razine Moundir Ghorab, Orlando Montenegro, Bernat Homs, Miquel Moreto · 2025

Read TuRTLe: A Unified Evaluation of LLMs for RTL Generation by Dario Garcia-Gasulla, Gokcen Kestor, Emanuele Parisi, Miquel Albertí-Binimelis, Cristian Gutierrez, Razine Moundir Ghorab, Orlando Montenegro, Bernat Homs, Miquel Moreto (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-01986.txt.

Introduction

Dario Garcia-Gasulla and co-authors grade language models on writing the code that becomes computer chips, in a 2025 paper. Register-transfer level code must be more than grammatical: it must work, build into real hardware, and respect power and size budgets. Ordinary code benchmarks miss those extra demands. Their framework, TuRTLe, unifies several existing tests into one automated harness.

Models are scored on syntax, function, successful synthesis, efficiency of the resulting design, and exact line completion. A broad set of open models was run through the gauntlet. Reasoning-style models, DeepSeek R1 among them, led across most criteria. Even the leaders left clear weaknesses in hardware-specific skills. The suite gives chip teams a shared yardstick for model choice.

Catalog record

CREATORDario Garcia-Gasulla, Gokcen Kestor, Emanuele Parisi, Miquel Albertí-Binimelis, Cristian Gutierrez, Razine Moundir Ghorab, Orlando Montenegro, Bernat Homs, Miquel Moreto
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-01986

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TuRTLe: A Unified Evaluation of LLMs for RTL Generation Dario Garcia-Gasulla, Gokcen Kestor, Emanuele Parisi, Miquel Albertí-Binimelis, Cristian Gutierrez, Razine Moundir Ghorab, Orlando Montenegro, Bernat Homs, Miquel Moreto Abstract Rapid advancements in LLMs have driven the adoption of generative AI in domains like Electronic Design Automation (EDA). Within the field of software development, EDA presents unique challenges derived from specific requirements of generated RTL code; RTL code must not only be syntactically correct and functionally accurate, but also synthesizable by hardware generators, while matching performance, power and area (PPA) constraints. These additional requirements introduce complexities that existing code-generation benchmarks often fail to capture, limiting their effectiveness in evaluating LLMs for RTL generation. To address this gap, we propose TuRTLe , a unified evaluation framework designed to systematically assess LLMs across key RTL generation tasks. TuRTLe integrates multiple existing benchmarks and automates the evaluation process, enabling a comprehensive assessment of LLM performance in syntax correctness, functional correctness, synthesis, PPA optimization, and exact line completion. Using this framework, a diverse set of forty open LLMs are assesed, tracking their strengths and weaknesses in EDA-specific tasks. Our results identify the best match for specific tasks ( e.g., base models are better in module completion tasks, instruct-tuned models are better in specification-to-RTL tasks), while finding that recent models with autoregressive reasoning chain perform the best overall. Some benchmarks, particularly within syntax correctness, show signs of saturation, while others remain as open problems for LLMs. I Introduction Advancements in large language models (LLMs) have unlocked new possibilities across a wide range of domains [ 1 , 2 ] . Domain-specific LLMs have gained significant attention due to their strong performance in specialized tasks, including financial engineering [ 3 ] , biomedical research [ 4 ] , and scientific computing [ 5 , 6 ] . In software related tasks, LLM can suggest code snippets, solve common coding challenges, and provide explanations of complex concepts [ 7 , 8 , 9 , 10 ] . In the field of Electronic Design Automation (EDA), researchers are increasingly exploring the use of LLMs to accelerate hardware design [ 11 , 12 , 13 ] . In the traditional digital system design flow, engineers must implement precise functionality using hardware description languages (HDLs). LLM-based solutions aim to bridge this gap by translating functional specifications directly into HDL code, such as Verilog. This approach has the potential to revolutionize hardware design and verification by streamlining Verilog coding, optimizing circuit implementations, and automating time-consuming design tasks [ 14 ] . Recent research has explored various techniques for Verilog code generation, including prompt engineering to enhance model responses [ 15 ] , training or fine-tuning on Verilog-specific datasets [ 16 , 17 , 18 , 19 , 20 ] , and agent-based approaches to improve iterative refinement and debugging [ 21 , 22 , 23 , 24 , 25 ] . However, several research questions about the quality and fidelity of the generated Verilog code remain largely unanswered. LLM-based solutions need to generate code that is syntactically and functionally correct, but in EDA the generated code must also be synthesizable to produce correct hardware components, and achieve reasonable performance in non-functional indicators such as area and power. To mitigate this issue and assess the effectiveness of current LLM for EDA, several benchmarks have been introduced, including VerilogEval [ 26 , 27 ] , RTL-Repo [ 28 ] , and RTLLM [ 29 ] . While these benchmarks evaluate different aspects of Verilog code generation, including single-line completion, single-module generation, and specification-to-RTL translation, none fully evaluates the entire hardware design flow, and some use different metrics. This makes it challenging to compare results across benchmarks and LLMs. A standardized evaluation methodology and infrastructure would allow researchers to systematically assess LLM capabilities, identify strengths and limitations in existing benchmarks and models, and drive further advancements in LLM-driven hardware design automation. This work introduces TuRTLe , a comprehensive evaluation framework for RTL generation. It integrates multiple benchmarks into a single fully-automated evaluation framework, providing a standardized and comprehensive assessment of code. Coverage is provided for different RTL design goals, including syntax and correctness of the generated code, synthesizability of the hardware circuits produced by hardware generators, together with their corresponding performance, area, and power metrics, using a novel metric based on comparison against human performance. These goals are tested under different conditions, such as module completion, and generation from specifications expressed in natural language. TuRTLe is used to benchmark forty models, providing a comprehensive, wide and updated review on the state of open models. This includes general purpose, coding, and RTL-specific LLMs, as well as base, instruct and reasoning models, showing the strengths and weaknesses of each alternative. Our analysis shows that temperature, context length and training strategy ( e.g. , base, instruct, reasoning) all play a crucial role in LLM performance across EDA-related tasks. Base models demonstrate stronger performance in module completion, effectively filling in partial RTL definitions, while instruct-tuned models are better at generating RTL from natural language specifications. Models employing reasoning prompting chains, exhibit remarkable performance across the board at the cost of increased inference time and token generation. Overall, our findings indicate that while current LLMs are highly reliable in generating syntactically correct RTL code, they still struggle with functional correctness and in achieving human-level non-functional indicators. II Evaluation Methodology Evaluating LLM-generated RTL code requires a comprehensive and systematic approach that considers the many aspects of the hardware design process. TuRTLe integrates existing benchmarks including VeriGen [ 16 ] , VerilogEval [ 26 , 27 ] , RTL-Repo [ 28 ] , and RTLLM [ 29 ] , within a single infrastructure, allowing for direct comparison of LLM performance across various RTL generation tasks. Focusing on versatility and automation in the construction and running of TuRTLe , we choose a widely adopted framework in the field of code generation ( BigCode Evaluation Harness [ 30 ] ) as a foundation. This tool can easily load a variety of generative models, and already implements multiple LLM tasks and metrics. This section first reviews three tasks involved in RTL generation that TuRTLe focuses on: single-line completion, module completion, and specification-to-RTL conversion, all in § II-A . Next, it outlines the five RTL design goals assessed by TuRTLe : line-level contextual accuracy, syntax correctness, functional correctness, synthesizability, and post-synthesis quality (see § II-B ). To evaluate LLM performance on these goals, we employ several metrics, including exact matching, Pass@ 1, and a novel PPA-Score (detailed in § II-C ). Table I shows the relationships between the various tasks, design goals, metrics, and tools used in the TuRTLe framework. II-A Generation Tasks Our evaluation framework assesses LLMs across three fundamental RTL code generation tasks, each representing a distinct level of complexity and scale. The smallest of tasks, in the sense of generation length, is Single-Line Completion (SLC) , which focuses on the model’s ability to predict the next line of code given a partial context. This task closely resembles auto-completion scenarios, where engineers rely on coding assistants to streamline development. The second task in complexity is Module Completion (MC) , which requires LLMs to generate a complete module based on a given function description or

Read the complete text