Comprehensive Verilog Design Problems: A Next-Generation Benchmark Dataset for Evaluating Large Language Models and Agents on RTL Design and Verification
Nathaniel Pinckney, Chenhui Deng, Chia-Tung Ho, Yun-Da Tsai, Mingjie Liu, Wenfei Zhou, Brucek Khailany, Haoxing Ren · 2025
Read Comprehensive Verilog Design Problems: A Next-Generation Benchmark Dataset for Evaluating Large Language Models and Agents on RTL Design and Verification by Nathaniel Pinckney, Chenhui Deng, Chia-Tung Ho, Yun-Da Tsai, Mingjie Liu, Wenfei Zhou, Brucek Khailany, Haoxing Ren (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-14074.txt.
Introduction
Nathaniel Pinckney and co-authors present a benchmark for language models on chip design tasks in a 2025 paper. The collection holds hundreds of problems across thirteen categories, written by experienced hardware engineers. Tasks cover writing register transfer level code, checking it, debugging, matching specifications, and technical questions. Each problem comes in a plain format and in an agent format where tools can be used.
Even the strongest current models passed no more than about a third of the code generation problems on the first try. Agent tasks, especially those reusing existing designs or verifying them, were particularly hard. Scoring uses open source tools plus model based judging for comprehension questions. The authors say the gaps show how far models remain from dependable real world hardware design automation.
Catalog record
| CREATOR | Nathaniel Pinckney, Chenhui Deng, Chia-Tung Ho, Yun-Da Tsai, Mingjie Liu, Wenfei Zhou, Brucek Khailany, Haoxing Ren |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-14074
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Comprehensive Verilog Design Problems: A Next-Generation Benchmark Dataset for Evaluating Large Language Models and Agents on RTL Design and Verification Nathaniel Pinckney, Chenhui Deng, Chia-Tung Ho, Yun-Da Tsai, Mingjie Liu, Wenfei Zhou, Brucek Khailany, Haoxing Ren Abstract We present the Comprehensive Verilog Design Problems (CVDP) benchmark, a new dataset and infrastructure to advance LLM and agent research in hardware design and verification. CVDP includes 783 problems across 13 task categories, covering RTL generation, verification, debugging, specification alignment, and technical Q&A authored by experienced hardware engineers. Problems are offered in both non-agentic and agentic formats. The benchmark introduces more realistic and challenging contexts than prior work, with state-of-the-art models achieving no more than 34% pass@1 on code generation. Agentic tasks—especially those involving RTL reuse and verification—are particularly difficult. Evaluation uses open-source tools and model scoring infrastructure, with comprehension tasks assessed via BLEU and LLM-based judging. CVDP reveals substantial gaps in current model capabilities, underscoring the need for continued research toward robust, real-world hardware design automation. 1 Introduction Large language models (LLMs) have seen widespread adoption in software development for code generation, bug fixing, question answering, test generation, and related tasks. Recently, agentic assistants such as Cursor ( Anysphere Inc. 2025 )—an AI-powered IDE based on Visual Studio Code—have gained traction for their ability to not only answer questions but also perform complex code edits and execute commands. By contrast, semiconductor hardware design has not benefited as significantly from LLMs. Generating Verilog RTL (Register-Transfer Level—the textual code used to design digital logic chips) with LLMs presents unique challenges, including the limited availability of high-quality training data ( Wang et al. 2025 ; Liu et al. 2025a ) and the relative recency of domain-specific benchmarks. Two widely used datasets are VerilogEval ( Liu et al. 2023 ; Ho et al. 2024 ) and RTLLM ( Lu et al. 2023 ; Liu et al. 2025b ), which report pass rates as high as 63% on GPT-4 and 94% for agentic approaches ( Pinckney et al. 2025 ; Ho et al. 2024 ). However, these benchmarks are narrow in scope and do not reflect the full complexity of hardware development workflows. Moreover, their high pass rates leave little headroom for measuring future improvements, limiting their usefulness as research drivers. VerilogEval and RTLLM rely on hand-crafted prompts and evaluate on small, self-contained problems. RTL-Repo ( Allam and Shalan 2024 ) introduces more realistic GitHub-derived contexts, prompting LLMs to complete redacted code regions. While it captures real-world structure, RTL-Repo focuses solely on code completion and does not test broader challenges like specification-to-RTL generation, debugging, or verification. Related benchmarks cover testbench stimuli ( Zhang et al. 2023 ), though close to 100% coverage of their benchmark is achievable by Claude 3.5 Sonnet, and formal assertions ( Liu et al. 2025b ). We introduce the Comprehensive Verilog Design Problems (CVDP) benchmark, which expands on prior work with broader task coverage and greater depth. CVDP includes 783 human-authored problems across 13 categories, including RTL generation, design verification, debugging, assertion creation, and technical comprehension. Tasks are provided in both Non-Agentic (single-turn) and Agentic (multi-turn, tool-using) formats. Previous benchmarks focus on single-turn prompts and evaluation infrastructure, while CVDP is designed to evaluate agents, with support for tool interaction, iterative workflows, and complex reasoning. CVDP addresses the growing need for benchmarks that reflect real-world hardware development. Problem categories cover tasks such as RTL/testbench generation, debugging, assertions, code modification, power and area optimization, question answering, and code-spec alignment. The dataset is intended to expand over time, evolving alongside improvements in LLM and agent capabilities, while continuing to offer meaningful challenge and headroom for future research. This work makes four key contributions: 1. The first agentic-oriented benchmark for Verilog RTL code generation, verification, and related tasks. The benchmark’s prompts and infrastructure are designed to evaluate Dockerized LLM-based agents on real-world problems with EDA tool use. 2. A broader benchmark that encompasses a wider range of hardware design and verification tasks. The benchmark is intended to support both model and agent research. Initial Non-Agentic categories were selected with greater agent workflows in mind, representing useful subtasks within larger design processes. 3. A more challenging benchmark , featuring tasks significantly more difficult than those in VerilogEval ( Liu et al. 2023 ; Pinckney et al. 2025 ) and RTLLM ( Lu et al. 2023 ). Prior benchmarks largely drew from public repositories and are increasingly saturated, with high pass rates from both models and agents. In contrast, the current benchmark offers data points crafted and QA’ed by experienced hardware engineers with more than 4 years of experience from scratch. As a result, we show that state-of-the-art models—including Claude 3.7 Sonnet, GPT-4.1, and LLaMA 3.1 405B—achieve no more than a 34% pass rate on code generation questions in our benchmark, providing substantial headroom for future research in LLM-driven hardware design. 4. Analysis of model failures examines why state-of-the-art models frequently fail across specific categories and offers insights into the key capabilities LLMs must develop before they can be reliably deployed for real-world hardware design and verification. RTL code represents only a small fraction of public GitHub repositories compared to software code, and much design knowledge remains proprietary within industry. Consequently, there is a strong need for an advanced, human-written, publicly available benchmark dataset—composed of real-world design problems authored by design and verification experts. We created CVDP to address this critical gap. 1 1 1 CVDP is available at https://github.com/NVlabs/cvdp_benchmark . 2 CVDP Dataset The CVDP dataset and infrastructure build on methodologies from software LLM benchmarks such as SWE-bench ( Jimenez et al. 2024 ) and Microsoft’s Copilot evaluation harness ( Agarwal et al. 2024 ). Whereas SWE-bench had access to a wide range of high-quality, open-source, software code repositories and well-documented resolved GitHub issues to pull from, similar high-quality RTL repositories are not as available in the open-source domain. Instead, we engaged a team of approximately 35 hardware engineers with more than 4 years of Verilog and verification experience to author problems across 13 task categories and difficulty levels, in both Non-Agentic and Agentic formats. In addition, subject matter experts with doctoral degrees in hardware design and/or engineering management experiences also reviewed each problem for accuracy, task fit, and appropriate scope, with intensive manual review during initial calibration batches to ensure data quality and task alignment. Once categories stabilized, LLM-based filtering was used to catch errors, such as missing context or incorrect category, and score ambiguity and consistency of the prompt. Sanity checks ensured all reference solutions passed and incomplete contexts failed as expected. Of the 1,313 problems written, 783 were retained after quality filtering described in Section 3 . As with any codebase, a benchmark cannot be entirely bug-free ( Ho et al. 2024 ). Errors may cap maximum achievable scores, and updated benchmark versions will be released as needed. Each datapoint, or “problem,” represents a multi-file repository extracted at evaluation time. A test harness—typically a CocoTB ( CocoTB 2025 ) simulation script—assesses correctness based on task type. CocoTB is a Python verification framework for testing RTL, and helps to automate the test harness. BLEU ( Papineni et al. 2002 ) scoring is used where code or natural language snippets are expected verbatim, while technical natural language