HWE-Bench: Benchmarking LLM Agents on Real-World Hardware Bug Repair Tasks
Fan Cui, Hongyuan Hou, Zizhang Luo, Chenyun Yin, Yun Liang · 2026
Read HWE-Bench: Benchmarking LLM Agents on Real-World Hardware Bug Repair Tasks by Fan Cui, Hongyuan Hou, Zizhang Luo, Chenyun Yin, Yun Liang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-14709.txt.
Introduction
Cui and four colleagues introduce HWE-Bench, the first repository benchmark for language-model agents fixing hardware bugs. Its 417 tasks come from bug-fix pull requests across six projects spanning Verilog, SystemVerilog, and Chisel. They cover RISC-V cores, systems-on-chip, and security roots of trust, each checked by project simulations. The best of seven models in four frameworks solved 70.7 percent, topping 90 percent on small cores but falling below 65 percent on complex chips. Failures centered on fault localization, hardware reasoning, and coordination across code, configuration, and verification files.
Catalog record
| CREATOR | Fan Cui, Hongyuan Hou, Zizhang Luo, Chenyun Yin, Yun Liang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39217 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.14709
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
HWE-Bench: Benchmarking LLM Agents on Real-World Hardware Bug Repair Tasks Fan Cui, Hongyuan Hou, Zizhang Luo, Chenyun Yin, Yun Liang Abstract . Existing benchmarks for hardware design primarily evaluate Large Language Models (LLMs) on isolated, component-level tasks such as generating HDL modules from specifications, leaving repository-scale evaluation unaddressed. We introduce HWE-Bench , the first large-scale, repository-level benchmark for evaluating LLM agents on real-world hardware bug repair tasks. HWE-Bench comprises 417 task instances derived from real historical bug-fix pull requests across six major open-source projects spanning both Verilog/SystemVerilog and Chisel, covering RISC-V cores, SoCs, and security roots-of-trust. Each task is grounded in a fully containerized environment where the agent must resolve a real bug report, with correctness validated through the project’s native simulation and regression flows. The benchmark is built through a largely automated pipeline that enables efficient expansion to new repositories. We evaluate seven LLMs with four agent frameworks and find that the best agent resolves 70.7% of tasks overall, with performance exceeding 90% on smaller cores but dropping below 65% on complex SoC-level projects. We observe larger performance gaps across models than commonly reported on software benchmarks, and difficulty is driven by project scope and bug-type distribution rather than code size alone. Our failure analysis traces agent failures to three stages of the debugging process: fault localization, hardware-semantic reasoning, and cross-artifact coordination across RTL, configuration, and verification components, providing concrete directions for developing more capable hardware-aware agents. Code: pku-liang/hwe-bench Data: henryen/hwe-bench Resource links to the HWE-Bench GitHub repository and Hugging Face dataset page. 1. Introduction The potential of Large Language Models (LLMs) in hardware design and Electronic Design Automation (EDA) has attracted growing interest ( 19 ) . Existing benchmarks such as VerilogEval ( 11 ) and RTLLM ( 14 ) have driven progress in hardware-oriented models including RTLCoder ( 12 ) , CodeV ( 26 ) , and OriGen ( 5 ) , but focus predominantly on component-level code generation from natural-language specifications. CVDP ( 20 ) advances beyond generation to encompass verification, debugging, and agentic interaction, yet its tasks are individual design problems isolated from their project context, without requiring the agent to navigate a full repository or invoke native verification flows. None of these benchmarks addresses the complexities of repository-level engineering, where real-world hardware projects demand reasoning across heterogeneous artifacts including RTL source, verification components, IP configurations, and build scripts, as well as interaction with project-native build and simulation flows. A fundamental question therefore remains open: can LLM-based agents perform repository-level bug repair within the full context of a real hardware project? Repository-level, execution-grounded evaluation has driven substantial progress in software engineering, where benchmarks such as SWE-bench ( 8 ) and its extensions ( 6 ; 15 ) enabled standardized, reproducible evaluation of agent capabilities. No comparable infrastructure exists for hardware, and building it poses distinct challenges. Open-source HDL repositories are scarce: Verilog and SystemVerilog comprise 100–1,000 \times fewer files than Python in standard code corpora ( 13 ) , let alone hardware DSLs such as Chisel. Even when suitable repositories are identified, constructing reproducible, verifiable benchmark instances remains difficult: where software benchmarks can rely on standardized build and test infrastructure, hardware verification depends on heterogeneous, often proprietary toolchains, and many bugs lack automated regression tests. Establishing reproducible validation for each task instance therefore demands substantial manual effort. Figure 1 . Overview of a task instance in HWE-Bench . Overview diagram of the HWE-Bench task workflow. To address this gap, we introduce HWE-Bench , the first large-scale, repository-level benchmark for evaluating LLM agents on real-world hardware bug repair tasks. As illustrated in Figure 1 , each task instance provides an LLM agent with a bug report and the full hardware repository; the agent interacts with the codebase through tools such as file reading, editing, and shell execution to produce a patch, which is then validated through the project’s native simulation suite in a containerized environment. HWE-Bench comprises 417 task instances curated from six major open-source projects (OpenTitan, Caliptra, XiangShan, Ibex, CVA6, and Rocket Chip), spanning both traditional HDLs (Verilog/SystemVerilog) and a hardware DSL (Chisel), with diverse design categories from RISC-V cores to system-on-chip platforms and security roots-of-trust. Each task is grounded in a fully containerized, execution-based evaluation environment. The benchmark is constructed through a systematic, largely automated pipeline that filters genuine hardware bug fixes from repository noise and establishes reproducible, containerized validation for each instance. We evaluate seven LLMs, both proprietary and open-source, using four agent frameworks across all six repositories. The best-performing agent achieves an overall resolved rate of 70.7%, but performance varies considerably: notably, we observe larger performance gaps between proprietary and open-source models than commonly reported on software benchmarks, suggesting that hardware tasks expose capability differences that software-focused evaluations do not capture. Across repositories, we observe a clear difficulty gradient governed primarily by project scope and bug-type distribution rather than code size alone, with resolved rates ranging from above 90% on smaller projects to below 65% on the largest SoC-level project. Our failure analysis identifies three recurring failure modes, fault localization, hardware-semantic reasoning, and cross-artifact coordination, as well as distinctive model-specific behavioral differences and repository-specific challenges. In summary, our contributions are as follows: • We present HWE-Bench , the first repository-level, execution-grounded benchmark for hardware bug repair, comprising 417 tasks across six repositories spanning both traditional HDLs (Verilog/SystemVerilog) and a hardware DSL (Chisel). • We develop a largely automated benchmark construction pipeline, enabling efficient expansion to new hardware repositories. • We evaluate seven LLMs with four agent frameworks across all repositories, revealing larger capability gaps between proprietary and open-source models than observed on software benchmarks, and identifying hardware-specific failure modes that inform future agent design. 2. Related Work Table 1 . Comparison of benchmarks for hardware and software engineering tasks. Benchmark Domain Level Task Type Languages Verification Environment VerilogEval ( 11 ) Hardware Component Code Generation Verilog/SV Simulation Isolated RTLLM ( 14 ) Hardware Component Code Generation Verilog/SV Simulation Isolated HWFixBench ( 7 ) Hardware File Bug Repair Verilog/SV LLM-as-Judge Isolated CVDP ( 20 ) Hardware Component Multi-task Verilog/SV Simulation Isolated SWE-bench ( 8 ) Software Repository Bug Repair Python Unit Tests Containerized SWE-bench Pro ( 6 ) Software Repository Bug Repair Multi-language Unit Tests Containerized HWE-Bench (Ours) Hardware Repository Bug Repair Verilog/SV + Chisel Simulation Containerized Hardware Engineering Benchmarks. Evaluation of LLMs for hardware has progressed from code generation to broader engineering tasks, as shown in Table 1 . Early benchmarks such as VerilogEval ( 11 ) and RTLLM ( 14 ) focused on generating individual modules from specifications, providing simulation-based correctness checks at the component level. CVDP ( 20 ) broadened both the task scope (adding verification, debugging, and agentic formats) and scale (783 problems), while HWFixBench ( 7 ) moved toward bug repair using real repository data. However, CVDP tasks are problem-scoped rather than repository-scoped: even in agentic settings, agents do not operate over the original project repository or its native regression flow. HWFixBench provides file-level inputs with LLM-based semantic matching rather than execution-based validation. HWE-Bench addresses both limitations by embedding tasks in full repositories and validating patches through native simulation in containerized environments. Software Engineering Benchmarks. Repository-level, execution-grounded evaluation was established most visibly in software engineering with SWE-bench (