Insights from Verification: Training a Verilog Generation LLM with Reinforcement Learning with Testbench Feedback
Ning Wang, Bingkun Yao, Jie Zhou, Yuchen Hu, Xi Wang, Nan Guan, Zhe Jiang · 2025
Read Insights from Verification: Training a Verilog Generation LLM with Reinforcement Learning with Testbench Feedback by Ning Wang, Bingkun Yao, Jie Zhou, Yuchen Hu, Xi Wang, Nan Guan, Zhe Jiang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-15804.txt.
Introduction
Ning Wang and co-authors train a chip-design language model on the verdicts of its own testbenches in a 2025 paper. Language models write Verilog, the hardware description language, fluently, but fluent code that computes the wrong thing is worthless. Functional correctness, checked by simulation, is the true goal of hardware design.
The bottleneck was missing test data. Their pipeline builds testbenches automatically, decomposing the job and using compiler-simulator feedback to keep the benches honest. Generated code then runs against those benches, and the pass-fail outcomes form preference pairs. Direct preference optimization trains the model to favor code that works over code that merely reads well. Across five public benchmarks the method beat leading baselines on functional correctness. The team open-sources its code, data, and models.
Catalog record
| CREATOR | Ning Wang, Bingkun Yao, Jie Zhou, Yuchen Hu, Xi Wang, Nan Guan, Zhe Jiang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-15804
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Insights from Verification: Training a Verilog Generation LLM with Reinforcement Learning with Testbench Feedback Ning Wang, Bingkun Yao, Jie Zhou, Yuchen Hu, Xi Wang, Nan Guan, Zhe Jiang Abstract Large language models (LLMs) have shown strong performance in Verilog generation from natural language description. However, ensuring the functional correctness of the generated code remains a significant challenge. This paper introduces a method that integrates verification insights from testbench into the training of Verilog generation LLMs, aligning the training with the fundamental goal of hardware design: functional correctness. The main obstacle in using LLMs for Verilog code generation is the lack of sufficient functional verification data, particularly testbenches paired with design specifications and code. To address this problem, we introduce an automatic testbench generation pipeline that decomposes the process and uses feedback from the Verilog compiler simulator (VCS) to reduce hallucination and ensure correctness. We then use the testbench to evaluate the generated codes and collect them for further training, where verification insights are introduced. Our method applies reinforcement learning (RL), specifically direct preference optimization (DPO), to align Verilog code generation with functional correctness by training preference pairs based on testbench outcomes. In evaluations on VerilogEval-Machine, VerilogEval-Human, RTLLM v1.1, RTLLM v2, and VerilogEval v2, our approach consistently outperforms state-of-the-art baselines in generating functionally correct Verilog code. We open source all training code, data, and models at https://anonymous.4open.science/r/VeriPrefer-E88B . I Introduction Large language models (LLMs) have shown promising capabilities in various software programming tasks, which has led hardware design researchers to investigate their applications in various hardware design tasks. An task is to use LLMs for automatic generation of hardware description language (HDL) code, such as Verilog, from design specifications written in natural language. Current works still aim to improve the functional correctness of open-source models. In this paper, we propose a method that incorporates verification insights provided by testbenches into the training of Verilog generation LLMs. Although the use of verification knowledge is not new and has been explored in previous work, such as OriGen [ 6 ] , which applies self-reflection with compiler feedback to fix code and generate high-quality training data, our approach is the first to align LLMs with the fundamental goal of Verilog generation: functional correctness. The main challenge in leveraging verification feedback for LLM training is the lack of sufficient functional verification data, specifically testbenches. Publicly available datasets rarely include detailed testbenches paired with design specifications and code, making it difficult to directly apply verification-based supervision. To address this limitation, we designed an automatic testbench generation pipeline given the Verilog code and design specification. This pipeline follows the principles of decomposition and feedback. Decomposition breaks down the entire pipeline into smaller subtasks to prevent LLMs from being overwhelmed by task complexity. Feedback is provided through the verilog compiler simulator (VCS), a simulation and debugging tool developed by Synopsys for Verilog designs, which verifies correctness in time and provides different types of feedback to reduce the hallucination. This pipeline collects verification insights and serves as a complementary source of knowledge from a different stage in the hardware design process. Although this pipeline cannot guarantee testbenches that cover all functional cases, it still supports training by providing basic functional verification. Fig. 1 : Overview of our work. We first use paired design specification and Verilog code to automatically generate testbenches. Then we prompt the fine-tuned model and test the generated code using the testbenches to collect verification insights. The code that passes more testcases is considered as preferred, and the other as less preferred. Finally, the design specification with the preference pairs is used for reinforcement learning. Our approach uses reinforcement learning (RL) to better align the Verilog generation model with functional correctness. We use RL because SFT only encourages the model to repeat patterns from the training data, which does not guarantee functional correctness. To address this, we prompt the fine-tuned LLM and collect generated code, then use testbenches to obtain preference pairs, where the preferred code passes more testcases and the less preferred code passes fewer. We apply direct preference optimization (DPO), an RL algorithm that learns directly from these code pairs without modeling a reward function explicitly. This choice is motivated by the fact that reward-based RL is prone to reward hacking, where LLM exploits unintended shortcuts in the reward function [ 1 ] . As a result, the model is not limited to repeating frequent patterns in the dataset, but is guided to produce outputs that better satisfy functional correctness. The choice of using DPO will be validated in Section V-C3 and Section V-D . We evaluated our approach on three established benchmarks: VerilogEval-Machine, VerilogEval-Human and RTLLM v1.1, and found that our LLMs consistently outperform the state-of-the-art baselines of the general, coding and Verilog generation LLMs. The application of DPO leads to substantial improvements across all variants of the model with different structures, families, and sizes on all benchmarks. We further validate our approach on the advanced RTLLM v2 and VerilogEval v2 benchmarks, where the design specifications are more consistent with those used by HDL engineers, and our LLMs continue to outperform all state-of-the-art baselines. The gains achieved demonstrate that reinforcement learning with testbench feedback can better align the model with hardware design tasks to generate functionally correct Verilog code. The ablation study demonstrates that our approach achieves the best performance compared to various alternative strategies to construct preference pairs and RL algorithms. Furthermore, the ablation study shows that SFT with verified data does not achieve the same improvements as RL, since the training objective of SFT is perplexity, which measures the low-level cumulative token probability rather than the overall functional assessment of the entire code. The contributions of this paper are as follows. • We propose an automatic testbench generation pipeline that decomposes the construction process and incorporates real verification feedback to ensure practicality and robustness. • We integrate verification insights into the training process, addressing a key aspect of the hardware design task that is often neglected by previous work. • We employ reinforcement learning with testbench feedback to better use verification insights, enabling the model to generate functionally correct Verilog code. • We release all code, data, and the final model to promote further research and provide a strong baseline on established benchmarks. II Related Work II- 1 Verilog Generation LLMs Many studies have advanced LLM-based Verilog code generation. [ 19 ] developed RTLCoder that outperforms GPT-3.5 by training on automatically generated datasets using GPT. [ 36 ] introduced BetterV, which creates training datasets by converting Verilog code to the C language, allowing LLMs to use their knowledge of general-purpose programming languages. CodeV [ 39 ] used multilevel summarization, reversing the traditional process by generating natural language descriptions from high-quality Verilog code. Furthermore, AutoVCoder [ 7 ] presented a systematic framework to improve Verilog code generation correctness through the generation of high-quality hardware datasets and the fine-tuning of two-round LLM. Furthermore, HaVen [ 34 ] addressed hallucinations in Verilog generation by classifying HDL-specific hallucinations and aligning LLM frameworks with engineering practices. OriGen [ 6 ] improved open-source LLM performance using self-reflection and code-to-code augmentation techniques. However, none of these approaches uses verification insights to train a Verilog generation LLM for better