Speculative Decoding for Verilog: Speed and Quality, All in One
Changran Xu 1,3,† , Yi Liu 1,3,† , Yunhao Zhou 1,3 , Shan Huang 2,3 , Ningyi Xu 2 , and Qiang Xu 1,3 · 2025
Read Speculative Decoding for Verilog: Speed and Quality, All in One by Changran Xu 1,3,† , Yi Liu 1,3,† , Yunhao Zhou 1,3 , Shan Huang 2,3 , Ningyi Xu 2 , and Qiang Xu 1,3 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-14153.txt.
Introduction
Changran Xu and co-authors (2025) speed up language models that write Verilog, the language used to design chips. Verilog appears rarely in training data, and standard tokenizers chop its structures into awkward pieces. Speculative decoding lets a model draft ahead and verify, but the authors retune where decoding pauses. They align the stops with tokens that carry syntax meaning. The model then learns the shape of Verilog code more cleanly.
On the RTLLM benchmark, the method ran up to five times faster than conventional decoding. Functional accuracy on ten tries rose by as much as seventeen points. Speed and quality usually trade against each other; here both improved. The authors argue the trick suits other specialized languages with rigid grammar and thin training data. For chip design tools, faster and more reliable generation could shorten the loop between an engineer's intent and working hardware description.
Catalog record
| CREATOR | Changran Xu 1,3,† , Yi Liu 1,3,† , Yunhao Zhou 1,3 , Shan Huang 2,3 , Ningyi Xu 2 , and Qiang Xu 1,3 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-14153
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Speculative Decoding for Verilog: Speed and Quality, All in One Changran Xu 1,3,† , Yi Liu 1,3,† , Yunhao Zhou 1,3 , Shan Huang 2,3 , Ningyi Xu 2 , and Qiang Xu 1,3 Abstract The rapid advancement of large language models (LLMs) has revolutionized code generation tasks across various programming languages. However, the unique characteristics of programming languages, particularly those like Verilog with specific syntax and lower representation in training datasets, pose significant challenges for conventional tokenization and decoding approaches. In this paper, we introduce a novel application of speculative decoding for Verilog code generation, showing that it can improve both inference speed and output quality, effectively achieving speed and quality all in one. Unlike standard LLM tokenization schemes, which often fragment meaningful code structures, our approach aligns decoding stops with syntactically significant tokens, making it easier for models to learn the token distribution. This refinement addresses inherent tokenization issues and enhances the model’s ability to capture Verilog’s logical constructs more effectively. Our experimental results show that our method achieves up to a 5.05× speedup in Verilog code generation and increases pass@10 functional accuracy on RTLLM by up to 17.19% compared to conventional training strategies. These findings highlight speculative decoding as a promising approach to bridge the quality gap in code generation for specialized programming languages. I Introduction The rapid advancement in large language models (LLMs) has transformed code generation across various programming languages [ 1 , 2 , 3 , 4 ] . These models, driven by advanced pre-training techniques, have achieved notable success in generating syntactically and semantically correct code for widely used languages like Python and C++. However, their application to specialized languages such as Verilog, which is critical for hardware design and verification, remains limited due to the unique challenges posed by Verilog’s syntax intricacies and its underrepresentation in training datasets. Unlike natural languages, programming languages like Verilog are governed by strict syntactic rules, where even minor deviations can lead to significant compilation failures or functional errors. Traditional tokenization methods, such as Byte Pair Encoding (BPE) [ 5 ] , fragment meaningful code structures into subword units, obscuring the logical relationships inherent in Verilog. Consequently, models often struggle to accurately capture the syntax and semantics of Verilog code. Grammar-based approaches attempt to mitigate this issue by representing code as sequences of grammar rules [ 6 , 7 , 8 , 9 , 10 , 11 ] . While effective for smaller models and datasets, these methods face scalability issues when applied to LLMs, due to vocabulary explosion and distribution shifts. Moreover, these methods remain largely unexplored for underrepresented languages like Verilog, which presents an additional layer of complexity. Fig. 1: A brief comparison of the performance and speed of our method against the M edusa method and the conventional next token prediction (NTP) approach. The experiments are conducted using the CodeLlama-7b model, with performance metrics evaluated on the RTLLM benchmark. To address these challenges, we propose a novel application of speculative decoding [ 12 , 13 , 14 , 15 ] for Verilog code generation. While speculative decoding has traditionally been employed to accelerate LLM inference by predicting multiple tokens simultaneously to reduce latency, our key insight is that speculative decoding, when carefully aligned with syntactically significant tokens, can also improve the quality of code generation . By structuring decoding stops around meaningful fragments in Verilog, such as identifiers, keywords, and operators, we enable the model to better capture the logical and structural relationships inherent in the language. This alignment simplifies the learning of Verilog’s token distribution, mitigating issues caused by conventional tokenization schemes and enhancing the syntactic and functional correctness of the generated code. Specifically, the main contributions of our work include: • To the best of our knowledge, this is the first application of speculative decoding aimed not only at improving inference speed but also enhancing output quality. • We propose a simple yet effective method for identifying syntactically significant tokens in Verilog using abstract syntax trees (ASTs) and modify the speculative decoding scheme by incorporating syntax-enriched labels to align decoding stops with these tokens. • Our decoding scheme further unlocks the potential of the original M edusa method [ 15 ] by training it with dynamic labels in a flexible manner, increasing the number of effective heads and further accelerating the speedup. We conduct experiments with two models, CodeT5p-220m-bimodal [ 2 ] and CodeLlama-7b [ 3 ] . The results show that our method achieves up to a 5.05× speedup in Verilog code generation and increases pass@10 functional accuracy on RTLLM [ 16 ] by up to 17.19% compared to conventional training strategies. Impressively, our approach also accelerates model inference by 1.42–2.29× over the original M edusa method, highlighting its ability to deliver both speed and quality simultaneously. II Related Work II-A LLM for Verilog Generation Recent advancements in LLMs have shown significant potential for automating the generation of Verilog code from high-level prompts [ 17 , 18 , 19 , 20 , 21 ] , demonstrating their ability to address the design challenges faced by hardware developers. While some works [ 22 , 23 , 24 , 25 , 26 , 27 ] have fine-tuned open-source models to improve Verilog code generation, they often treat Verilog code as if it were natural language, using tokenizers originally designed for natural language models without modification. This approach often leads to the fragmentation of meaningful Verilog code structures, which hinders the model’s ability to accurately capture the syntactic structure of Verilog and generate syntactically correct code. Furthermore, the scarcity of Verilog code datasets exacerbates the challenge of effectively learning Verilog’s syntax. To address this issue, we propose a novel application of speculative decoding for Verilog code generation. By integrating conventional tokenization with Verilog’s syntactical structure, our method allows the model to generate more accurate Verilog code more efficiently. II-B Syntax-Aware Tokenization for Code Driven by recent developments in LLMs, a variety of pretrained models for programming languages have emerged [ 1 , 2 , 3 , 4 , 22 , 23 , 24 , 25 , 26 , 27 ] . However, most of these models represent code as token sequences using the BPE algorithm for pre-training, which often hinders their ability to capture code’s syntactic structure and fails to ensure the syntactic correctness of the generated output. To address this limitation, some models [ 28 , 29 , 30 ] have attempted to explicitly incorporate syntax by representing code as AST sequences, which are obtained by traversing the AST in pre-order and recording the symbol of each node. However, to maintain the tree structure in the node sequences, these approaches introduce additional nodes, which significantly increase sequence length and GPU memory usage, potentially compromising model performance. Furthermore, these models only leverage the AST representation in the encoder and do not ensure syntactic correctness during code generation. Meanwhile, several non-pretrained code generation models [ 6 , 7 , 8 , 9 , 10 ] have used grammar rule sequences to represent code. Specifically, they traverse the AST in pre-order and record the grammar rules used to expand each non-terminal. The integration of grammar rules has been shown to improve performance. However, these models have