VeriInteresting: An Empirical Study of Model–Prompt Interactions in Verilog Code Generation

Luca Collini, Andrew Hennessee, Patrick Yubeaton, Siddharth Garg, Ramesh Karri · 2026

Read VeriInteresting: An Empirical Study of Model–Prompt Interactions in Verilog Code Generation by Luca Collini, Andrew Hennessee, Patrick Yubeaton, Siddharth Garg, Ramesh Karri (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-08715.txt.

Introduction

Luca Collini and colleagues map how language models and prompts interact in Verilog code generation. They test small and large models, general and specialized, under a controlled factorial design across prompts, structured outputs, rewrites, chain-of-thought, in-context learning, and evolutionary tuning. The study finds which model-prompt patterns hold across two benchmarks and which stay tied to one pairing.

Catalog record

CREATORLuca Collini, Andrew Hennessee, Patrick Yubeaton, Siddharth Garg, Ramesh Karri
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56227
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.08715

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

VeriInteresting: An Empirical Study of Model–Prompt Interactions in Verilog Code Generation Luca Collini, Andrew Hennessee, Patrick Yubeaton, Siddharth Garg, Ramesh Karri Abstract Rapid advances in language models (LMs) have created new opportunities for automated code generation while complicating trade-offs between model characteristics and prompt design choices. In this work, we provide an empirical map of recent trends in LMs for Verilog code generation, focusing on interactions among model reasoning, specialization, and prompt engineering strategies. We evaluate a diverse set of small and large LMs, including general-purpose, reasoning, and domain-specific variants. Our experiments use a controlled factorial design spanning benchmark prompts, structured outputs, prompt rewriting, chain-of-thought reasoning, in-context learning, and evolutionary prompt optimization via Genetic-Pareto. Across two Verilog benchmarks, we identify patterns in how model classes respond to structured prompts and optimization, and we document which trends generalize across LMs and benchmarks versus those that are specific to particular model–prompt combinations. 1 Introduction The landscape of language models (LMs) is advancing rapidly, with new architectures, training paradigms, and inference-time techniques emerging continuously. In software engineering, LMs are dominant productivity tools, substantially improving code generation and iteration. This growth in software capability increases the demand placed on the hardware ecosystem that supports it. Despite this demand, hardware engineers have not seen comparable productivity gains. Hardware design is a constrained, human-driven process, governed by the strict semantics of hardware description languages (HDLs) and complex power, performance, area, and cost trade-offs. Unlike Python, Verilog is verbose and structural, requiring models to maintain long-context coherence for precise signal connectivity. The stakes in hardware are higher; a plausible-looking Verilog module may simulate correctly in isolation, yet violate timing constraints or introduce race conditions that surface in late-stage verification or fabrication, raising reliability concerns. Applying LMs to hardware design is complicated by a scarcity of structured data. Most hardware designs constitute high-value, closed-source intellectual property, limiting open-source training data and restricting retraining or proprietary fine-tuning. Many organizations are reluctant to rely on third-party models or cloud-based services, necessitating inference-time methods that extract maximal performance from existing LMs without access to private data. In this work, we present a large-scale empirical study of LMs for Verilog generation, focusing on the interplay of reasoning capabilities, prompt design, and prompt optimization. We evaluate 18 LMs across two Verilog benchmarks with complementary evaluation protocols (simulation-based testing and formal equivalence checking): 7 commercial large language models (LLMs) and 11 open-source small language models (SLMs), including 4 SLMs fine-tuned for Verilog. Our study spans prompting strategies such as structured prompting ( 11 ) , chain-of-thought reasoning ( 23 ) , in-context learning ( 3 ) , and prompt refinement ( 14 ; 20 ) . Finally, we use Genetic-Pareto (GEPA) prompt optimization ( 1 ) —an inference-only improvement method—to optimize system prompts without violating IP constraints or incurring fine-tuning costs. Our study will identify which trends generalize in Verilog generation and provide practical guidance for deploying LMs in hardware design, answering four research questions: • RQ1: How does model scale compare to domain-specific specialization under limited training data? • RQ2: How sensitive is Verilog generation performance to prompting strategies? • RQ3: Do finetuned LMs outperform strongly prompted general-purpose LMs? • RQ4: How stable are trends across Verilog benchmarks? 2 Related Works Language models have demonstrated strong baseline capabilities on programming and software engineering tasks, including code generation, completion, repair, and reasoning over complex specifications ( 10 ; 7 ) . Emerging research shows that prompt design and inference-time strategies can improve model performance on programming tasks. Many techniques have been explored, including structured prompts that enforce intermediate representations or schemas ( 12 ) , retrieval-augmented prompting that incorporates external documentation ( 27 ) , multi-stage pipelines that rewrite, refine, or debug generated programs ( 5 ) , and model-aware in-context learning ( 13 ) . More recently, automated prompt optimization methods have been proposed that iteratively refine prompts using task-level feedback to improve performance on code generation tasks ( 9 ; 25 ) . These results suggest that performance on coding tasks is shaped not only by model capacity, but also by how effectively the task is specified and scaffolded at inference time. In parallel, there has been growing interest in applying LMs to hardware design workflows, including HDL generation from natural language specifications ( 21 ; 16 ; 17 ) , simulation- and formal-guided verification, and iterative debug loops ( 2 ) . While tasks such as Verilog generation are syntactically similar to conventional code generation, they are semantically distinct; hardware descriptions encode concurrency, timing, and microarchitectural behavior, and are evaluated through downstream synthesis and verification toolchains. As a result, hardware design imposes stricter correctness requirements, deeper dependence on external tools, and heightened sensitivity to subtle semantic errors that may not be apparent from code structure alone. These differences raise a natural question: to what extent do prompting and reasoning techniques that have proven effective in software-centric settings transfer to hardware design tasks? To our knowledge, this question has not yet been systematically examined across models and benchmarks. We address this gap through a controlled, large-scale evaluation of prompting and inference-time optimization strategies for Verilog generation across multiple benchmarks and a diverse set of model classes, including commercial frontier LMs, open-source baselines, and Verilog-specialized LMs. Because these benchmarks differ in task formulation and evaluation protocols, our design enables us to (i) assess robustness of prompting gains across LMs and settings, and (ii) analyze how LM specialization and embedded reasoning influence the effectiveness of prompt engineering. 3 Background Generating Register-Transfer Level (RTL) code differs fundamentally from standard software code generation, as hardware correctness requires guarantees across all possible input states rather than validation over a finite test set. Accordingly, Verilog generation benchmarks vary not only in task sources and specification styles, but also in their evaluation protocols. The most common approach is dynamic simulation, in which generated designs are compiled and executed against testbenches, and outputs are compared to a reference for a limited set of inputs. While analogous to software unit testing, simulation provides only partial coverage and may miss corner-case errors. An alternative is formal equivalence checking, which uses symbolic methods to prove that a generated circuit is functionally identical to a trusted reference implementation for all input states. Although this offers a stronger notion of correctness, it relies on a golden design and incurs a higher computational cost. 3.1 Simulation-Based Evaluation: Verilog Eval Verilog Eval ( 15 ) represents the standard for simulation-based evaluation. Originally released in 2023, it comprises 156 tasks from the HDLBits platform, ranging from basic logic gates to complex finite state machines. Verilog Eval uses the Icarus Verilog simulator to check for functional correctness. A generated design is deemed correct if its transient simulation outputs match those of a reference solution across a fixed set of test vectors. Recent updates in “Revisiting Verilog Eval” ( 18 ) have modernized this framework to support full specification-to-RTL generation (rather than just code completion) and added infrastructure to evaluate in-context learning strategies. 3.2 Verification-Based Evaluation: VeriThoughts To overcome the coverage limitations of simulation, VeriThoughts ( 26 ) introduces a benchmark grounded in formal verification. Unlike Verilog Eval, which checks input/output pairs, VeriThoughts uses Yosys ( 24 ) open synthesis

Read the complete text