A Novel Mutation Based Method for Detecting FPGA Logic Synthesis Tool Bugs
Yi Zhang, He Jiang, Xiaochen Li, Shikai Guo, Peiyu Zou, Zun Wang · 2025
Read A Novel Mutation Based Method for Detecting FPGA Logic Synthesis Tool Bugs by Yi Zhang, He Jiang, Xiaochen Li, Shikai Guo, Peiyu Zou, Zun Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-15536.txt.
Introduction
Yi Zhang and co-authors present VERMEI, a mutation-based method to find bugs in FPGA logic synthesis tools. These tools turn hardware descriptions such as Verilog into gate-level forms, and their faults can cause wrong behaviour and security risk. Past test methods make programs too simple in logic to stress the tools well.
VERMEI finds dead logic in seed programs, then prunes or inserts fragments there to build equal variants with complex control flow, sampling fragments from past designs by a Bayesian plan. Comparing outputs of seed and variant by differential testing flags bugs. On Yosys, Vivado and Quartus it beat past methods, and in five months it reported fifteen bugs, nine confirmed as new.
Catalog record
| CREATOR | Yi Zhang, He Jiang, Xiaochen Li, Shikai Guo, Peiyu Zou, Zun Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 72 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-15536
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Novel Mutation Based Method for Detecting FPGA Logic Synthesis Tool Bugs Yi Zhang, He Jiang, Xiaochen Li, Shikai Guo, Peiyu Zou, Zun Wang Abstract FPGA (Field-Programmable Gate Array) logic synthesis tools are the key components in the EDA (Electronic Design Automation) toolchain, which convert hardware designs written in description languages (e.g., Verilog) into gate-level representations for FPGAs. However, defects in these tools may lead to unexpected behaviors and pose security risks. Therefore, it is crucial to harden these tools through testing. Although several methods have been proposed to automatically test FPGA logic synthesis tools, the challenge still need to be addressed, namely the insufficient semantic logical complexity of test program. In this paper, we propose VERMEI, a new method for testing FPGA logic synthesis tools. VERMEI consists of three modules: preprocessing, equivalent mutation, and bug identification. The preprocessing module identifies zombie logic (inactive code with no impact on the circuit’s output) in seed programs through simulation and coverage analysis. The equivalent mutation module generates the equivalent variants of seed programs by pruning or inserting logic fragments in zombie areas. It uses Bayesian sampling to extract logic fragments from historical Verilog designs for insertion, making the generated equivalent variants have complex control flows and structures, addressing the challenge. The bug identification module, based on differential testing, compares the synthesized outputs of seed and variant programs to identify bugs. Experiments over Yosys, Vivado, and Quartus demonstrate that VERMEI outperforms the state-of-the-art methods in testing FPGA logic synthesis tools. Within five months, VERMEI reported 15 bugs to vendors, 9 of which were confirmed as new bugs. I Introduction FPGA synthesis tools are used to automatically convert high-level design descriptions written in hardware description languages (such as Verilog ) into gate-level netlists that can be implemented on FPGAs. They serve as a critical link between theoretical hardware design and the actual circuit implementation in EDA [ 1 , 2 , 3 , 4 , 5 ] . These tools play a vital role in enabling digital computation and building reliable hardware devices. For instance, the design and implementation of many modern digital systems, including microprocessors, accelerators, and embedded devices, depend on FPGA logic synthesis tools to ensure efficient and accurate hardware realization [ 6 , 4 ] . Like other application software, FPGA logic synthesis tools are prone to bugs [ 7 , 8 , 9 , 10 , 11 ] . These bugs can affect the accuracy of hardware design and may manifest indirectly as FPGA failures, which can lead engineers to mistakenly attribute these hardware failures to bugs within the hardware models themselves rather than to the FPGA logic synthesis tools. To this end, several methods have been proposed to generate random Verilog test programs for testing FPGA logic synthesis tools. The first method, VlogHammer, was introduced by Clifford Wolf [ 12 ] , which defines and samples a subset of Verilog syntax to generate test programs. However, it cannot generate HDL code with multiple modules or behavioral-level constructs like “always” blocks. Verismith [ 13 ] was later proposed as an extension of VlogHammer,uses an Abstract Syntax Tree (AST)-based approach to generate pseudo-random and valid Verilog test programs. However, Verismith is limited in its ability to generate semantically complex test programs, and the deterministic generation patterns often fail to explore deeper corner cases in synthesis tools. EvoHDL [ 14 ] uses the fixed CPS models to transform into various types of HDL programs (i.e., Verilog, VHDL, and SystemVerilog) for testing FPGA logic synthesis tools. However, although the CPS models are converted into different HDL types to create diverse test inputs, the limited module set and the fixed control and data flow patterns constrain the diversity and complexity of the generated programs’ structural and semantic features. Therefore, despite the fact that existing methods have helped developers detect bugs in FPGA logic synthesis tools to some extent, the test programs they generate still have significant room for improvement in bug detection capability. The complexity and variability of real-world FPGA synthesis workflows present major challenges for testing FPGA logic synthesis tools. Challenge: Insufficient Semantic Logical Complexity. Current state-of-the-art Verilog test program generators typically cover only a limited subset of combinatorial patterns within syntax structures. Specifically, Verismith generates test cases by randomly sampling the Verilog syntax specification, while EvoHDL generates structurally valid HDL programs by systematically combining a fixed set of CPS (Continuation-Passing Style) models. Due to the inherent limitations of these generation strategies and configurations, the resulting test programs often have simple structures, containing only basic combinational patterns. This makes it difficult to simulate the diversity and complexity of industrial-scale designs. As a result, such test programs are not effective in uncovering deeper bugs in FPGA logic synthesis tools. Therefore, a key challenge is how to generate Verilog test programs with higher semantic logic complexity to more closely align with the needs of real industrial designs, and thus to test FPGA logic synthesis tools more thoroughly. In this paper, we propose VERMEI ( VER ilog M utation based on E quivalent I nput), an effective technique to generate Verilog test programs with diverse input patterns and higher complexity for detecting bugs in FPGA logic synthesis tools. VERMEI consists of three modules: preprocessing, equivalent mutation, and bug identification. Specifically, the preprocessing module first constructs a robust simulation environment (i.e., a testbench) based on the input variables (including clock signals) of the seed Verilog test program. It then performs simulation and coverage analysis to identify zombie logic (Section II-B ) (inactive code with no impact on the circuit’s output) in the Verilog program. Next, the equivalent mutation module generates equivalent variants of the seed test program by iteratively trimming or inserting new logic fragments in the zombie logic areas. During the insertion process, the equivalent mutation module uses a Bayesian mechanism to sample candidate logic fragments with representative and differentiated characteristics from historical Verilog circuit designs. This process incorporates industry-standard scenarios and various Verilog structural features, such as complex control flow structures (e.g., while, case, and forever loops), nested conditional statements, and intricate computation structures. In this way, the equivalent variants with higher semantic and logical complexity are generated, allowing us to overcome the Challenge . Finally, the bug identification module uses differential testing to validate the generated Verilog variants. It compares the outputs of the seed program and its variants after synthesis under the same stimuli. If a variant’s output differs from the seed program, it is considered to trigger a bug. To assess the effectiveness of VERMEI, we conducted the evaluation over the FPGA logic synthesis tools. First, we compare VERMEI against the state-of-the-art methods (i.e., Verismith and EvoHDL). The results demonstrates that VERMEI achieved a 100% improvement in bug detection effectiveness compared to Verismith and a 60% improvement compared to EvoHDL. Within five months, we reported a total of 15 bugs in the latest versions of various FPGA logic synthesis tools, of which 9 bugs have been confirmed as new bugs. Second, we examine the influence of mutation operations. Using the variants of VERMEI insert (i.e., VERMEI with only inserting operators) and VERMEI prune (i.e., VERMEI with only pruning operators), VERMEI outperforms its variants by detecting 14.69% \sim 34.29% more bugs on average.