A Novel Interactive-Guided Differential Testing Approach for FPGA Simulation Debugger Tools

Shikai Guo, Xiaoyu Wang, Xiaochen Li, Zhihao Xu, He Jiang · 2025

Read A Novel Interactive-Guided Differential Testing Approach for FPGA Simulation Debugger Tools by Shikai Guo, Xiaoyu Wang, Xiaochen Li, Zhihao Xu, He Jiang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-01138.txt.

Introduction

Chip design tools that simulate and debug programmable chips carry a heavy trust load, since a wrong hint can send an engineer to fix the wrong line. Shikai Guo and four co-authors build a test method for such tools in 2025, aimed at a widely used commercial suite. Their method, named DB-Hunter, sets two runs side by side and hunts for gaps.

A test must cover both the chip design and a chain of debug acts, such as setting pause points and stepping through code. That dual need makes the tools hard to check. The new method guides the test with user input as it goes, which helps it reach deep states. Finding bugs here matters for fields from telecoms to cars and spacecraft, where these chips are common.

Catalog record

CREATORShikai Guo, Xiaoyu Wang, Xiaochen Li, Zhihao Xu, He Jiang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-01138

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Novel Interactive-Guided Differential Testing Approach for FPGA Simulation Debugger Tools Shikai Guo, Xiaoyu Wang, Xiaochen Li, Zhihao Xu, He Jiang Abstract . Field-Programmable Gate Array (FPGA) development tool chains are widely used in FPGA design, simulation, and verification in critical areas like communications, automotive electronics, and aerospace. Commercial FPGA tool chains such as Xilinx’ Vivado aids developers in swiftly identifying and rectifying bugs and issues in FPGA designs through a robust built-in debugger, ensuring the correctness and development efficiency of the FPGA design. Hardening such FPGA chip debugger tools by testing is crucial since engineers might misinterpret code and introduce incorrect fixes, leading to security risks. However, FPGA chip debugger tools are challenging to test as they require assessing both RTL designs and a series of debugging actions, including setting breakpoints and stepping through the code. To address this issue, we propose a interactive differential testing approach called DB-Hunter to detect bugs in Vivado’s FPGA chip debugger tools. Specifically, DB-Hunter consists of three components: RTL design transformation component, debug action transformation component, and interactive differential testing component. By performing RTL design and debug action transformations, DB-Hunter generates diverse and complex RTL designs and debug actions, to thoroughly test the Vivado debugger using interactive differential testing to detect bugs. In three months, DB-Hunter reported 18 issues, including 10 confirmed as bugs by Xilinx Support, 6 bugs had been fixed in last version. DOI: 10.1145/1122445.1122456 Journal: JACM Volume: 1 1 1 1 1. Introduction Field-Programmable Gate Array (FPGA) Engineers rely heavily on complex simulation verification tools to design FPGA chips (e.g., MicroBlaze and ARM Cortex-M) ( 1 ) to utilize in crucial domains such as communication, automotive electronics, and aerospace ( 2 , 3 , 4 ) . Such FPGA chip debugger tools could help developers in swiftly identifying and rectifying bugs in FPGA designs through a robust built-in debugger, which assists engineers in the rapid and precise development and debugging of FPGA projects. Commercial FPGA chip debugger tools, such as the built-in debugger in Xilinx’s Vivado, enable developers to directly trace the runtime behavior of FPGA designs. It allows engineers to pause execution at points of interest using breakpoints, inspect the intermediate program state (such as the values of local variables), and closely follow the control flow through single-stepping. When the debugger of the FPGA chip debugger tools have a bug, it can cause engineers to make incorrect changes to the program, which leads to the potential risk. In addition, as the scale of FPGA usage increases and the complexity of logic circuits customizable, hardening such FPGA chip debugger tools (such as Vivado debugger) by testing is crucial since engineers might misinterpret code and introduce incorrect fixes, leading to development bugs or inject subtle unexpected behaviors ( 5 , 6 ) . (a) Initial test run (b) Follow-up test run Figure 1 . A bug about breakpoint misplacement An example of this type of bug is shown in Figure 1 ( : expected breakpoint location, : actual breakpoint location) 1 1 1 https://support.xilinx.com/s/feed/0D54U00007QysW8SAJ?language=en_US . We anticipate placing a breakpoint at a certain location, but due to the Vivado debugger bug, the breakpoints do not move along with the program after inserting blank lines or comments. This behavior may lead engineers to mistakenly believe that lines with breakpoints have not been executed at all, resulting in incorrectly changes to the FPGA design code. Specifically, Figure 1 (a) shows a breakpoint inserted at line 3. Subsequently, a blank line is added after line 1, causing the code below line 1 to shift down by one line, and the breakpoint should move down with it. However, during actual execution, we observed that the breakpoint at line 3 did not move along with the original code at line 3, as shown in Figure 1 (b). Since the Vivado debugger inaccurately describes the actual execution of the FPGA design code, engineers may make changes to the program for worse (e.g., error fixes). One effective approach for detecting bugs in debuggers is through automated or random testing. In this process, test inputs are randomly generated, and a test oracle ( 7 ) is employed to assess the success or failure of the tests. Notably, automation technology tailored for the Chromium browser’s Javascript debugger has recently been introduced ( 8 ) . The technique randomly generates debug actions, such as setting breakpoints, resuming execution, or single-stepping through code, based on syntax. Then, these actions are applied to a specified program slated for debugging. Finally, a differential testing is used to compare the debug traces output by the debugger. However, the differing syntactic structures and statement execution sequences between JavaScript and Verilog languages render the conversion approaches applicable to debugging JavaScript debuggers ineffective for Verilog debuggers. Therefore, a specialized conversion approach specifically tailored for the Verilog language is required. However, due to the interactive nature of the debugger, debug actions persist throughout the entire debugging session. The results in debug action transformations sometimes rely on the current state of the debugger and necessitate interactive transformations of debug actions during the testing process. How to build equivalent debugging operations suitable for debugger testing during debugging is a significant challenge. To overcome these challenges, we propose DB-Hunter, a approach for bug detection in Vivado debugger tool using interactive differential testing. DB-Hunter consists of three main components: RTL design transformation component, debug action transformation component, and interactive differential testing component. By transforming RTL designs and debugging actions separately, DB-Hunter generates a more varied and complex set of RTL design variants. The increased complexity introduced by these RTL design variants may lead to the triggering of more bugs during the debugging process in the Vivado debugger. Since these RTL design variants do not affect the functionality of the original RTL design and the Vivado debugger, they are considered equivalent. Finally, using the variant files generated post-program and debug action transformations, we rerun Vivado and record debugging traces. If DB-Hunter detects inconsistencies in the output of RTL design and debug actions, it may indicate the presence of bugs in the debugger. In addition, DB-Hunter is iterative, wherein the RTL design variants generated by one transformation mode can serve as input for other types of transformations. We can thus apply the transformations a second time on the follow-up test run to obtain yet another test input. With these new inputs, we can apply program and action transformations again, or even continue indefinitely. If the comparison on the debugging traces is successful, iterative test continues by transforming the inputs of the follow-up RTL design once more with a randomly selected transformation. This process continues until the RTL design fails (and a potential bug is found) or until we reach a maximum number of iteration rounds. Experiments shows that DB-Hunter excels in detecting bugs in the Vivado debugger. In three months, DB-Hunter has detected 18 bugs within Vivado, where 10 have been officially confirmed by Xilinx 2 2 2 https://support.xilinx.com/s/topic/0TO2E000000YKY4WAO/simulation-verification?language=en_US as new bug issues, and 6 has been fixed. In addition, we also verified the ability of the DB-Hunter transformation component and different iteration numbers to detect Vivado debugger bugs. This article presents the following contributions: • To the best of our knowledge, we are the

Read the complete text