A Comparative Study of Fuzzers and Static Analysis Tools for Finding Memory Unsafety in C and C++
Keno Hassler, Philipp Görz, Stephan Lipp · 2025
Read A Comparative Study of Fuzzers and Static Analysis Tools for Finding Memory Unsafety in C and C++ by Keno Hassler, Philipp Görz, Stephan Lipp (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-22052.txt.
Introduction
Keno Hassler, Philipp Görz, and Stephan Lipp pit two bug-hunting crafts against each other in a 2025 security paper. Memory safety flaws cause over seventy percent of serious software vulnerabilities. Fuzzers hunt them by flooding programs with random inputs at run time, while static analyzers reason over source code without running it. The team ran five analyzers and thirteen fuzzers over more than one hundred known flaws in C and C++ programs.
Fuzzers found a strikingly similar set of bugs to one another, while analyzers reported more varied catches. Best of all, the two crafts barely overlapped, so together they found far more than either alone. The manual checking of which reports mapped to which flaws grounds the comparison.
Catalog record
| CREATOR | Keno Hassler, Philipp Görz, Stephan Lipp |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 69 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-22052
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Comparative Study of Fuzzers and Static Analysis Tools for Finding Memory Unsafety in C and C++ Keno Hassler, Philipp Görz, Stephan Lipp Abstract . Over 70% of security vulnerabilities in critical software systems today result from memory safety violations. To address this challenge, fuzzing and static analysis are widely used automated methods to discover such vulnerabilities. Fuzzing generates random program inputs to identify faults at runtime, while static analysis reasons about the code to detect potential vulnerabilities. Although these techniques share a common goal, they take fundamentally different approaches and have evolved largely independently. In this paper, we present an empirical analysis of five static analyzers and 13 fuzzers, applied to over 100 known security vulnerabilities in C/C++ programs. We measure the detection rate for each tool and vulnerability to evaluate how the approaches differ and complement each other. We find that fuzzers discover a very similar set of bugs, while static analyzers report more diverse sets, and identify clear leaders for each group. Comparing the union of all fuzzers with that of all static analyzers, we observe they are nearly disjoint. In a second step, we manually validate the report-to-bug mapping we developed for the evaluation and discuss more qualitative aspects of limitations, usability, and integration into the development process. We examine how widely these bug finding tools are used in critical open-source projects. We advise developers on choosing tools to harden their software and identify barriers to adoption as well as future research opportunities. 1. Introduction In recent years, more and more security vulnerabilities have been reported in critical software systems. Many of these vulnerabilities are related to memory unsafety. For instance, a third of all recorded security vulnerabilities (tracked as CVEs) and seven of the 20 most frequently reported bug classes (tracked as CWEs) are due to memory safety violations. More than 70% of reported vulnerabilities in the Chrome browser Taylor et al., 2022 as well as in the iOS, macOS Kehrer, 2019 , and Android Van der Stoep & Zhang, 2019 operating systems Gaynor, 2019 are related to memory unsafety. In fact, 72% of zero-day vulnerabilities in the last 12 years are due to memory safety violations enabled by memory-unsafe languages Google, 2025 . To scale bug finding beyond manual code audits, developers have turned to automated vulnerability discovery tools based on techniques such as static analysis and fuzz testing (in short fuzzing ). For instance, Google considers fuzzing their first line of defense Taylor et al., 2021 , while Meta reports that static analysis accounts for 70% of vulnerabilities discovered internally Rodriguez & Greene, 2022 . Illustrating the popularity of fuzzing, the OSS-Fuzz project Serebryany, 2017 continuously tests more than 1,000 open source software (OSS) projects, including many popular and security-critical projects. Regarding the popularity of static analysis, Beller et al. Beller et al., 2016 found in 2016 that approximately half of the 122 surveyed open-source projects employ static analysis. Fuzzing detects bugs by provoking unexpected program states, while static analysis examines the program’s source code. In a fuzzing setup , a fuzzer passes automatically generated inputs to the program under test via a harness —a fuzzer-specific entry point that is usually written manually. In a static analysis setup , only the static analyzer is required, though in some cases the program needs to be compiled. Unlike fuzzers, static analyzers are known to report false positives, i.e., benign code falsely flagged as buggy. But how do fuzzing and static analysis compare in their ability and effectiveness at detecting real bugs? Fuzzing and static analysis share the fundamental goal of finding bugs in programs, yet there is little research comparing these two approaches systematically. This can be explained by two factors: first, these tools are developed by largely separate communities; second, fuzzing and static analysis tools require fundamentally different project integrations, essentially requiring extra work to integrate the tools. These differences make a systematic comparison of both approaches particularly interesting. Is it worth the effort to employ both approaches in parallel, and can the approaches benefit from learning from each other, bringing both research communities closer together? In this paper, we adopt the perspective of an open-source C/C++ project maintainer who wants to use a bug finding tool to discover memory safety vulnerabilities. Where should they focus their resources? Currently, there are no clear guidelines for choosing one approach over the other. There is also no systematic analysis of their relative strengths and weaknesses. To develop such a systematic analysis, we define clear selection criteria and evaluate over 100 CVEs across seven open-source projects (using Magma Hazimeh et al., 2020 ), employing five static analyzers and 13 different fuzzers. Concretely, we • quantitatively compare static analysis tools and fuzzers from an end-to-end perspective (the number of bugs detected), exploring their strengths and weaknesses, • manually validate our metrics, evaluate the qualitative aspects of limitations, usability and resource usage and investigate how tools are used in the wild, and • derive recommendations for developers on how to add security tooling into their workflows. We find that each category of tools has a clear winner, but both find almost disjoint sets of bugs. After manual analysis, a majority of the bugs in our dataset is not detected by any tool. Outline. We start with an overview of the relevant background and related work ( Section 2 ). After defining the scope of our study ( Section 3 ), we conduct our analysis in two parts. The first part is an empirical evaluation ( Section 4 ) of freely available fuzzers and static analysis tools focusing on quantitative aspects of comparison. In the second part, we conduct a qualitative analysis ( Section 5 ), based on the empirical part. We discuss the implications of our analysis for project maintainers and research communities in Section 6 and conclude the paper in Section 7 . Data Availability. To make our findings reproducible and foster further research in this direction, we make our evaluation scripts and data openly available at https://anonymous.4open.science/r/fuzzing-vs-static-artifact-8973/ . 2. Background In this section, we explain the types of vulnerabilities we are interested in, before briefly summarizing the state-of-the-art in bug finding tools and the benchmarks commonly used to evaluate them. 2.1. Memory Unsafety in C/C++ In this work, we focus on one of the most prevalent classes of security vulnerabilities: memory unsafety and related types of vulnerabilities prevalent in C/C++ programs. As discussed before, more than 70% of the reported vulnerabilities in Chrome, iOS, macOS, and Android Taylor et al., 2022 ; Kehrer, 2019 ; Van der Stoep & Zhang, 2019 and 72% of zero-days are due to memory safety violations Google, 2025 . Software systems can be subject to many types of bugs and vulnerabilities. However, memory safety violations are (almost) unique to C/C++. While many modern languages provide memory safety by design, in C/C++ it is primarily the developer’s responsibility to uphold the safety contract. In exchange, C/C++ offers more low-level control and (in absence of runtime checks and garbage collection) a high performance. To be specific, we exclude bugs that are generally not security-relevant, such as unused variables or violations of coding conventions reported by a linter. Furthermore, bugs exclusive to other languages are