Large-Scale Empirical Analysis of Continuous Fuzzing: Insights from 1 Million Fuzzing Sessions

Tatsuya Shirai, Olivier Nourry, Yutaro Kashiwa, Kenji Fujiwara, Yasutaka Kamei, and Hajimu Iida · 2025

Read Large-Scale Empirical Analysis of Continuous Fuzzing: Insights from 1 Million Fuzzing Sessions by Tatsuya Shirai, Olivier Nourry, Yutaro Kashiwa, Kenji Fujiwara, Yasutaka Kamei, and Hajimu Iida (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-16433.txt.

Introduction

Tatsuya Shirai leads five co-authors in a 2025 study of fuzzing, a way to hunt bugs by feeding odd inputs to code. Many teams now fuzz in short bursts each time code changes. The group asks what this steady fuzzing truly finds. They pull logs from OSS-Fuzz, a free fuzzing service run by Google.

The logs cover about 1.12 million runs across 878 projects in total. Many bugs sat in the code before steady fuzzing even began. Reach into new code kept growing as runs went on, and new reach led to new bugs. The findings give teams a clear view of what non-stop fuzzing buys them.

Catalog record

CREATORTatsuya Shirai, Olivier Nourry, Yutaro Kashiwa, Kenji Fujiwara, Yasutaka Kamei, and Hajimu Iida
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-16433

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Large-Scale Empirical Analysis of Continuous Fuzzing: Insights from 1 Million Fuzzing Sessions Tatsuya Shirai, Olivier Nourry, Yutaro Kashiwa, Kenji Fujiwara, Yasutaka Kamei, and Hajimu Iida Abstract Software vulnerabilities are constantly being reported and exploited in software products, causing significant impacts on society. In recent years, the main approach to vulnerability detection, fuzzing, has been integrated into the continuous integration process to run in short and frequent cycles. This continuous fuzzing allows for fast identification and remediation of vulnerabilities during the development process. Despite adoption by thousands of projects, however, it is unclear how continuous fuzzing contributes to vulnerability detection. This study aims to elucidate the role of continuous fuzzing in vulnerability detection. Specifically, we investigate the coverage and the total number of fuzzing sessions when fuzzing bugs are discovered. We collect issue reports, coverage reports, and fuzzing logs from OSS-Fuzz, an online service provided by Google that performs fuzzing during continuous integration. Through an empirical study of a total of approximately 1.12 million fuzzing sessions from 878 projects participating in OSS-Fuzz, we reveal that (i) a substantial number of fuzzing bugs exist prior to the integration of continuous fuzzing, leading to a high detection rate in the early stages; (ii) code coverage continues to increase as continuous fuzzing progresses; and (iii) changes in coverage contribute to the detection of fuzzing bugs. This study provides empirical insights into how continuous fuzzing contributes to fuzzing bug detection, offering practical implications for future strategies and tool development in continuous fuzzing. This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible. I Introduction In recent years, a significant number of vulnerabilities have impacted the open-source software ecosystem [ 1 ] . With over 96% of commercial code making use of open-source software (OSS) [ 2 ] , any vulnerability present in OSS has the potential to spread across the world. The risk and impacts of vulnerability propagation was clearly demonstrated by vulnerabilities such as Heartbleed [ 3 ] , Log4Shell [ 4 ] , and RegreSSHion [ 5 ] . The Heartbleed vulnerability exposed security flaws in approximately 2.0 million HTTPS servers [ 6 ] and RegreSSHion allowed arbitrary code execution with root privileges on an estimated 14 million servers [ 7 ] . Fig. 1 : Architecture of OSS-Fuzz The current state-of-the-art automated approach to detect software vulnerabilities is fuzz testing (fuzzing) [ 8 ] . It works by generating large amounts of testing inputs to trigger unintended behavior such as simple crashes, to memory overflows which can be exploited to gain elevated privileges. There are three types of fuzzing strategies: black-box fuzzing, white-box fuzzing and grey-box fuzzing. Black-box fuzzing works by randomly sending large amounts of inputs to the target software. For white-box and grey-box fuzzing, the fuzzer is given knowledge (full knowledge for white-box, partial knowledge for grey-box) about the codebase so that it can send its inputs to specific lines of code. As of 2025, most modern state-of-the-art fuzzers such as AFL++ and libfuzzer are greybox fuzzers and follow a coverage-guided fuzzing approach (CGF) [ 9 ] . CGF refers to a fuzzing strategy where the fuzzer tries to maximize the coverage ( i.e., the number of source code lines executed) during fuzzing to increase the chances of finding vulnerabilities. Several studies investigated the effectiveness of coverage-based fuzzing. In their study on the reliability of coverage-guided fuzzing, Böhme et al. [ 10 ] found that there is a strong correlation between code coverage and the number of detected fuzzing bugs ( i.e., bugs identified by fuzzing). Liyanage et al. [ 11 ] conducted another study where they fuzzed continuously for seven days to study the evolution of code coverage over time. Their results indicate that fuzzing coverage increases logarithmically over time, meaning that long fuzzing sessions give diminishing return over time. Despite many studies reporting the efficacy of fuzzing, some studies investigated the current limitations of fuzzing and its pitfalls. For example, Nourry et al. [ 12 ] conducted a survey with fuzzing experts to investigate the challenges of fuzzing. Their results revealed several challenges related to the configuration, compilation, and usage of fuzzers. Based on their results, they concluded that the high barrier of entry of fuzzing hinders the adoption of fuzz testing by non-experts ( i.e., average developers). To address the complexity and difficulty of adopting fuzzing, external fuzzing providers have gained significant popularity in recent years. As of 2025, the main provider, OSS-Fuzz, is developed and maintained by Google to fuzz critical open-source software systems. One of the major reasons behind the popularity of OSS-Fuzz is its implementation of Continuous Fuzzing through seamless integration within the continuous integration (CI) process [ 13 ] . By executing fuzzing periodically and automatically, OSS-Fuzz can provide continuous fuzzing to participating software systems, making early detection of fuzzing bugs more likely. Traditional fuzzing typically involves executing a fuzzer on a fixed version of the software over an extended period. An industry report suggests that effective fuzzing often spans at least one week, with 30 days considered ideal to search for fuzzing bugs [ 14 ] . In contrast, continuous fuzzing adopts a different strategy by conducting brief but frequent fuzzing sessions, aiming to identify fuzzing bugs in near real-time as the software evolves. In their study on continuous fuzzing, Klooster et al. [ 15 ] showed that short sessions of 15 minutes can also be effective to detect critical bugs. In 2021, Zhu et al. [ 16 ] conducted a study on grey-box fuzzing and found that 77% of the fuzzing-related bugs were introduced by recent code changes. Continuous fuzzing provides the ability to fuzz quickly and frequently, while eliminating the need for developers to manually configure the fuzzers. This makes it a particularly appealing approach for non-experts who seek to improve the security of their OSS systems. While continuous fuzzing offers some advantages over traditional methods, it is still unclear whether the effectiveness of shorter, more frequent fuzzing sessions is sustained across multiple repeated runs. It is also not obvious whether existing findings on traditional fuzzing can be generalized to continuous fuzzing or not. For example, traditional fuzzing involving long fuzzing sessions on a specific commit might allow a fuzzer to reach deeper parts of the code ( i.e., higher coverage), leading to more fuzzing bugs being found [ 17 ] . While prior work [ 18 ] and OSS-Fuzz [ 19 ] have proven that short fuzzing sessions are also effective, it remains unclear how repeated short fuzzing sessions in continuous fuzzing can progressively increase code coverage and bug detection. In this study, we aim to investigate and clarify how the number of fuzzing sessions can impact fuzzing bug detection and code coverage. To do so, we analyze historical fuzzing data from 878 open-source projects participating in the OSS-Fuzz initiative. To the best of our knowledge, this is the first large-scale study investigating the effectiveness of continuous fuzzing. The main contributions of this study are as follows: 1. Empirical Analysis of Continuous Fuzzing at Scale. We conducted a large-scale empirical investigation on the effectiveness of continuous fuzzing by analyzing more than 3.95 million

Read the complete text