FirmReBugger: A Benchmark Framework for Monolithic Firmware Fuzzers
Mathew Duong, Michael Chesser, Guy Farrelly, Surya Nepal, Damith C. Ranasinghe · 2026
Read FirmReBugger: A Benchmark Framework for Monolithic Firmware Fuzzers by Mathew Duong, Michael Chesser, Guy Farrelly, Surya Nepal, Damith C. Ranasinghe (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-15774.txt.
Introduction
Mathew Duong and four co-authors build a yardstick for firmware fuzzers in a 2026 arXiv paper. Monolithic firmware runs countless devices. Fuzz testing it is an active research field. New methods claim gains in this hard domain. Checking those claims is messy, though. Metrics like code coverage and unique crashes are unreliable guides. Two fuzzers can post similar coverage while finding very different bugs.
Their answer is FirmReBugger, a benchmark framework. It is bug-based rather than coverage-based. A benchmark keyed to real bugs measures what users care about: found flaws. Fair comparison speeds progress across the field. Firmware flaws are security risks in routers, cameras, and industrial gear. Better fuzzers, honestly ranked, mean safer devices.
Catalog record
| CREATOR | Mathew Duong, Michael Chesser, Guy Farrelly, Surya Nepal, Damith C. Ranasinghe |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 82 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-15774
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FirmReBugger: A Benchmark Framework for Monolithic Firmware Fuzzers Mathew Duong, Michael Chesser, Guy Farrelly, Surya Nepal, Damith C. Ranasinghe Abstract Monolithic Firmware is widespread. Unsurprisingly, fuzz testing firmware is an active research field with new advances addressing the unique challenges in the domain. However, understanding and evaluating improvements by deriving metrics such as code coverage and unique crashes are problematic, leading to a desire for a reliable bug-based benchmark. To address the need, we design and build FirmReBugger , a holistic framework for fairly assessing monolithic firmware fuzzers with a realistic, diverse, bug-based benchmark. FirmReBugger proposes using bug oracles—C syntax expressions of bug descriptors—with an interpreter to automate analysis and accurately report on bugs discovered, discriminating between states of detected , triggered , reached and not reached . Importantly, our idea of benchmarking does not modify the target binary and simply replays fuzzing seeds to isolate the benchmark implementation from the fuzzer while providing a simple means to extend with new bug oracles. Further, analyzing fuzzing roadblocks, we created FirmBench , a set of diverse, real-world binary targets with 313 software bug oracles. Incorporating our analysis of roadblocks challenging monolithic firmware fuzzing, the bench provides for rapid evaluation of future advances. We implement FirmReBugger in a FuzzBench-for-Firmware type service and use FirmBench to evaluate 9 state-of-the art monolithic firmware fuzzers in the style of a reproducibility study, using a 10 CPU-year effort, to report our findings. 1 Introduction The surge in embedded systems, from cube-satellites to smart door locks and autonomous vehicles, is driven by small, low cost, and low-power microcontrollers. The proliferation of embedded systems creates more targets, new attack vectors, opportunities, and incentives for adversaries [ 40 ] . Testing firmware to identify vulnerabilities prior to exploitation or device failure is critically important. For example, Hanna et al. ’s [ 27 ] assessment of defibrillator firmware in a commercial product revealed critical vulnerabilities, including a buffer overflow that could allow remote code execution. Unsurprisingly, embedded systems’ rising prevalence and associated security concerns are driving advances in automated software testing methods like fuzzing. However, uncovering potential security vulnerabilities in firmware with fuzzing is challenged by hardware limitations, resource constraints, and the complexity of direct firmware interactions with hardware. Consequently, firmware fuzzing is an active topic of research interest where recent methods [ 21 , 55 , 44 , 56 , 35 , 19 , 20 , 9 , 10 ] re-host firmware using emulation on more resourceful computing platforms to bypass the resource limitations of embedded hardware. Emulation provides the ability to add sophisticated instrumentation to guide a fuzzer, facilitates introspection, and enables better detection and reproducibility of crashes whilst avoiding the need to synchronize hardware and emulation environments with device-in-the-loop methods. Despite the benefits of re-hosting, applying automated testing methods like fuzzing remains challenging, especially for monolithic firmware. Monolithic firmware directly manages all interactions with hardware, operating without support or supervisory control of an operating system, and combines all essential software components into a single unified entity. Most notably, in contrast to traditional applications, monolithic firmware interacts directly with peripherals (such as modems, GPS units, and sensors) through Memory Mapped Input Output (MMIO) or Direct Memory Access (DMA) mechanisms and interrupts. These low-level interactions are complex and may be across a diverse group of peripherals with behaviors differing between manufacturers. This poses significant challenges for employing a de-facto, grey-box, mutation-based fuzzing method to drive re-hosted firmware to discover software bugs. Advances in fuzz testing firmware aim to overcome the pertinent challenges to achieve scalable, efficient, and effective fuzzers by facilitating the firmware’s interaction with a diverse group of peripheral devices across various different microcontroller designs employing a range of different instruction set architectures (ISAs). But, a key question with regards to the evaluation of fuzzers, to support both the growth and rapid developments of new techniques is: How do we fairly assess improvements to the bug finding capabilities of monolithic firmware fuzzers? 1.1 Unique Problems Challenging Assessment Assessing the performance of fuzzers, in general, relies on empirical means to evaluate their bug-discovering capabilities. Code coverage and unique crash counts are adopted from domains outside of firmware targets to compare if one firmware fuzzer is “better” than another [ 21 , 55 , 44 , 56 , 19 , 20 , 9 , 10 , 45 ] . Unfortunately, adopting these accepted performance measures to evaluate monolithic firmware fuzzers is not without its unique set of problems. We delve into these issues in detail in Section 3 and summarised them below: • Code Coverage . The relationship of coverage achieved by a fuzzer to its bug-finding ability support it as a performance metric [ 24 , 29 , 31 ] . However, its use in evaluating monolithic firmware fuzzers can lead to misleading results. For example, a fuzzer exploiting a bug to reach otherwise unreachable code can inflate the coverage metric, giving the illusion of better performance—a scenario more commonly encountered in embedded systems (see Section 3.1 ). • Unique Crashes. Unique crashes are a more direct measure of the number of bugs found. But, implementing it accurately in the firmware domain is challenging. Differentiating and deduplicating crashes require capturing a bug’s context. Common identifiers, such as coverage profiles and stack hashes are prone to variability in program behavior, making consistent and reliable crash classification difficult. The problem is further exacerbated in the firmware domain where program flow is more complex due to the interrupt driven nature of firmware (see Section 3.2 ). A bug-based benchmark is recognized as an approach to alleviate the imperfect nature of coverage and problems with unique crash metrics. In general, bug-based benchmarks such as LAVA [ 16 ] with LAVA-M, Magma [ 28 ] , UNIFUZZ [ 32 ] , Google’s FuzzBench [ 36 ] , FixReverter [ 54 ] , and FuzzProBench [ 39 ] are proposed for non-firmware targets. However, a benchmark for evaluating monolithic firmware fuzzers does not yet exist. Notably, recent method for designing a benchmark in FixReverter [ 54 ] require access to source code, which is often unavailable with firmware images. And the bug-oracle approach in Magma [ 28 ] integrates the benchmark into the fuzzing logic and leads to the reported leaky oracle problem [ 28 ] . In addition, given the developing nature of firmware fuzzing there is a lack of effective tooling for automating the triaging of bugs. The alternative—curating direct performance results such as time-to-find bugs—necessitates manual crash analysis and triaging for each fuzzer, which is both time-consuming and error-prone. Consequently, we are motivated to design a bug-based benchmarking framework for monolithic firmware fuzzers. We consider the unique challenges posed in the domain and facilitate automated and fair evaluation of fuzzers. We summarize our efforts in the following sections. 1.2 Our Work We analyze the problems with empirical performance evaluation for firmware images (Section 3 ) and propose FirmReBugger , a new benchmark framework together with FirmBench target sets—challenging, real-world, benchmark binaries with a diverse set of bugs. Importantly, we curate binary targets through a careful consideration of existing monolithic firmware fuzzing challenges we discuss in Section 5.1 . Benchmark Design Goals. First, we extend the collective wisdom from the design of benchmarks