Revisiting Code Debloating with Ground Truth-based Evaluation CCS: Security and privacy Software security engineering CCS: Software and its engineering Software testing and debugging
Muhammad Bilal, Moiz Ali, Mohit Kumar, Fareed Zaffar, Fahad Shaon, Ashish Gehani, Sazzadur Rahaman · 2026
Read Revisiting Code Debloating with Ground Truth-based Evaluation CCS: Security and privacy Software security engineering CCS: Software and its engineering Software testing and debugging by Muhammad Bilal, Moiz Ali, Mohit Kumar, Fareed Zaffar, Fahad Shaon, Ashish Gehani, Sazzadur Rahaman (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-17717.txt.
Introduction
Bilal and six colleagues revisit application-level code debloating through ground-truth-based evaluation, replacing proxies like test correctness, code size, and gadget counts. Their analysis of eight debloaters, Blade, Chisel, Cov, CovA, Lmcas, Trimmer, Occam, and Razor, spans source-to-source, IR-to-IR, and binary-to-binary tools. Dynamic tools often remove up to 94 percent of code that should stay, while static tools over-retain through over-approximation and sometimes add code via specialized variants. False removals and retentions cause incorrect function, inconsistency, and exploitable flaws.
Catalog record
| CREATOR | Muhammad Bilal, Moiz Ali, Mohit Kumar, Fareed Zaffar, Fahad Shaon, Ashish Gehani, Sazzadur Rahaman |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59160 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.17717
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Revisiting Code Debloating with Ground Truth-based Evaluation CCS: Security and privacy Software security engineering CCS: Software and its engineering Software testing and debugging Muhammad Bilal, Moiz Ali, Mohit Kumar, Fareed Zaffar, Fahad Shaon, Ashish Gehani, Sazzadur Rahaman Abstract . Program debloating aims to remove unused code to reduce performance overhead, attack surfaces, and maintenance costs. Over time, debloating has evolved across multiple layers (container, library, and application), each building on the principles of application-level debloating. Despite its central role, application-level debloating continues to rely on imperfect proxies for measuring performance, such as test-case-driven evaluation for correctness, code size for runtime efficiency, and gadget count reduction for estimating security posture. While there is widespread skepticism about using such imperfect proxies, the community still lacks standardized methodologies or benchmarks to assess the true performance of application-level software debloating. This experience paper aims to address the gap. We revisit the foundations of application-level debloating through a ground-truth–based evaluation paradigm. Our analysis of eight state-of-the-art debloaters – Blade , Chisel , Cov , Cov A , Lmcas , Trimmer , Occam , and Razor – uncovers insights previously unattainable through traditional evaluations. These tools collectively span the spectrum of source-to-source, IR-to-IR, and binary-to-binary transformation paradigms, characterizing a holistic reassessment across abstraction levels. Our analysis reveals that while dynamic analysis–based tools often remove up to 94% of code that should be retained, static analysis–based approaches exhibit the opposite behavior, showing high false retention rates due to coarse-grained dependency over-approximation. Additionally, static analyses may add code by introducing specialized variants of functions. False retentions and removals not only cause functional incorrectness but may also lead to systematic inconsistency, robustness failures, and exploitable vulnerabilities. 1. Introduction Code bloat occurs when software accumulates features over time, most of which remain unused in typical usage scenarios ( 30 ; 17 ) . Automated software debloating aims to identify and remove unnecessary code (or features) to avoid unnecessary complexity, resource consumption, and potential attack surfaces ( 5 ; 6 ; 16 ; 29 ) . Existing debloating paradigms can be organized into three broad categories: application-level (e.g., ( 14 ; 16 ; 19 ; 2 ; 19 ) ), library-level (e.g., ( 20 ; 18 ) ), and container-level (e.g., ( 11 ; 21 ; 12 ; 15 ) ). Although the targets differ, existing library- and container-level debloaters are also built on the principles of application-level debloating. This indicates that understanding and improving application-level debloating paradigms in terms of their ability to remove as much bloat as possible without compromising functionality or security will directly inform the other paradigms. Despite playing a central role, the evaluation of existing application-level debloating paradigms largely relies on imperfect proxy metrics. For instance, correctness is typically assessed through test-case–based evaluation, while performance and security improvements are inferred from reductions in code size or ROP gadget counts. A critical limitation of such reliance is that these proxy metrics often create a false sense of accomplishment . For example, most existing test-case–driven debloaters report substantial reductions in code size and ROP gadget counts, with little to no reported loss in correctness—findings that were later shown to be misleading by others ( 4 ; 8 ) . This occurs because the test suites used for evaluation frequently fail to capture subtle yet semantically important incorrect behavior. However, test-case-driven debloating paradigms enjoy a widely accepted optimism that providing high-quality and extensive test cases can ensure high-quality debloated software. To date, this belief remains unchallenged, as most evaluations in this domain primarily rely on test cases themselves. This dependency on test cases limits the evaluation mechanisms to what the test cases cover. Similarly, static-analysis–based paradigms ( 16 ; 2 ) often exhibit correct behavior after debloating; however, their corresponding code and ROP gadget reductions tend to be low. In fact, the introduction of multiple specialized variants of functions may increase gadget count. This raises an important open question: are the retained code regions genuinely necessary, or do they merely reflect conservative over-approximations of their underlying analysis engine—a question that remains largely unexplored. To address this gap, we revisit the foundations of evaluating application-level debloating paradigms and introduce a set of ground-truth–based evaluation metrics. Specifically, we propose a methodology that involves the manual creation of debloated program versions to serve as empirical ground truth. Following this methodology, we created a suite of 11 ground-truth payload applications—which includes widely-studied programs ranging from 5K to 76K LoC. These manually curated versions allow for precise measurement of correctness, performance, and security—providing a more reliable benchmark for assessing the true effectiveness of automated debloating tools. Next, by using the ground-truth–based evaluation, we seek to investigate previously unanswered questions regarding the correctness and efficacy of existing debloating paradigms. Specifically, by using our ground truth dataset, we comprehensively evaluated 8 state-of-the-art debloating tools: Blade , Chisel , Cov , Cov A , Lmcas , Occam , Razor , and Trimmer . These tools typically operate at two extremes: aggressive dynamic analysis-based debloaters that maximize code reduction at the cost of correctness, and conservative static analysis-based debloaters that preserve functionality but achieve minimal reduction ( 4 ; 9 ) . Since extensive test-case-based evaluation of these tools already exists, we focus our evaluation on the ground truth dataset. Our study reveals that despite achieving high code reduction rates, dynamic analysis-based debloaters ( Blade , Chisel , Cov , Razor ) can inadvertently remove up to 94% of code that should be retained, producing unstable and potentially insecure programs, with exploitable vulnerabilities. Most importantly, we observed several critical issues that are challenging to capture using test cases. For example, essential error logging and handling functions are frequently removed since they are very difficult to exercise with test cases. In multi-threaded programs, we found these debloaters often remove critical thread synchronization primitives, introducing race conditions. These issues may pass the initial test cases but can lead to deadlocks or system crashes in concurrent load environments. On the other hand, static analysis-based approaches ( Lmcas , Trimmer , Occam ) prioritize functional correctness of the debloated program, but frequently retain code from 100% of the functions not required in the context of debloating. Hybrid approaches ( Cov A ) attempt to combine dynamic analysis and static analysis, aiming to find the balance between code reduction and functional correctness. Nevertheless, we observed that the limitations of dynamic analysis critically influence the overall performance, as usually static analysis is used around the paths exercised with dynamic analysis, to improve overall coverage. Our contribution can be summarized as follows: • Development of a comprehensive ground truth dataset for systematic evaluation of source code debloaters, consisting of 11 widely-studied programs ranging from 5K to 76K LoC, including 9 GNU core utilities, 1 SIR benchmark program, and nginx . • Our analysis reveals that while dynamic analysis–based tools often remove up to 94% of code that should be retained, static analysis–based approaches exhibit the opposite behavior, showing high false retention rates due to coarse-grained dependency over-approximation. • Identification of 7 critical issues in existing debloating tools through detailed case studies, out of which only 3 were previously known—underscoring the strength of our ground truth-based evaluation. 2. Ground Truth Creation Dataset selection. We created the