A Soundness and Precision Benchmark for Java Debloating Tools Conference: Proceedings of the 2025 Workshop on Software Supply Chain Offensive Research and Ecosystem Defenses; October 13–17, 2025; Taipei, Taiwan Proceedings of the 2025 Workshop on Software Supply Chain Offensive Research and Ecosyste

Jonas Klauke, Tom Ohlmer, Stefan Schott, Serena Elisa Ponta, Wolfram Fischer, Eric Bodden · 2025

Read A Soundness and Precision Benchmark for Java Debloating Tools Conference: Proceedings of the 2025 Workshop on Software Supply Chain Offensive Research and Ecosystem Defenses; October 13–17, 2025; Taipei, Taiwan Proceedings of the 2025 Workshop on Software Supply Chain Offensive Research and Ecosyste by Jonas Klauke, Tom Ohlmer, Stefan Schott, Serena Elisa Ponta, Wolfram Fischer, Eric Bodden (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-20679.txt.

Introduction

Jonas Klauke and five co-authors test tools that shrink Java programs by cutting unused code brought in through shared libraries. Most projects pull in far more code than they run, so trimming can help, yet cutting a needed part breaks the program or changes what it does.

Their test suite, Deblometer, holds fifty nine small cases with hand checked lists of what must stay and what can go, covering many Java features. Three well known tools are scored, and all three cut needed code in some cases, with dynamic class loading fooling every tool. One tool keeps more bloat, another cuts too deep, and a third suffers from weak handling of notes in code, so none is fully safe yet.

Catalog record

CREATORJonas Klauke, Tom Ohlmer, Stefan Schott, Serena Elisa Ponta, Wolfram Fischer, Eric Bodden
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-20679

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Soundness and Precision Benchmark for Java Debloating Tools Conference: Proceedings of the 2025 Workshop on Software Supply Chain Offensive Research and Ecosystem Defenses; October 13–17, 2025; Taipei, Taiwan Proceedings of the 2025 Workshop on Software Supply Chain Offensive Research and Ecosystem Defenses (SCORED ’25), October 13–17, 2025, Taipei, Taiwan DOI: 10.1145/3733827.3765530 ISBN: 979-8-4007-1915-8/2025/10 CCS: Software and its engineering Functionality CCS: Software and its engineering Completeness CCS: Software and its engineering Software reliability CCS: General and reference Validation CCS: General and reference Reliability Jonas Klauke, Tom Ohlmer, Stefan Schott, Serena Elisa Ponta, Wolfram Fischer, Eric Bodden Abstract . Modern software development reuses code by importing libraries as dependencies. Software projects typically include an average of 36 dependencies, with 80% being transitive, meaning they are dependencies of dependencies. Recent research indicates that only 24.9% of these dependencies are required at runtime, and even within those, many program constructs remain unused, adding unnecessary code to the project. This has led to the development of debloating tools that remove unnecessary dependencies and program constructs while balancing precision by eliminating unused constructs and soundness by preserving all required constructs. To systematically evaluate this trade-off, we developed Deblometer , a micro-benchmark consisting of 59 test cases designed to assess support for various Java language features in debloating tools. Each test case includes a manually curated ground truth specifying necessary and bloated classes, methods, and fields, enabling precise measurement of soundness and precision. Using Deblometer , we evaluated three popular Java debloating tools: Deptrim, JShrink, and ProGuard. Our evaluation reveals that all tools remove required program constructs, which results in changed semantics or execution crashes. In particular, the dynamic class loading feature introduces unsoundness in all evaluated tools. Our comparison shows that Deptrim retains more bloated constructs, while ProGuard removes more required constructs. JShrink’s soundness is significantly affected by limited support for annotations, which leads to corrupted debloated artifacts. These soundness issues highlight the need to improve debloating tools to ensure stable and reliable debloated software. † † cc-license: by 1. Introduction The software supply chain consists of all processes and components involved in the development, building, and delivery of software. One part of it is the management of third-party libraries. This is typically handled by package managers like Maven, npm, or PIP, which automatically download and integrate the dependencies of a modern software project. These dependencies are either directly added by the developers, inherited from a parent project, or transitively added by other dependencies. There are, on average, 36 dependencies in an industry software project, with 80\% being transitive ( Dann et al. 2022 ) . However, Soto et al. ( Soto-Valero et al. 2021 ) showed that 75.1\% ( 2.7\% direct, 15.4\% inherited, 57\% transitive) are bloated and therefore, they are not executed in the project. As a result, a large portion of dependencies in the software supply chain are maintained and updated without impacting the project itself. Many researchers ( Azad et al. 2019 ; Ghavamnia et al. 2022 ; Hu et al. 2023 ; Ponta et al. 2021 ; Wang et al. 2023 ; Williams et al. 2021 ) demonstrated that debloating successfully decreases the attack surface by removing vulnerabilities in the project dependencies. However, if code required at runtime is incorrectly debloated (i.e., removed), it will cause execution failures ( Brown et al. 2024 ; Liu et al. 2024 ) or it may even lead to new vulnerabilities introduced by changed behavior during execution ( Brown and Pande 2019 ) . This raises the demand for debloating tools that are sound , ensuring all code required at runtime is preserved, and precise , effectively removing all bloated code. Debloating can be performed at different granularity levels. One broader granularity is the complete removal of entire dependencies from the project; however, a more fine-grained form of debloating is the disposal of program constructs inside the dependencies. Typical program constructs of Java dependencies are classes, methods, or fields. These are also considered bloated if they are not accessed during the execution of a given software project. For example, this could be caused by a dependency that provides capabilities to manage the file system, while the project only utilizes it to read files. In this case, all classes, methods, and fields related to writing files become bloated program constructs from the perspective of the project. Ponta et al. ( Ponta et al. 2019 ) have mapped vulnerabilities to fix commits of open-source dependencies to pinpoint them to the actual program constructs, revealing that the vulnerability does not affect the entire dependency. By removing these program constructs, the attack surface of the project is further reduced. Bloated program constructs also pose a threat even if they are not executed, since they can be exploited by deserialization attacks ( OWASP Foundation, Inc 2025 ) that misuse the mechanism to execute them. In this paper, we investigate the soundness and precision of debloating tools that are capable of removing bloated program constructs in Java. We have created a manually curated micro-benchmark called Deblometer where we exactly know which program constructs are required and which are bloated. Similar to the micro-benchmark for call graphs developed by Reif et al ( Reif et al. 2018 ) , we evaluate the soundness and precision in Deblometer on encapsulated Java language features to directly spot the shortcomings of the debloating tools. To find Java language features of interest, we have investigated current literature about debloating, established micro-benchmarks in similar problem areas and the Java language feature documentation resulting in 59 test cases highlighting the usage of 13 Java language features. Each test case is evaluated on the correct removal of the following program constructs: classes, interfaces, methods, and fields. The set of needed and unneeded program constructs is defined by a manually created ground truth for each evaluated language feature. We evaluated Deptrim ( Soto-Valero et al. 2023 ) , JShrink ( Bruce et al. 2020 ) , and ProGuard ( Guardsquare nv. 2025 ) on Deblometer . All tools support the removal of the evaluated program constructs. Our evaluation suggests that Deptrim tends to remove fewer program constructs, which results in a higher soundness score but a lower precision score. In contrast, ProGuard favors more aggressive debloating, resulting in a lower soundness score but a higher precision score. The investigation of JShrink also reveals soundness and precision issues, especially due to its lack of support for Java annotations, which causes JShrink to produce corrupted JAR files that can no longer be executed. All tools struggle with sound resolution of the dynamic class loading language feature, while JShrink and ProGuard also show no support for reflection language features. The execution of unsound debloated test cases revealed crashes and altered behavior during execution. This highlights the need for improvement in debloating tools to produce stable debloated dependencies without causing crashes or modifying program semantics. This paper consists of the following contributions: • A micro-benchmark 1 1 1 https://github.com/secure-software-engineering/Deblometer for Java debloating tools that remove bloated classes, methods, and fields. It consists of 59 test cases with manually curated ground truths categorized by 13 different language features. • A systematic evaluation 2 2 2 https://zenodo.org/records/16994277 of the soundness and precision of three popular

Read the complete text