A Practical Solution to Systematically Monitor Inconsistencies in SBOM-based Vulnerability Scanners Conference: The 41st ACM/SIGAPP Symposium on Applied Computing; March 23–27, 2026; Thessaloniki, Greece The 41st ACM/SIGAPP Symposium on Applied Computing (SAC ’26), March 23–27, 2026, Thessaloniki, G
Martin Rosso, Muhammad Asad Jahangir Jaffar, Alessandro Brighente, Mauro Conti · 2025
Read A Practical Solution to Systematically Monitor Inconsistencies in SBOM-based Vulnerability Scanners Conference: The 41st ACM/SIGAPP Symposium on Applied Computing; March 23–27, 2026; Thessaloniki, Greece The 41st ACM/SIGAPP Symposium on Applied Computing (SAC ’26), March 23–27, 2026, Thessaloniki, G by Martin Rosso, Muhammad Asad Jahangir Jaffar, Alessandro Brighente, Mauro Conti (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-17710.txt.
Introduction
Martin Rosso and three co-authors build a crash-test rig for the scanners that check software ingredient lists for known flaws. A Software Bill of Materials names every library inside a product, and scanners match that list against flaw databases. When a scanner chokes quietly, a team believes its product is clean while a known hole sails through.
Their rig, SVS-TEST, feeds scanners 16 hand-built ingredient lists whose true flaws are known exactly. Seven real-world scanners faced the rig, and their grades diverged wildly. Several failed silently on perfectly valid lists, the worst outcome the team found. All results are public, and the scanner makers heard the findings first. The rig now lets buyers grade scanners before trusting them.
Catalog record
| CREATOR | Martin Rosso, Muhammad Asad Jahangir Jaffar, Alessandro Brighente, Mauro Conti |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 81 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-17710
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Practical Solution to Systematically Monitor Inconsistencies in SBOM-based Vulnerability Scanners Conference: The 41st ACM/SIGAPP Symposium on Applied Computing; March 23–27, 2026; Thessaloniki, Greece The 41st ACM/SIGAPP Symposium on Applied Computing (SAC ’26), March 23–27, 2026, Thessaloniki, Greece DOI: 10.1145/3748522.3779717 ISBN: 979-8-4007-2294-3/2026/03 CCS: Security and privacy Vulnerability scanners CCS: General and reference Reliability CCS: General and reference Evaluation Martin Rosso, Muhammad Asad Jahangir Jaffar, Alessandro Brighente, Mauro Conti Abstract . Software Bill of Materials (SBOM) provides new opportunities for automated vulnerability identification in software products. While the industry is adopting SBOM-based Vulnerability Scanning (SVS) to identify vulnerabilities, we increasingly observe inconsistencies and unexpected behavior, that result in false negatives and silent failures. In this work, we present the background necessary to understand the underlying complexity of SVS and introduce SVS-TEST, a method and tool to analyze the capability, maturity, and failure conditions of SVS-tools in real-world scenarios. We showcase the utility of SVS-TEST in a case study evaluating seven real-world SVS-tools using 16 precisely crafted SBOMs and their respective ground truth. Our results unveil significant differences in the reliability and error handling of SVS-tools; multiple SVS-tools silently fail on valid input SBOMs, creating a false sense of security. We conclude our work by highlighting implications for researchers and practitioners, including how organizations and developers of SVS-tools can utilize SVS-TEST to monitor SVS capability and maturity. All results and research artifacts are made publicly available and all findings were disclosed to the SVS-tool developers ahead of time. † † cc-license: by 1. Introduction Since 17 , the OWASP ( OWASP ) Top 10 project recurrently named the usage of outdated and known to be vulnerable software dependencies among their Top 10 security risks for (web) applications 17 . This is confirmed also in the 18 Open Source Software Risks: OWASP dedicates at least four Top 10 risks to dependency-induced threats 18 . In response to these growing risks, the United States promoted the use of SBOM ( SBOM ) with the 2021 US Executive Order on Improving the Nation’s Cybersecurity . While the European Cyber Resilience Act (CRA) and Product Liability Directive (PLD) do not strictly mandate the use of SBOM , they explicitly confirm the manufacturers’ responsibility to deliver secure (software/smart) products and to provide security updates in a timely manner throughout their lifetime – something that SBOM is promising to achieve. SBOM provides a structured way to list software components, and is thus a key technology facilitating automated vulnerability scanning via svs ( svs ). As a result, SBOM and svs recently gained (new) traction in academia and industry alike. To maximize utility of this technology, it is important that researchers and practitioners understand the challenges, complexities, and limitations of svs . In practice, however, we observe significant dissonance between domain experts and practitioners. On the one hand, the academic literature identified a “lack of maturity in SBOM tooling” and the need for “more reliable, user-friendly, standard-conformable, and interoperable enterprise-level SBOM tools” 28 to mature the technology. On the other hand, we observe how the industry and practitioners perceive svs and svs -tools as solution to software security and to comply with legal requirements, but underestimate the limitations of this technology 8 ; 15 . Our work directly targets this dissonance. First, we summarize the svs process, highlighting key challenges and limitations and uncovering the hidden complexity of svs -tools in an extensive Background section. Once a theoretical background is established, we shift our focus to real-world svs -tools and how these tools operate in practice and how their limitations and shortcomings may impact an organization’s perceived security in practice. To do so, we develop svstts , a methodology to experimentally test svs -tools in realistic edge-case scenarios (i.e., attacks are explicitly not in scope). We present the utility of our solution in a case study. Evaluating a total of 7 established svs -tools against a manually created ground truth dataset of 16 realistic SBOM s, we observe major differences in the way that reliable svs -tools operate and what implicit assumptions they make on the input SBOM . Our results show that in practice, svs -tools may silently fail to lookup components from the input SBOM s, thus creating a false sense of security. This work has direct impact on svs -tool users, developers, and the svs community as a whole. We transparently informed svs -tool developers of our findings resulting in prompt updates of the respective tools. We transparently make our test cases and results available in our artifact repository. 1 1 1 All artifacts available at doi:10.5281/zenodo.17921254 . 2. Background and Related Literature In this section, we introduce the concept of SBOM ( SBOM ), how the industry performs component identification, and present the benefits and limitations of svs -tools. 2.1. SBOM ( SBOM ) A BOM ( BOM ) is a structured inventory of all components constituting a product. While the traditional BOM describes a physical product, an SBOM focuses on software and digital components. Typically, an SBOM contains metadata about the product itself, how and when the SBOM was created, and a list (or tree) of components 13 with their transitive relation in a structured machine-readable format. Two widely adopted SBOM formats exist: CycloneDX and the SPDX ( SPDX ). Both are built on top of structured and machine-readable data formats, such as JSON and XML, and allow expressing various pieces of information about the product, components, or vulnerabilities (cf. BOV ) and vulnerability exploitability (cf. vex ). In their latest versions, both formats are feature equivalent in terms of cybersecurity and vulnerability management, but use different vocabularies. For simplicity, we will stick to CycloneDX nomenclature. CycloneDX 2 2 2 https://cyclonedx.org/docs/ , sometimes abbreviated as CDX, is a project of the OWASP Foundation started in 2017. As of writing, the latest version of the standard is 1.6 . An earlier version of CycloneDX is specified in ECMA International standard ECMA-424 . System Package Data Exchange (SPDX) 3 3 3 https://spdx.dev/specifications was proposed by the Linux Foundation in 2011 and is currently available in version 3.0.1 . Originally developed to ensure software license compatibility of dependencies with the host project, it supports a variety of features and use cases today. An older version ( 2.2.1 ) is standardized as ISO/IEC 5962:2021. 2.2. Component Identifiers At the heart of every SBOM is a component list. To be useful for vulnerability scanning, the NTIA ( NTIA ) stipulates that every component should indicate at least a name, version, and unique identifier 13 . Both common SBOM formats, CycloneDX and SPDX , are rather permissive and allow adding components (in SPDX : Packages ) without any identifying piece of information. Although there are multiple means to (uniquely) identify a component, only the following two are frequently used for vulnerability identification. CPE ( CPE ) is a nist ( nist ) standard format to specify (software, hardware, or operating system) components 6 . A CPE is composed of multiple attributes separated by a colon (:). It usually contains a vendor name, product name, and product version. In general, attributes can be empty or hold special wildcard characters 6 . The CPE dictionary provides a list of official CPE names