A Large Scale Empirical Analysis on the Adherence Gap between Standards and Tools in SBOM Journal: TOSEM Volume: 1 1 1 DOI: 10.1145/3788692 CCS: Software and its engineering Maintaining software CCS: Security and privacy Software security engineering
Chengjie Wang, Jingzheng Wu, Hao Lyu, Xiang Ling, Tianyue Luo, Yanjun Wu, Chen Zhao · 2026
Read A Large Scale Empirical Analysis on the Adherence Gap between Standards and Tools in SBOM Journal: TOSEM Volume: 1 1 1 DOI: 10.1145/3788692 CCS: Software and its engineering Maintaining software CCS: Security and privacy Software security engineering by Chengjie Wang, Jingzheng Wu, Hao Lyu, Xiang Ling, Tianyue Luo, Yanjun Wu, Chen Zhao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-05622.txt.
Introduction
Chengjie Wang and 6 co-authors present, in a 2026 study, the first large-scale check of how faithfully software bill of materials tools follow their standards. A software bill of materials lists a program's components to improve supply-chain transparency and security. Standards exist, but tools that generate these lists may quietly diverge from them.
Their automated framework evaluated tens of thousands of generated files from six tools across thousands of real repositories, then repeated the check a year later. Compliance support was inadequate, tools disagreed sharply with each other and with their own earlier versions, and license accuracy was poor. The authors analyze causes, suggest practical fixes, and release their code and data for further research.
Catalog record
| CREATOR | Chengjie Wang, Jingzheng Wu, Hao Lyu, Xiang Ling, Tianyue Luo, Yanjun Wu, Chen Zhao |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 117 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-05622
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Large Scale Empirical Analysis on the Adherence Gap between Standards and Tools in SBOM Journal: TOSEM Volume: 1 1 1 DOI: 10.1145/3788692 CCS: Software and its engineering Maintaining software CCS: Security and privacy Software security engineering Chengjie Wang, Jingzheng Wu, Hao Lyu, Xiang Ling, Tianyue Luo, Yanjun Wu, Chen Zhao Abstract . A Software Bill of Materials (SBOM) is a machine-readable artifact that systematically organizes software information, enhancing supply chain transparency and security. To facilitate the exchange and utilization of SBOMs, organizations such as the Linux Foundation and OWASP have proposed SBOM standards. Following standards, organizations have developed tools for generating and utilizing SBOMs. However, limited research has examined the adherence of these SBOM tools to standard specifications, a gap that could lead to compliance failures and disruptions in SBOM utilization. This paper presents the first large-scale, two-stage empirical analysis of the adherence gap, using our automated evaluation framework, SAP . The evaluation, comprising a baseline evaluation and a one-year longitudinal follow-up, covers 55,444 SBOMs generated by six SBOM tools from 3,287 real-world repositories. Our analysis reveals persistent, fundamental limitations in current SBOM tools: (1) inadequate compliance support with policy requirements; (2) poor tool consistencies, including inter-tool consistency rates as low as 7.84% to 12.77% for package detection across languages, and significant longitudinal inconsistency, where tools show low consistency with their own prior versions; and (3) mediocre to poor accuracy for detailed software information, e.g. , accuracy of package licenses below 20%. We analyze the root causes of these gaps and provide practical solutions. All the code, replication docker image, evaluation results are open sourced at GitHub and Zenodo 1 1 1 Available at https://github.com/dw763j/SAP and https://doi.org/10.5281/zenodo.14998624 . for further researches. † † cc-license: by 1. Introduction Modern software development increasingly relies on software supply chains, the complex networks of third-party libraries, frameworks, and dependencies that underpin contemporary applications ( 70 ) . Code reuse facilitated by software supply chains substantially improves development efficiency in modern software projects ( 50 ) . However, supply chain opacity may lead developers to inadvertently incorporate vulnerable or license-violating software ( 53 ; 3 ; 71 ; 61 ) . These risks necessitate proactive governance to address security and compliance challenges ( 97 ; 34 ; 43 ; 75 ; 39 ) . A Software Bill of Materials (SBOM) is a structured document detailing software metadata across multiple data fields, aiming to enhance software transparency and security ( 59 ; 7 ; 66 ; 11 ) . Consequently, their adoption is widespread. For instance, the Linux Foundation conducts a survey that reports that 90% of the 341 surveyed organizations are implementing or planning SBOM strategies ( 17 ) . Furthermore, governments in the US and EU have initiated policies that leverage SBOMs for improved security management on the software supply chain ( 86 ; 1 ) . To ensure the software supply chain transparency and security, organizations propose SBOM standards to standardize the process of exchanging software metadata. These standards provide an interoperable and consistent data structure for the SBOM, enabling effective exchange and processing. Among the community, key SBOM standards include the Software Package Data Exchange (SPDX) proposed by the Linux Foundation, the CycloneDX by the OWASP, and Software Identification Tagging (SWID Tagging) by the National Institute of Standards and Technology (NIST) ( 90 ; 57 ; 26 ; 89 ; 55 ) . Guided by these standards, organizations develop SBOM tools to generate and consume SBOMs ( 56 ; 19 ; 79 ; 44 ; 47 ; 4 ) . They enable the automated generation and consumption of SBOMs, thereby promoting broader SBOM adoption ( 49 ; 12 ; 78 ; 48 ) . However, for SBOMs to fulfill their purpose as interchangeable data exchange artifacts, the tools creating them must adhere to the SBOM standards. Existing studies already highlight that current tools are often insufficient, producing inaccurate or incomplete SBOMs ( 72 ; 45 ; 9 ; 94 ) . From the perspective of the implementation process from SBOM standards to SBOM tools, these shortcomings could come from a deeper, more fundamental problem, which we term the “adherence gap” . The “adherence gap” is the discrepancy between the required structure, content, and quality attributes defined by SBOM standards and the actual output generated by SBOM tools. When tools fail to fully adhere to these specifications, the resulting SBOMs suffer from compliance failures and inconsistencies, disrupting their utilization and hindering the goal of improving software supply chain transparency and security. Table 1. Inherent data quality attributes defined by ISO/IEC 25012 ( 27 ) Attribute Definition Interpretation Compliance The degree to which data has attributes that adhere to standards, conventions, or regulations in force and similar rules relating to data quality in a specific context of use. SBOM tools adhere to the structure and mandatory data fields defined by the SBOM standards. Consistency The degree to which data has attributes that are free from contradiction and are coherent with other data in a specific context of use. SBOM tools produce consistent information in SBOMs between each other. Accuracy The degree to which data has attributes that correctly represent the true value of the intended attribute of a concept or event in a specific context of use. SBOM tools produce accurate information about software. Thus, we set out to evaluate this adherence gap between SBOM standards and tools. To achieve this, we focus on the fundamental function of SBOMs as data exchange artifacts. SBOMs are designed to be reliably shared and processed across organizational boundaries. As such, the SBOM tools must adhere to the SBOM standards with the lens of established data quality principles, e.g. , characteristics from ISO/IEC 25012 ( 27 ) . Table 1 presents the three data quality attributes that we systematically analyse, complying with ISO/IEC 25012: compliance, consistency, and accuracy. These attributes address distinct but interdependent quality dimensions that form a hierarchical dependency chain. Compliance establishes the foundation: without structural adherence to standards, SBOMs cannot be reliably processed. Consistency builds upon this foundation to enable SBOM interchangeability across tools. Accuracy represents the highest utility layer, determining whether SBOMs can effectively inform security and compliance decisions. We design SAP , an automated and extensible S BOM g ap evaluation framework to assess these attributes between SBOM standards and SBOM tools. It mainly consists of three modules: first, SAP executes SBOM tools through a standardized workflow in the generation module to generate SBOMs (§ 3.1 ). It then draws information from SBOMs in the extraction module (§ 3.2 ) and assesses the data quality attributes between SBOM standards and SBOM tools in the evaluation module (§ 3.3 ). For this study, we employ a two-stage design to provide a dynamic view of the SBOM ecosystem. We first collect 3,287 real-world GitHub repositories spanning C/C++, Java, and Python languages (§ 4.1 ). We conduct a baseline evaluation using six advanced SBOM tools at October 2024, and then perform a one-year longitudinal follow-up with their October 2025 versions, yielding a total of 27,795 and 27,649 SBOMs, respectively. The follow-up evaluations are commonly utilized for measuring evolving targets, which is suited for the evolving SBOM tools ( 42 ; 65 ) . Furthermore, we