An LLM-based Quantitative Framework for Evaluating High-Stealthy Backdoor Risks in OSS Supply Chains
Zihe Yan Kai Luo, Haoyu Yang, Yang Yu, Zhuosheng Zhang, Guancheng Li · 2025
Read An LLM-based Quantitative Framework for Evaluating High-Stealthy Backdoor Risks in OSS Supply Chains by Zihe Yan Kai Luo, Haoyu Yang, Yang Yu, Zhuosheng Zhang, Guancheng Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-13341.txt.
Introduction
Zihe Yan and 5 co-authors build a scoring framework for hidden backdoor risk in open-source software supply chains. Modern projects reuse many third-party parts, but weak upkeep and thin review can let stealthy attacks slip in, as the XZ-Util case showed. Judging maintainer behavior with fixed patterns alone misses much of that risk.
The framework models an attack stage by stage and sets a metric for each stage. Language models then judge review activity and privilege changes by meaning, not by hand-made rules. Applied to sixty-six high-priority Debian packages, the study found the current supply chain exposed to several kinds of security risk.
Catalog record
| CREATOR | Zihe Yan Kai Luo, Haoyu Yang, Yang Yu, Zhuosheng Zhang, Guancheng Li |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-13341
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
An LLM-based Quantitative Framework for Evaluating High-Stealthy Backdoor Risks in OSS Supply Chains Zihe Yan Kai Luo, Haoyu Yang, Yang Yu, Zhuosheng Zhang, Guancheng Li Abstract In modern software development workflows, the open-source software supply chain significantly contributes to efficient and convenient engineering practices. With increasing system complexity, it has become a common practice to use open-source software as third-party dependencies. However, due to the lack of maintenance for underlying dependencies and insufficient community auditing, ensuring the security of source code and the legitimacy of repository maintainers has become a challenge, particularly in the context of high-stealth backdoor attacks such as the XZ-Util incident. To address these problems, we propose a fine-grained project evaluation framework for backdoor risk assessment in open-source software. Our evaluation framework models highly stealthy backdoor attacks from the attacker’s perspective and defines targeted metrics for each attack stage. Moreover, to overcome the limitations of static analysis in assessing the reliability of repository maintenance activities, such as irregular committer privilege escalation and insufficient review participation, we employ large language models (LLMs) to perform semantic evaluation of code repositories while avoiding reliance on manually crafted patterns. The effectiveness of our framework is validated on 66 high-priority packages in the Debian ecosystem, and the experimental results reveal that the current open-source software supply chain is exposed to a series of security risks. Code — https://github.com/XuanwuLab/HSBRiskEvaluator 1 Introduction As the cornerstone of modern software engineering, open-source software has been deeply integrated into a wide range of systems, from desktop applications ( 9 ) and mobile platforms ( 6 ) to cloud computing infrastructures. By leveraging collaborative development, code reuse, and community-driven models, open-source software significantly lowers technical barriers and accelerates software innovation and iteration. Developers worldwide can share components and reuse modules to efficiently build complex application ecosystems. Many mainstream architectures, such as Linux, Docker, and AI frameworks, heavily rely on the openness and reusability of open-source software. However, the openness and extensive dependency propagation of open-source software have also introduced increasing risks to the software supply chain. In recent years, attackers have gradually evolved traditional poisoning attacks into more targeted and persistent advanced threats. A typical example is the XZ Utils incident ( 17 ) , where the adversary did not merely rely on a one-off code injection but instead maintained a long-term presence within the community, steadily building trust and waiting for the right moment to strike. These attacks exhibit distinctive characteristics: • High stealth , achieved by leveraging techniques such as binary payload embedding ( 10 ) , formatting obfuscation, and pseudo-random naming to deeply bury backdoors within critical dependency paths; • High complexity , as the attack chains often span multiple repositories and downstream build systems ( 15 ) , making detection through standard review processes extremely difficult; • Human-oriented tactics ( 5 ) , where attackers use social engineering to integrate into maintainer circles and masquerade as trusted contributors, significantly lowering defenders’ vigilance against potential threats. As open-source dependencies become increasingly intertwined and automated build tools proliferate, once such backdoors are merged into the main branch, their impact can rapidly propagate across projects and platforms, creating systemic risks that are extremely hard to contain. To address these threats, existing studies have explored defense mechanisms based on static code analysis, security auditing, software bill of materials (SBOM) generation ( 21 ) , and dependency tracking ( 20 ) . However, these approaches often suffer from two limitations: (1) the lack of attacker-centric modeling to anticipate which repositories are likely targets; and (2) the reliance on known patterns, which fails to detect evolving stealth strategies. Consequently, there is an urgent need for a systematic and quantifiable risk assessment method to help developers and platform maintainers identify high-value poisoning targets and formulate effective defense strategies. In this work, we propose a systematic risk assessment framework for highly stealthy poisoning attacks targeting open-source repositories. From the attacker’s perspective, our framework models the target selection process based on four core dimensions: (1) Dependency Impact , which measures the propagation effect of the component in the software ecosystem; (2) Payload Concealment , which assesses the feasibility of embedding and disguising malicious code in the project; (3) Community Quality , which captures the strength of community mechanisms in defending against malicious code contributions from attackers; and (4) Continuous Integration , Based on this metric system, we integrate LLMs to perform static and semantic analysis of repositories, ultimately building an automated assessment tool. This tool generates a multi-dimensional risk matrix to provide structured, quantitative scores of poisoning risk in open-source software. The main contributions of this work are as follows: 1. A unified evaluation framework is proposed to quantify and assess supply chain poisoning risks in open-source Linux packages hosted on GitHub. The framework integrates both structural and behavioral risk factors. 2. A fully automated assessment tool is developed and released. The tool leverages both static heuristics and LLM-based semantic analysis to support large-scale, code-aware risk evaluation. 3. We empirically evaluate 66 high-priority packages in the Debian ecosystem, revealing that Community Quality remains the most prevalent attack surface across projects. This insight is further validated through a case study on the xz repository, a real-world incident that reflects our framework’s key risk indicators. Figure 1: The attack framework and detection mechanism. 2 Related Work 2.1 Software Supply Chain Attack Although most work has focused on framework design for efficiency improvement ( 18 ) , security issues have recently attracted increasing attention due to the rising trend of software supply chain attack incidents. Such vulnerability can exist throughout the entire software development lifecycle and pose threats to various dependency environments (e.g., npm, PyPI, GitHub). 20 studied how attackers exploit package managers to achieve arbitrary code execution in open-source software supply chains. Their study analyzed seven ecosystems, including npm and PyPI, and identified three install-time and four runtime techniques that attackers use to execute malicious code. 4 systematically analyzed the software supply chain security issues in GitHub Actions, identifying risks such as dependency confusion, poor permission management, command injection, and credential exposure in reusable Actions and workflows. Because such attacks are often stealthily embedded in dependencies or build processes, they can impact a wide range of downstream projects and users, resulting in extensive and severe consequences. 2.2 Backdoor Detection and Risk Assessment in Open Source Repositories To systematize the detection of risks in open source software (OSS) ( 1 ) , numerous risk assessment frameworks have been proposed, including OpenSSF Scorecard ( 23 ) , NIST 800-161 ( 3 ) , and BSIMM ( 12 ) . These frameworks support both pre-release warnings and post-release issue detection ( 3 ) . Building on them, 7 mapped the metrics of ten supply chain frameworks to ATT&CK attack techniques ( 22 ) and analyzed incidents such as SolarWinds ( 2 ) , Log4j ( 11 ) , and XZ Utils ( 14 ) . However, their approach only identified attack-related tasks and could not serve as a general standard for evaluating OSS security. It also overlooked highly stealthy backdoor threats, particularly when attackers infiltrate project communities. To mitigate this gap, 16 introduced human factors as an attack vector, while 8 further proposed reputation-based metrics to assess maintainer