Generating Proof-of-Vulnerability Tests to Help Enhance the Security of Complex Software CCS: Security and privacy Software security engineering CCS: Security and privacy Intrusion/anomaly detection and malware mitigation CCS: Software and its engineering Software libraries and repositories CCS: Software and its engineering Software maintenance tools

Shravya Kanchi, Xiaoyan Zang, Ying Zhang, Danfeng (Daphne) Yao, Na Meng · 2026

Read Generating Proof-of-Vulnerability Tests to Help Enhance the Security of Complex Software CCS: Security and privacy Software security engineering CCS: Security and privacy Intrusion/anomaly detection and malware mitigation CCS: Software and its engineering Software libraries and repositories CCS: Software and its engineering Software maintenance tools by Shravya Kanchi, Xiaoyan Zang, Ying Zhang, Danfeng (Daphne) Yao, Na Meng (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-03956.txt.

Introduction

Shravya Kanchi and four co-authors built PoVSmith, a tool that generates proof-of-vulnerability tests on its own for applications built on third-party libraries. When a library vulnerability is reachable from application code, developers often demand concrete, runnable evidence before acting, because many reported flaws pose no practical risk to their specific application. Crafting such tests by hand is hard, and prior automation fell short.

PoVSmith combines call path analysis, exemplar tests, code context, and feedback into prompts that guide a coding agent and a large language model through generation, execution, and assessment. Evaluated on thirty-three application-library Java pairs, it revealed 158 unique application entry points calling vulnerable library APIs and correctly identified nearly all of them with their call paths.

Catalog record

CREATORShravya Kanchi, Xiaoyan Zang, Ying Zhang, Danfeng (Daphne) Yao, Na Meng
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.03956

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Generating Proof-of-Vulnerability Tests to Help Enhance the Security of Complex Software CCS: Security and privacy Software security engineering CCS: Security and privacy Intrusion/anomaly detection and malware mitigation CCS: Software and its engineering Software libraries and repositories CCS: Software and its engineering Software maintenance tools Shravya Kanchi, Xiaoyan Zang, Ying Zhang, Danfeng (Daphne) Yao, Na Meng Abstract . Developers create modern software applications (Apps) on top of third-party libraries (Libs). When library vulnerabilities are reachable through application code, the applications can be vulnerable to software supply chain attacks. Prior work shows that developers often require concrete and executable evidence, i.e., proof-of-vulnerability (PoV) tests , to decide whether a reported dependency vulnerability poses a practical security risk to their application. However, manually crafting such tests is challenging, and existing tool support is insufficient to automate the procedure. To streamline test generation, we created PoVSmith—a new approach that combines call path analysis, exemplar test, code context, and feedback into multiple prompts to guide a coding agent (i.e., Codex) and a large language model (i.e., GPT) for test generation, execution, and assessment. We evaluated PoVSmith on 33 ⟨ A ​ p ​ p , L ​ i ​ b ⟩ \langle App,Lib\rangle Java program pairs, where each App depends on a vulnerable Lib. PoVSmith revealed 158 unique application-level entry points (i.e., public methods) calling vulnerable library APIs; 152 (96%) of them were correctly found, together with the call paths properly recognized. With such method call information, PoVSmith generated 152 tests, 84 (55%) of which demonstrated feasible ways of attacking Apps by exploiting Lib vulnerabilities. PoVSmith substantially outperforms the state-of-the-art LLM-based approach, as it reduces human involvement while dramatically improving test quality. Our work contributes (1) a novel approach of agent-based test generation, (2) an iterative code refinement process driven by execution feedback, and (3) LLM-based quality assessment grounded in both the test context and execution logs. 1. Introduction Software supply chain (SSC) is an aggregation of all the people, processes, and technologies involved in producing or updating a piece of software ( 69 ) . Software supply chain vulnerabilities are security flaws, in external dependencies or third-party libraries (Libs) that software applications (Apps) rely on. Such weaknesses can get exploited by hackers to attack Apps for data theft, remote access, or ransomware deployment (see Figure 1 ). Figure 1. The threat model of software supply chain attacks SSC has recently become a primary attack surface ( 66 ) . In 2025 alone, there is a 73% increase in malicious open-source packages or Libs, revealing a troubling acceleration of SSC risks as these Libs may ultimately get used by lots of downstream Apps ( 1 ) . Cyble’s data—based on attacks claimed by threat actors on dark web data leak sites—shows that SSC attacks surged in October 2025 (i.e., 41 attacks), setting a new record more than 30% higher than the previous peak in April ( 56 ) . High-profile attacks like SolarWinds ( 60 ) , Log4j ( 62 ) , and MOVEit ( 5 ) highlighted how a single compromised library dependency can get exploited to affect thousands of software systems built on top of it. SSC attacks have become increasingly severe in terms of their breadth, depth, and overall impact. Motivation: To mitigate supply chain attacks, people created tools to detect vulnerable Lib dependencies ( 49 ; 18 ) or vulnerable API usage ( 54 ; 21 ; 51 ) in Apps, and to suggest repairs for security hardening ( 17 ; 79 ; 22 ) . However, none of these tools demonstrate how Lib vulnerabilities enable practical exploits that target downstream Apps. Recent work ( 77 ; 34 ) shows that given vulnerability reports, developers often require concrete and executable code to demonstrate proof-of-concept exploits, in order to assess the security risks and prioritize bug fixes. We use Proof-of-vulnerability (PoV) test to refer to such demo code that exploits vulnerabilities in Libs to launch attacks to Apps. Problem Statement: Given a vulnerable Lib and an App calling vulnerable APIs of that Lib, we aim to generate PoV test(s) for Apps to demonstrate the malicious program behaviors enabled by the vulnerable Lib version. There are only three tools to solve this problem ( 31 ; 35 ; 78 ) . Experimentally, they all fail to generate PoV tests in most cases ( 78 ) . Figure 2. PoVSmith has four phases Our Solution: To bridge the gap between developers’ need and current tool limitation, we introduce PoVSmith, an approach to generate PoV tests as JUnit tests customized for Java-based Apps. As shown in Figure 2 , PoVSmith has four phases: call path analysis, PoV test generation, test execution, and test evaluation. Given a vulnerable Lib version and an App using that Lib, Phase I prompts an AI coding agent—Codex ( 16 ) —to search the program context of App, for a) public methods calling the vulnerable Lib API(s) and b) the complete call path of each found method. With call paths and a user-provided exemplar test, Phase II prompts Codex to iteratively generate a PoV test T , assess the test based on its build/execution status, diagnose the root cause for any encountered unexpected build/test issue, and refine T accordingly. Phase III takes T from Phase II, conducts automatic build, and further executes T if it builds successfully. Phase IV takes in the build/execution logs output by Phase III together with T , prompting a large language model (LLM)—GPT—to evaluate whether the generated test successfully triggers the vulnerability. Phase IV outputs the evaluation result, which is sent together with T for developer review. For evaluation, we applied PoVSmith to a dataset of 33 \langle App,Lib\rangle Java program pairs, where each App depends on a vulnerable Lib and the vulnerabilities correspond to 20 CVE entries. In Phase I, PoVSmith revealed 158 unique app-level entry points (i.e., public methods) calling vulnerable APIs; 152 of these methods were correctly found, together with their call paths properly identified. With such call path information, PoVSmith generated 152 tests, 141 of which compiled smoothly and 84 tests successfully triggered vulnerabilities. Phase III correctly labeled 103 tests. We also experimented with two alternative AI coding agents (i.e., Gemini Code Assist ( 23 ) and Mistral Vibe ( 45 ) ), finding Codex to work most effectively. Finally, we empirically compared PoVSmith with a state-of-the-art LLM-based approach ( 78 ) , finding PoVSmith to work much better by producing a lot more vulnerability-triggering tests. To sum up, this paper makes the following contributions: (1) New methodology. We invented an agent-based approach PoVSmith. It detects public methods in Apps that (in)directly call vulnerable APIs in Libs, iteratively generates PoV tests for each method, evaluates tests via automatic build and testing, and assesses test quality via LLM-based self-critique. (2) New characterization of agent and LLM capabilities. We systematically evaluated PoVSmith’s effectiveness in call path analysis, iterative PoV test generation, and test assessment. We characterized its effectiveness when using different coding agents. We also empirically compared PoVSmith with a state-of-the-art tool ( 78 ) . (3) New dataset. We created a dataset of 207 verified call paths, showing how vulnerable APIs of Libs can influence runtime behaviors of Apps. They belong to 33

Read the complete text