Any PoC: Universal Proof-of-Concept Test Generation for Scalable LLM-Based Bug Detection DOI: XXXXXXX.XXXXXXX Conference: Conference Title; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Software and its engineering Software testing and debugging CCS: Computing methodologies Multi-agent systems CCS: Security and privacy Software security engineering

Zijie Zhao, Chenyuan Yang, Weidong Wang, Yihan Yang, Ziqi Zhang, Lingming Zhang · 2026

Read Any PoC: Universal Proof-of-Concept Test Generation for Scalable LLM-Based Bug Detection DOI: XXXXXXX.XXXXXXX Conference: Conference Title; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Software and its engineering Software testing and debugging CCS: Computing methodologies Multi-agent systems CCS: Security and privacy Software security engineering by Zijie Zhao, Chenyuan Yang, Weidong Wang, Yihan Yang, Ziqi Zhang, Lingming Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-11950.txt.

Introduction

Zhao and five colleagues present ANYPoC, a multi-agent framework turning candidate bug reports into executable proof-of-concept tests. Naive agents lean toward success and may reward-hack with plausible but non-functional tests. ANYPoC fact-checks each report, iteratively builds and runs a proof of concept while gathering traces, then independently re-runs and scrutinises it. It also grows a knowledge base. On twelve large systems it produces 37 percent more valid proofs for true positives, rejects 9.7 times more false positives, and finds 121 new bugs, 108 confirmed and 92 fixed.

Catalog record

CREATORZijie Zhao, Chenyuan Yang, Weidong Wang, Yihan Yang, Ziqi Zhang, Lingming Zhang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57598
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.11950

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Any PoC: Universal Proof-of-Concept Test Generation for Scalable LLM-Based Bug Detection DOI: XXXXXXX.XXXXXXX Conference: Conference Title; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Software and its engineering Software testing and debugging CCS: Computing methodologies Multi-agent systems CCS: Security and privacy Software security engineering Zijie Zhao, Chenyuan Yang, Weidong Wang, Yihan Yang, Ziqi Zhang, Lingming Zhang Abstract . While recent LLM-based agents can identify many candidate bugs in source code, their reports remain static hypotheses that require manual validation, limiting the practicality of automated bug detection. We frame this challenge as a test generation task: given a candidate report, synthesizing an executable proof-of-concept test, or simply a PoC — such as a script, command sequence, or crafted input — to trigger the suspected defect. Automated PoC generation can act as a scalable validation oracle, enabling end-to-end autonomous bug detection by providing concrete execution evidence. However, naive LLM agents are unreliable validators: they are biased toward “success” and may reward-hack by producing plausible but non-functional PoCs or even hallucinated traces. To address this, we present Any PoC, a general multi-agent framework that (1) analyzes and fact-checks a candidate bug report, (2) iteratively synthesizes and executes a PoC while collecting execution traces, and (3) independently re-executes and scrutinizes the PoC to mitigate hallucination and reward hacking. In addition, Any PoC also continuously extracts and evolves a PoC knowledge base to handle heterogeneous tasks. Any PoC operates on candidate bug reports regardless of their source and can be paired with different bug reporters. To demonstrate practicality and generality, we apply Any PoC, with a simple agentic bug reporter, on 12 critical software systems across diverse languages/domains (many with millions of lines of code), including Firefox, Chromium, LLVM, OpenSSL, SQLite, FFmpeg, and Redis. Compared to the state-of-the-art coding agents, e.g., Claude Code and Codex, Any PoC produces 1.3 × \times more valid PoCs for true-positive bug reports and rejects 9.8 × \times more false-positive bug reports. To date, Any PoC has discovered 122 new bugs ( 105 confirmed , 86 already fixed ), with 45 generated PoCs adopted as official regression tests. 1. Introduction Automated bug detection based on Large Language Models (LLMs) has been extensively studied in recent years ( Anthropic 2025 ; OpenAI 2026 ; team 2024 ; Wu et al. 2025 ; Cursor 2026 ; Zhou et al. 2024 ; Du et al. 2024 ; Yang et al. 2025b ) . Different from traditional static analysis ( LLVM Project 2026a ; GitHub 2026 ; Semgrep 2026 ) and dynamic testing ( Arya et al. 2016 ; Google 2026 ; Fraser and Arcuri 2011 ; Pacheco and Ernst 2007 ) , recent LLM agents can autonomously explore the codebase and detect bugs in large systems. Those LLM-based bug detection tools from both industry ( team 2024 ; OpenAI 2025 ; Cursor 2026 ; Anthropic 2025 ; GitHub 2025 ) and academia ( Guo et al. 2025 ; Wu et al. 2025 ) have shown great promise by finding hundreds of bugs and security vulnerabilities in real-world software systems. Existing LLM-based bug detection techniques often suffer from high false positive rates ( Guo et al. 2025 ; Wu et al. 2025 ; Kaplan 2026 ) . LLM agents may fail to gather the full context, misinterpret code semantics, or hallucinate, leading to invalid bug reports. While there are tools to mitigate this issue by using another model to validate the reports ( Li et al. 2024b ) , the final outputs typically remain textual bug reports rather than concrete evidence. Thus, developers often need to manually verify whether the reports are real, which is difficult and time-consuming ( Du et al. 2026 ; Anthropic 2026 ) . This human-validation bottleneck fundamentally limits the scalability of LLM-based bug detection. Figure 1 (a) shows the number of true bugs (green area) and false bugs (red area) as the computation increases. Due to the limitations of human validation (orange vertical line), LLM-based bug detection systems cannot easily scale to handle more bugs (indicated by the shaded area). Figure 1. Computation scalability of bug finding. In contrast, dynamic testing and fuzzing techniques ( Arya et al. 2016 ; Google 2026 ; Pacheco and Ernst 2007 ) rely on concrete test executions and oracles (such as crashes and sanitizers ( Serebryany et al. 2012 ; Song et al. 2019 ; Stepanov and Serebryany 2015 ) ), to reliably indicate the presence of a bug without manual verification. For example, continuous and scalable fuzzing campaigns can discover tens of thousands of bugs ( Arya et al. 2016 ) . As shown in Figure 1 (b), strong oracles allow fuzzing to scale to a large amount of computation with low false positive rates (the small red area). In recent years, LLM-based fuzzing techniques ( Yang et al. 2025a ; Xia et al. 2024b ; Deng et al. 2023 ; Meng et al. 2024 ; Ou et al. 2024 ) have also been widely studied, and can substantially complement traditional fuzzers. However, we are limited by the fuzzing ceiling as shown in Figure 1 (b). This ceiling arises from several factors: fuzzers typically rely on a restricted set of runtime oracles, can only exercise executable modules that are available and runnable, and must explore an enormous input and path space that is often intractable for real-world systems. In practice, especially for large-scale systems with complex configurations, environment requirements, or hardware dependencies, the fuzzed binary often represents only a small fraction of the full codebase. As a result, even with more computation or LLM assistance, dynamic testing eventually reaches diminishing returns as discovering new behaviors and bug-triggering paths becomes increasingly difficult. Our insight is to combine the advantages of both lines. LLM-based analysis for bug detection does not have a plateau because it can continuously cover more paths and edge cases through abstract reasoning. Fuzzing/testing does not rely on manual verification and can scale to larger computation because of strong automated oracles. We want to use strong and automatic oracles to validate the bugs from LLM-based detection systems so that we can effectively scale computation to more bugs. As shown in Figure 1 (c), the number of bugs is not limited by the fuzzing ceiling nor human effort. To this end, we cast bug report validation as a test-generation problem: synthesizing an executable artifact that reliably triggers the suspected defect described in a candidate bug report. We call this artifact a proof-of-concept (PoC) test, or simply a PoC. A PoC can take different forms, such as a standalone script, command sequence, or crafted input, depending on the target system and bug type. By providing concrete execution-based evidence rather than a textual hypothesis alone, PoCs can substantially reduce the manual effort required to verify reported bugs and thus serve as a practical validation oracle for LLM-based bug detection. Recent work has started to explore test or PoC generation from bug reports, including agentic techniques ( Ahmed et al. 2025 ; Marques et al. 2025 ; Simsek et al. 2025 ; Nitin et al. 2025 ) . However, reliably turning PoC generation into a general-purpose validation oracle remains challenging ( § 3 ): • C1: LLM reward hacking. LLMs are rewarded to produce outputs

Read the complete text