Reducing False Positives in Static Bug Detection with LLMs: An Empirical Study in Industry CCS: Software and its engineering Maintaining software CCS: Computing methodologies Artificial intelligence
Xueying Du 1 , Jiayi Feng 1 , Yi Zou 1 , Wei Xu 2 , Jie Ma 2 , Wei Zhang 2 , Sisi Liu 2 , Xin Peng 1 , Yiling Lou 1 · 2026
Read Reducing False Positives in Static Bug Detection with LLMs: An Empirical Study in Industry CCS: Software and its engineering Maintaining software CCS: Computing methodologies Artificial intelligence by Xueying Du 1 , Jiayi Feng 1 , Yi Zou 1 , Wei Xu 2 , Jie Ma 2 , Wei Zhang 2 , Sisi Liu 2 , Xin Peng 1 , Yiling Lou 1 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-18844.txt.
Introduction
Xueying Du and eight colleagues study how to cut false flags from tools that hunt flaws in code at rest. Such tools scan code with no run and flag spots that may hold a bug. They are prized in firms and in labs. But they cry wolf a lot, most of all in big firm code. Each false flag takes a human check, and the checks eat the gain the tool was meant to give.
The team test if language models can sort true flags from false ones at firm scale. A good sort could cut the check load by a wide share. It could also win back trust in the hunt tools. The study is set in real firm code, not in toy code. That makes its grades of use to teams who face the same wolf cry each day.
Catalog record
| CREATOR | Xueying Du 1 , Jiayi Feng 1 , Yi Zou 1 , Wei Xu 2 , Jie Ma 2 , Wei Zhang 2 , Sisi Liu 2 , Xin Peng 1 , Yiling Lou 1 |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-18844
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Reducing False Positives in Static Bug Detection with LLMs: An Empirical Study in Industry CCS: Software and its engineering Maintaining software CCS: Computing methodologies Artificial intelligence Xueying Du 1 , Jiayi Feng 1 , Yi Zou 1 , Wei Xu 2 , Jie Ma 2 , Wei Zhang 2 , Sisi Liu 2 , Xin Peng 1 , Yiling Lou 1 Abstract . Static analysis tools (SATs) are widely adopted in both academia and industry for improving software quality, yet their practical use is often hindered by high false positive rates, especially in large-scale enterprise systems. These false alarms demand substantial manual inspection, creating severe inefficiencies in industrial code review. While recent work has demonstrated the potential of large language models (LLMs) for false alarm reduction on open-source benchmarks, their effectiveness in real-world enterprise settings remains unclear. To bridge this gap, we conduct the first comprehensive empirical study of diverse LLM-based false alarm reduction techniques in an industrial context at Tencent, one of the largest IT companies in China. Using data from Tencent’s enterprise-customized SAT on its large-scale Advertising and Marketing Services software, we construct a dataset of 433 alarms (328 false positives, 105 true positives) covering three common bug types. Through interviewing developers and analyzing the data, our results highlight the prevalence of false positives, which wastes substantial manual effort (e.g., 10 - 20 minutes of manual inspection per alarm). Meanwhile, our results show the huge potential of LLMs for reducing false alarms in industrial settings (e.g., hybrid techniques of LLM and static analysis eliminate 94–98% of false positives with high recall). Furthermore, LLM-based techniques are cost-effective, with per-alarm costs as low as 2.1–109.5 seconds and $0.0011–$0.12, representing orders-of-magnitude savings compared to manual review. Finally, our case analysis further identifies key limitations of LLM-based false alarm reduction in industrial settings. 1. Introduction Static analysis tools (SATs) play a crucial role in ensuring software security, reliability, and maintainability. They scan codebases without extensive test execution ( 4 ; 6 ; 5 ; 48 ; 49 ; 51 ; 55 ; 63 ) , facilitating early bug explosion during the software development process. To date, both academia and industry have proposed many static analysis tools (e.g., Github CodeQL ( 10 ) , CppCheck ( 12 ) , Facebook Infer ( 24 ) ), which are able to detect diverse bug types (e.g., Null-Pointer-Dereference and Use-After-Free) and have been widely adopted in practice ( 61 ; 15 ; 23 ) . However, SATs often suffer from high false positive rates when applied to large-scale software systems in practice. For example, previous studies ( 15 ; 61 ) show that state-of-the-art open-source SATs (e.g., CodeQL and Infer) exhibit over 95% false alarm rates when detecting Null Pointer Dereference (NPD) bugs in large-scale projects such as the Linux Kernel. This limitation arises from the need to over-approximate program behaviors to guarantee soundness, producing spurious warnings along infeasible paths. The underlying models of SATs typically abstract away complex features like pointer aliasing, external libraries, and conditional compilation, leading to mismatches with real-world behavior. Consequently, especially under heightened security requirements, the trade-off between soundness and precision favors conservative reporting, resulting in warnings that are formally justifiable but practically infeasible, ultimately overwhelming developers with false alarms. In practice, bug alarms reported by SATs would be manually inspected by developers to determine whether they correspond to true bugs or false positives. Such manual inspection is labor-intensive and time-consuming. Our survey within Tencent (i.e., one of the largest IT companies in China) reveals that developers spend on average more than 10 minutes reviewing one single static bug alarm, with many alarms even requiring multiple rounds of inspection. Moreover, given the large scale of enterprise software codebase, one single scan can produce hundreds of alarms. Therefore, the large volume of false positives leads to significant waste of manual effort, highlighting the necessity of automatic false alarm reduction techniques. Traditional false alarm reduction techniques rely on symbolic execution ( 32 ; 18 ) or model checking ( 44 ; 54 ) , which suffer from scalability and generalization issues when applied to large-scale enterprise software systems. Learning-based false positive reduction techniques train neural networks on existing false and true positives, which require massive labeled training data for each bug type and thus have limited generalizability across different bug types or for an enterprise where training data is unavailable. Recently, given the rapid development of Large Language Models (LLMs), some work ( 15 ; 61 ) proposes to reduce false alarms with LLMs, and their experimental results have revealed the potential of LLMs in reducing false alarms reported by open-source SATs on synthetic and open-source projects (e.g., Juliet ( 25 ) and Gawk ( 20 ) ). Nevertheless, it still remains unclear how LLM-based false alarm reduction techniques perform in industry, i.e., for real-world proprietary enterprise software . In particular, (1) proprietary enterprise software differs significantly from open-source software , especially for the code characteristics, development models, and code review practices. For instance, enterprise codebases typically enforce strict internal standards and follow rapid DevOps driven iterations, whereas open-source projects adopt more flexible, community-defined norms and evolve at a slower pace. Moreover, for code review practices, open-source projects benefit from extensive peer reviews by diverse contributors, while proprietary enterprises tend to have more centralized reviews. These differences may impact the types and distributions of bugs in each environment. (2) Enterprise-customized SATs also differ a lot from open-source SATs (e.g., CodeQL). In fact, existing open-source SATs often fall short of meeting the comprehensive and customized needs of enterprises, particularly in terms of efficiency, workflow integration, security, and cost. As a result, enterprises often develop customized SATs tailored to their specific needs and software. The performance differences between these enterprise-developed and open-source tools can significantly influence the effectiveness of false positive reduction. To fill the knowledge gap, in this work, we perform an extensive study to empirically investigate how LLMs perform in reducing static false alarms for large-scale industrial software systems at Tencent ( 53 ) . Tencent is one of the largest IT companies in China, which covers diverse businesses such as social media, digital entertainment, and cloud computing with billions of active users ( 58 ; 9 ) . We collect data from the real-world static bug detection pipeline at Tencent, which includes the bug alarms reported by its enterprise-customized SAT (i.e., BkCheck ) on the large-scale software of its Advertising and Marketing Services (AMS) business line (i.e., over hundreds of components). Based on the historical data, we construct a real-world industry false alarm reduction dataset of 433 bug alarms (i.e., 328 false positives and 105 true positives), which covers the three most frequent bug types, including Null Pointer Dereference (NPD), Out-of-Bounds (OOB), and Divide-by-Zero (DBZ) reported by BkCheck in Tencent. Based on our dataset, we empirically study the performance of LLMs in reducing false alarms for industrial software by answering the following research questions. RQ1 (Prevalence of false positives in industry). We first perform interviews with Tencent developers to investigate the application practice of SATs. SATs serve as a critical gated check-in during the code review process at Tencent, and all the reported bug alarms would be manually reviewed by developers, with