Vulnerability Detection in Popular Programming Languages with Language Models
Syafiq Al Atiiq, Christian Gehrmann, Kevin Dahlén · 2024
Read Vulnerability Detection in Popular Programming Languages with Language Models by Syafiq Al Atiiq, Christian Gehrmann, Kevin Dahlén (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-15905.txt.
Introduction
Language models hunt software flaws unevenly across programming languages, Syafiq Al Atiiq, Christian Gehrmann, and Kevin Dahlen find in a 2024 paper. Most earlier work stuck to C and C++. Their study widens the lens to JavaScript, Java, Python, PHP, and Go, with the old pair as a baseline.
A curated fixes dataset supplied real flaws per language. Top models were fine-tuned and scored per language. JavaScript detection ran far ahead, reaching practical levels, while C and C++ lagged. Code shape and training mix may explain the gap. Security teams should not assume a model that shines in one language guards another.
Catalog record
| CREATOR | Syafiq Al Atiiq, Christian Gehrmann, Kevin Dahlén |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-15905
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Vulnerability Detection in Popular Programming Languages with Language Models Thanks: This work is partially supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation and by framework grant RIT17-0032 from the Swedish Foundation for Strategic Research. The computations and data handling were enabled by resources provided by the National Academic Infrastructure for Supercomputing in Sweden (NAISS), partially funded by the Swedish Research Council through grant agreement no. 2022-06725. Syafiq Al Atiiq, Christian Gehrmann, Kevin Dahlén Abstract Vulnerability detection is crucial for maintaining software security, and recent research has explored the use of Language Models (LMs) for this task. While LMs have shown promising results, their performance has been inconsistent across datasets, particularly when generalizing to unseen code. Moreover, most studies have focused on the C/C++ programming language, with limited attention given to other popular languages. This paper addresses this gap by investigating the effectiveness of LMs for vulnerability detection in JavaScript, Java, Python, PHP, and Go, in addition to C/C++ for comparison. We utilize the CVEFixes dataset to create a diverse collection of language-specific vulnerabilities and preprocess the data to ensure quality and integrity. We fine-tune and evaluate state-of-the-art LMs across the selected languages and find that the performance of vulnerability detection varies significantly. JavaScript exhibits the best performance, with considerably better and more practical detection capabilities compared to C/C++. We also examine the relationship between code complexity and detection performance across the six languages and find only a weak correlation between code complexity metrics and the models’ F1 scores. 1 Introduction Vulnerability detection is a critical component of software security, as undetected vulnerabilities can lead to severe consequences [ 24 ] . With the increasing complexity of software, manual detection becomes impractical, requiring the development of automated techniques 1 1 1 https://lcamtuf.coredump.cx/afl/ . In recent years, deep learning-based approaches have shown promise in vulnerability detection, particularly using language model (LM) [ 17 ] . LMs, such as BERT [ 10 ] , GPT [ 28 ] , and Sonnet [ 37 ] , have demonstrated outstanding performance in code understanding and generation tasks. These models can capture rich semantic and syntactic information from a large codebase, potentially well-suited for analyzing source code [ 1 ] . Several studies have explored LM’s application for vulnerability detection in C/C++ code [ 6 , 44 , 40 ] . However, the results have been mixed, with challenges like high false positive rates (FPR) and limited generalization to unseen code [ 7 , 11 , 2 ] . This raises a question about the effectiveness of LM for in real-world scenarios. While C/C++ is a widely used language, particularly in system-level software, there is a growing need to address the same problem in other programming languages 2 2 2 https://www.tiobe.com/tiobe-index/ . Languages such as JavaScript, Java, Python, PHP, and Go are extensively used in web development, enterprise applications, and data analysis 3 3 3 https://survey.stackoverflow.co/2024/ . Vulnerabilities in these languages may have significant implications, given their widespread adoption and the sensitive nature of the applications they support 4 4 4 https://owasp.org/www-project-top-ten/ . To bridge this gap, this paper investigates the effectiveness of LMs for vulnerability detection in popular programming languages. Our experimental evaluation covers JavaScript, Java, Python, PHP, and Go. We use the CVEFixes dataset [ 4 ] , containing a diverse set of vulnerabilities across multiple languages, and preprocess it to create language-specific subsets. These languages are chosen because they have the highest number of samples in the dataset. By fine-tuning and evaluating state-of-the-art (SOTA) language models on these subsets, we assess their performance in detecting vulnerabilities. The paper’s main contributions are the following: • We clean and adapt the large CVEFixes dataset to perform an empirical study on the vulnerability detection performance differences between JavaScript, Java, Python, PHP, Go, and C/C++. • We analyze and present correlation figures for the relationship between code complexity and vulnerability detection performance in the investigated dataset. • A public release of the curated dataset, scripts, and experimental results to support open science and replicate our findings: https://github.com/syafiq/llm_vd . The remainder of this paper is organized as follows. Section 2 presents the background and related work on vulnerability detection techniques and language models. Section 3 provides the problem definition of our work. Section 4 describes our dataset, models, and our fine-tuning setup. Section 5 presents the results and analysis of our experiments. Section 6 discusses the limitations of our work, followed by conclusions and the implications of our findings in section 7 . 2 Background and Related Work 2.1 Vulnerability Detection Techniques Over the years, various techniques have been developed to detect vulnerabilities in software systems, ranging from traditional approaches to more advanced deep learning-based methods [ 19 , 20 , 18 ] . Traditional Approaches Traditional vulnerability detection approaches include manual code review [ 30 ] , static analysis [ 38 ] , and dynamic analysis [ 3 ] . Manual code review is a labor-intensive process where security experts manually inspect the source code to identify vulnerabilities [ 5 ] . Static analysis techniques examine the source code without executing it to identify potential vulnerabilities [ 8 ] . Dynamic analysis involves executing the program with specific inputs and monitoring its behavior to detect runtime vulnerabilities. [ 32 ] . While these traditional approaches have effectively detected certain vulnerabilities, they have limitations. Manual code review is time-consuming [ 23 ] and relies heavily on the expertise of the reviewers. Static analysis techniques can suffer from high false positives [ 39 ] , while dynamic analysis may miss vulnerabilities not triggered by the specific inputs used during testing. Non-Language Model-based Deep Learning Approaches In recent years, deep learning-based approaches have gained attention for their ability to automatically learn features from large amounts of data and detect complex patterns. These approaches have been applied to various software engineering tasks, including vulnerability detection [ 6 , 17 , 44 ] . Several studies have explored using deep learning models, such as convolutional neural networks (CNNs) and graph neural networks (GNNs). VulDeePecker [ 17 ] uses CNNs to detect vulnerabilities in C/C++ code. Devign [ 44 ] combines GNNs with code property graphs to detect vulnerabilities in C/C++ programs. While the deep learning-based approaches have shown promising results, they often require extensive feature engineering and may struggle to capture the complex semantics and dependencies present in source code [ 43 ] . Traditional deep learning models typically rely on manual feature engineering to extract relevant information from the code, which can be time-consuming and require domain expertise. Moreover, these models may have difficulty understanding the intricate relationships between code elements, such as variable scoping, control flow, and data dependencies, and capturing the context and intent behind the code. Attention mechanisms used in transformer-based models have been proposed to address these limitations. These mechanisms can help capture long-range dependencies and global context. However, even with these advancements, deep learning models may still struggle to fully capture source code’s complex semantics and nuances, especially when dealing with diverse programming languages and coding styles. 2.2 Vulnerability Detection with Language Model The language model (LM) is a class of deep learning models trained