A Multi-Dataset Evaluation of Models for Automated Vulnerability Repair
Zanis Ali Khan, Aayush Garg, Qiang Tang · 2025
Read A Multi-Dataset Evaluation of Models for Automated Vulnerability Repair by Zanis Ali Khan, Aayush Garg, Qiang Tang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-04987.txt.
Introduction
Zanis Ali Khan and co-authors benchmark two code models, CodeBERT and CodeT5, on automatic repair of security flaws. Fixing general bugs with AI has advanced fast. Patching vulnerabilities has lagged behind. The team tests both models across six datasets in four programming languages, on flaws they trained on and flaws never seen.
Both models stumble when context comes in fragments. CodeBERT coped better with sparse context. CodeT5 read complex flaw patterns better and scaled further. Fine-tuning lifted scores on familiar data. It did not carry to unseen datasets. The authors map these limits and offer guidance for building repair tools that hold up on real-world code.
Catalog record
| CREATOR | Zanis Ali Khan, Aayush Garg, Qiang Tang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-04987
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Multi-Dataset Evaluation of Models for Automated Vulnerability Repair Zanis Ali Khan, Aayush Garg, Qiang Tang Abstract Software vulnerabilities pose significant security threats, requiring effective mitigation. While Automated Program Repair (APR) has advanced in fixing general bugs, vulnerability patching—a security-critical aspect of APR—remains underexplored. This study investigates pre-trained language models, CodeBERT and CodeT5, for automated vulnerability patching across six datasets and four languages. We evaluate their accuracy and generalization to unknown vulnerabilities. Results show that while both models face challenges with fragmented or sparse context, CodeBERT performs comparatively better in such scenarios, whereas CodeT5 excels in capturing complex vulnerability patterns. CodeT5 also demonstrates superior scalability. Furthermore, we test fine-tuned models on both in-distribution (trained) and out-of-distribution (unseen) datasets. While fine-tuning improves in-distribution performance, models struggle to generalize to unseen data, highlighting challenges in robust vulnerability detection. This study benchmarks model performance, identifies limitations in generalization, and provides actionable insights to advance automated vulnerability patching for real-world security applications. 1 Introduction Software vulnerabilities remain a constant threat to contemporary software systems, leaving them susceptible to exploitation by malicious actors. These vulnerabilities, which include problems like injection flaws and memory management errors, can result in unauthorized access, data breaches, and service interruptions [ 34 ] . Addressing these issues is essential to ensure the reliability and security of software systems [ 1 ] . However, the manual effort required to detect and fix these vulnerabilities is time-consuming, prone to errors, and struggles to match the growing complexity and scale of today’s software ecosystems [ 23 ] . Automated Program Repair (APR) has gained traction as a promising approach to tackle this issue, employing computational methods to autonomously generate fixes for software bugs [ 3 ] . While APR has achieved notable progress in addressing general software defects, the specialized area of vulnerability-focused program repair—which deals with the unique challenges of security vulnerabilities—remains underdeveloped. Unlike general-purpose bug fixes, patches for vulnerabilities often require addressing the flaw in a way that is not only functionally correct but also generalizable across variations of the vulnerability [ 9 ] . This makes vulnerability patching a more nuanced subset of Automated Program Repair (APR), where the ability to generate broadly applicable fixes becomes especially important. Existing techniques in vulnerability-focused Automated Program Repair (APR) predominantly depend on either static analysis tools or traditional machine learning models trained on specific vulnerability patterns. Although these approaches have demonstrated potential in identifying vulnerabilities, their capacity to generate meaningful and effective patches remains limited. For instance, static analysis tools are highly effective at detecting vulnerabilities but often struggle to produce practical fixes [ 36 ] . Similarly, conventional machine learning models are hindered by their dependence on restricted datasets [ 14 ] , which limits their generalizability and effectiveness across a wide range of programming languages and vulnerability types [ 38 ] . Recent advancements in deep-learning have paved the way for automated vulnerability patching, particularly with the emergence of pre-trained language models tailored for code. Models like CodeBERT [ 12 ] and CodeT5 [ 39 ] utilize large-scale code corpora to capture both syntactic and semantic structures, facilitating tasks such as code generation, summarization, and translation [ 17 ] . Their ability to discern patterns from extensive datasets makes them a promising tool for vulnerability-focused program repair. However, the practical application of these models remains challenging. Due to substantial differences in syntax, semantics, and vulnerability characteristics across programming languages, existing pre-trained models, which are often designed for monolingual or domain-specific tasks, may struggle with generalization [ 8 ] . Evaluating their performance across diverse languages is therefore a crucial yet underexplored area of research [ 19 ] . This paper systematically evaluates the performance of pre-trained language models in vulnerability-focused program repair, specifically analyzing CodeBERT and CodeT5 in generating patches for known vulnerabilities across six datasets covering four programming languages. We assess their effectiveness using CodeBLEU and CrystalBLEU scores and explore their generalizability by evaluating performance on both in-distribution and out-of-distribution datasets, providing insights into their strengths and limitations. Our results show that while both models excel in generating vulnerability patches, they exhibit distinct limitations. CodeT5 generally outperforms CodeBERT in accuracy, especially on datasets with complex vulnerability patterns. However, both models struggle with fragmented contexts and sparse data, which limits their ability to produce correct fixes in such settings. Additionally, while fine-tuning improves performance on in-distribution datasets, both models face challenges in generalizing to out-of-distribution datasets, highlighting limitations in detecting and patching vulnerabilities in unseen scenarios. Hence, our contributions in this paper are threefold: • We provide an evaluation of CodeBERT and CodeT5 for vulnerability-focused program repair, covering a diverse set of 6 datasets across multiple programming languages. • We establish benchmarks for model performance in generating vulnerability patches, serving as a foundation for evaluating pre-trained models in dataset-driven vulnerability patching scenarios. • We identify key limitations in model generalization, particularly the challenges of fine-tuning and performance on out-of-distribution datasets. 2 Related Work Software vulnerabilities refer to security gaps or defects within code that can be leveraged by malicious actors to compromise systems [ 33 ] . One notable example is the buffer overflow vulnerability, which arises when a program tries to write more data into a buffer than it can hold, leading to the overflow spilling into neighboring memory areas. This can allow attackers to inject and execute harmful code [ 18 ] . As these vulnerabilities grow more complex, they pose substantial obstacles to developing and deploying robust countermeasures. While vulnerability detection has been extensively studied, significantly less attention has been given to generating patches. Traditional static analysis tools have long been used for detection, but their reliance on predefined rules often makes it difficult to identify complex patterns [ 2 ] . In contrast, AI-driven methods have gained traction for their ability to process vast codebases and uncover intricate security flaws. Models like CodeBERT [ 13 ] and GraphCodeBERT [ 21 ] have proven effective in analyzing source code, contributing to advancements in vulnerability detection and assessment [ 16 ] . Additionally, large language models (LLMs) such as OpenAI’s GPT-4, Meta AI’s Llama2, and Mistral AI’s Mistral have demonstrated strong adaptability in tackling vulnerability detection tasks [ 22 ] . Conversely, creating effective patches continues to be a significant challenge. The majority of research on automated patch generation is centered on fixing general code defects rather than targeting vulnerabilities directly. The subsequent sections will explore methodologies within this broader context. 2.1 Traditional Approaches to Code Repair Automated code repair traditionally falls into two categories: heuristic-based and constraint-based [ 20 ] . Heuristic methods search for patches that pass all tests, often using transformation schemas for efficiency [ 29 ] . Approaches like GenProg [ 28 ] and PAR [ 27 ] leverage genetic programming, while others use random or deterministic strategies to refine the search. Constraint-based methods employ symbolic execution [ 5 ] to guide patch generation by exploring multiple execution paths. Tools such as SemFix [ 31 ] and Angelix [ 30 ] derive repair constraints, while techniques like Nopol [ 41 ] target specific cases, such as repairing conditional expressions. 2.2 ML-Based Code Repair Machine learning has emerged as a key technique for automating code repair,