R+R: Security Vulnerability Dataset Quality Is Critical
Anurag Swarnim Yadav, Joseph N. Wilson · 2025
Read R+R: Security Vulnerability Dataset Quality Is Critical by Anurag Swarnim Yadav, Joseph N. Wilson (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-06387.txt.
Introduction
Word tools that hunt code flaws learn from flaw sets, and those sets are oft rot: twin lines, wrong tags, and half samples. Anurag Swarnim Yadav and Joseph Wilson bare that rot in a 2025 study of sets used at head meets. One top fix tool's fame grade fell to a scrap when twins were struck from its school and test sets.
Tags in the top ten flaw kinds were wrong in more than half the checks. Half the samples were half built. Bare one in three was both true and whole. Schooled on a big clean fix stock, the same tools did well. Set worth, the pair say, is the whole game.
Catalog record
| CREATOR | Anurag Swarnim Yadav, Joseph N. Wilson |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-06387
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
R+R: Security Vulnerability Dataset Quality Is Critical Anurag Swarnim Yadav, Joseph N. Wilson Abstract Large Language Models (LLMs) are of great interest in vulnerability detection and repair. The effectiveness of these models hinges on the quality of the datasets used for both training and evaluation. Our investigation reveals that a number of studies featured in prominent software engineering conferences have employed datasets that are plagued by high duplication rates, questionable label accuracy, and incomplete samples. Using these datasets for experimentation will yield incorrect results that are significantly different from actual expected behavior. For example, the state-of-the-art VulRepair Model, which is reported to have 44% accuracy, on average yielded 9% accuracy when test-set duplicates were removed from its training set and 13% accuracy when training-set duplicates were removed from its test set. In an effort to tackle these data quality concerns, we have retrained models from several papers without duplicates and conducted an accuracy assessment of labels for the top ten most hazardous Common Weakness Enumerations (CWEs). Our findings indicate that 56% of the samples had incorrect labels and 44% comprised incomplete samples—only 31% were both accurate and complete. Finally, we employ transfer learning using a large deduplicated bug-fix corpus to show that these models can exhibit better performance if given larger amounts of high-quality pre-training data, leading us to conclude that while previous studies have over-estimated performance due to poor dataset quality, this does not demonstrate that better performance is not possible. I Introduction There are many recent publications presenting systems that use neural models to detect and repair security vulnerabilities in programs [ 1 , 2 , 3 , 4 , 5 , 6 ] . In the past few months a number of reports have surfaced concerning the quality of the datasets employed to train and test these systems in an effort to characterize their performance [ 6 , 7 ] . We join these other works by reporting here on the many data quality problems embodied in what we will call the VulRepair dataset , first employed in the work of Fu et al. [ 2 ] This dataset is a combination of Big-Vul [ 8 ] and CVEfixes [ 9 ] . We first encountered the data quality problems in the VulRepair dataset while attempting to understand the performance of several systems that employed it. At first, we took the results at face value. However, when replicating their behavior and looking at the specific repairs being made by the systems, it became clear that they exhibited effects associated with data leakage–the model was able to make repairs to vulnerable code that required it to have information available only in the repaired code. Once we identified the existence of numerous duplicate samples in the data collection, we realized there might be other data quality issues as well. Some of the problems in BigVul and CVEfixes datasets are reported by Ding et al. [ 7 ] in the context of vulnerability detection, which deals primarily with the issue of duplicate samples that lead to data leakage. This paper, on the other hand, reports initial results of our comprehensive review of the quality of the VulRepair data set and its effect on expected model performance. A number of papers have been written employing the VulRepair dataset. The first is the report describing VulRepair in November 2022 [ 2 ] , which introduced this dataset’s use in fine-tuning a CodeT5-based system. This same dataset was employed by Zhang et al. in an August 2023 paper evaluating CodeT5, CodeBERT, GraphCodeBERT, UniXcoder, and CodeGPT models trained on the VulRepair dataset [ 3 ] . Huang et al. employed the VulRepair dataset in a September 2023 paper benchmarking security-related tasks with CodeT5, CodeBERT, GraphCodeBERT, UniXcoder, and PLBART [ 4 ] . Fu et al. reported further work leveraging VRepair’s pre-training bug-fix dataset in conjunction with the VQM vulnerability fine-tuning dataset for program repair tasks. That report asserts that VRepair [ 1 ] was initially pre-trained on a dataset having 23,607 C/C++ functions, however, this claim appears to be incorrect as VRepair was pre-trained on a bug-fix dataset that has 534,848 training samples and 10,000 validation samples (see Chen et al. [ 1 ] last sentence of Section 4.21). We conducted a brief analysis of both the pre-training bug-fix dataset and VQM vulnerability fine-tuning dataset [ 10 ] used in that paper. The pre-training dataset contains 21,246 training samples and 2,362 validation samples. Our review revealed 18,622 duplicated entries in the training set and 782 duplicates in the validation set. After removing those, 1,579 cross-set duplicates (in both train and validation) were identified, that is to say, all of the validation set samples were present in the training set. Additionally, our analysis uncovered a substantial overlap between the bug-fix dataset and the VQM vulnerability fine-tuning dataset. Specifically, there were 511 matching entries in the test set, 243 in the validation set, and 1,747 in the training set of the vulnerability dataset that overlapped with the bug-fix dataset. The detailed analysis is discussed in Section VI . These findings make it clear that retraining is necessary in order to understand the expected performance of the models reported in that work. In this paper, we analyze the effect some of the data quality problems in the VulRepair dataset have had on skewing the expected performance of VulRepair, CodeBERT, GraphCodeBERT, and UniXcoder when applied to repair of security vulnerabilities. In addition we consider the impact of pre-training models with the deduplicated VRepair bug-fix dataset, addressing the duplicate entries present in the original VRepair pre-training dataset as discussed in Section III-A3 . Our analysis incorporates the following steps: 1. Replicate the results of previous papers using a more robust approach involving finding the mean performance of multiple models fine-tuned using different seeds. 2. Determine the impact of duplicate samples on the performance of these models. 3. Determine the performance impact of inconsistent labeling. 4. Partially identify and analyze the inaccurate labeling. 5. Partially identify and analyze the incomplete data samples. 6. Determine the impact of pre-training with the deduplicated VRepair bug-fix dataset. These data-quality-driven analyses are addressed by the following research questions: • RQ1 : Can we replicate the results of VulRepair, CodeBERT, GraphCodeBERT, and UniXcoder? • RQ2A : What is the performance of the models after removing all duplicate data from the VulRepair dataset (removing cross-set duplicates from the training dataset)? • RQ2B : What is the performance of the models after removing all duplicates from the VulRepair dataset (removing cross-set duplicates from the test dataset)? • RQ3A : What is the performance of the models after removing duplicates and inconsistently labeled samples from the VulRepair dataset (removing cross-set duplicates and inconsistent samples from the training dataset)? • RQ3B : What is the performance of the models after removing duplicates and inconsistently labeled samples from the VulRepair dataset (removing cross-set duplicates and inconsistent samples from the test dataset)? • RQ4A : What effect does removing duplicate data from VulRepair have on the repair of vulnerabilities included in the top ten CWEs. • RQ4B : How does identifying and analyzing inaccurate and incomplete samples of the top ten CWEs in the test set of the deduplicated VulRepair dataset