Code-Centric Detection of Vulnerability-Fixing Commits: A Unified Benchmark and Empirical Study
Nils Loose, Joseph Bienhüls, Kristoffer Hempel, Felix Mächtle, Thomas Eisenbarth · 2026
Read Code-Centric Detection of Vulnerability-Fixing Commits: A Unified Benchmark and Empirical Study by Nils Loose, Joseph Bienhüls, Kristoffer Hempel, Felix Mächtle, Thomas Eisenbarth (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-13138.txt.
Introduction
Loose and four co-authors rigorously re-evaluate code language models for detecting vulnerability-fixing commits, which advisory databases often record late or never. Consolidating 20 fragmented datasets spanning over 180,000 commits and training 270 models, they isolate the code-only signal under group-stratified evaluation. Model size helps but not enough, and extra code context gives no reliable gain under strict false-positive budgets. At a 0.5% false-positive rate every fine-tuned code-only model misses at least 80% of fixes. A manual audit finds label error concentrated in commits lacking CVE links.
Catalog record
| CREATOR | Nils Loose, Joseph Bienhüls, Kristoffer Hempel, Felix Mächtle, Thomas Eisenbarth |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56631 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.13138
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Code-Centric Detection of Vulnerability-Fixing Commits: A Unified Benchmark and Empirical Study Nils Loose, Joseph Bienhüls, Kristoffer Hempel, Felix Mächtle, Thomas Eisenbarth Abstract . Automated detection of vulnerability-fixing commits (VFCs) is critical for timely security patch deployment, as advisory databases lag patch releases by a median of 25 days and many fixes never receive advisories. We present a comprehensive evaluation of code language model based VFC detection through a unified framework consolidating over 20 fragmented datasets spanning more than 180 000 180\,000 commits. Across over 180 180 experiments with fine-tuned models from 125 125 M to 14 14 B parameters, we find no evidence that models acquire transferable security-relevant code understanding from code changes alone. When commit messages are available, they dominate model attention, and when removed, an attribution analysis shows that enriching diffs with additional intra-procedural semantic context does not shift model attention toward the code changes. Group-stratified evaluation exposes approximately 17 17 % performance drops compared to random splits, while temporal splits on aggregated datasets prove unreliable due to compositional shift in the underlying project distributions. At a false positive rate of 0.5 0.5 % all fine-tuned code-only models miss over 93 93 % of vulnerabilities. Larger and more diverse training data or generative approaches show preliminary improvements but do not resolve the underlying limitations. To support future research on code-centric VFC detection, we release our unified framework and evaluation suite. 1. Introduction Figure 1 . Temporal overview of existing VFC datasets, their label source, size and dependent datasets. The advisory label class includes other, similar, sources of information such as bug trackers. The synthetic label indicates that at least a subset of samples is synthetic. Manual verification indicates that some effort was spent validating at least a subset of the samples. VulnPP: VulnPatchPairs. Todo Modern software supply chains create cascading security dependencies where vulnerabilities impact numerous downstream projects. Organizations must identify and apply patches before attackers exploit the disclosure window. However, Imtiaz et al. 2023 reveal that advisory publication lags patch release by a median of 25 days, leaving systems vulnerable to attacks when developers rely on vulnerability databases such as the National Vulnerability Database (NVD) ( NIST [n. d.] ) . Silent security patches present an even greater challenge, as they never receive public advisories, leaving downstream software vulnerable. As the vast majority of reused software is maintained in open-source repositories, an accurate analysis of each commit across upstream dependencies could result in safer software ecosystems with faster patch deployment times. However, as manual analysis is infeasible at scale, automated techniques for identifying vulnerability-fixing commits (VFCs) have become essential. More broadly, as code changes in diff format have become the standard unit of review in modern development, from manual code review to AI-assisted programming, understanding the reasoning capabilities of current models on software changes is relevant well beyond the security domain. Code LMs ( Feng et al. 2020 ; Guo et al. 2022 ; Wang et al. 2021c ) have shown promise across code understanding tasks, leading to their adoption for Security Patch Detection ( SPD ), the task of identifying commits that fix vulnerabilities. While existing approaches report strong results when combining code with commit messages ( Tang et al. 2023 ; Wang et al. 2021b ; Tang et al. 2025 ; Sabetta and Bezzi 2018 ; Nguyen-Truong et al. 2022 ; Chen et al. 2024 ; Sun et al. 2023b ) , the actual capabilities of these models for understanding security-relevant code changes without relying on textual information remain largely unclear. Chen et al. 2024 and Sun et al. 2023b performed ablation studies evaluating their systems with and without the commit message and observed substantial drops in performance, suggesting a strong reliance on textual signals. Wang et al. 2023 proposed the first graph-based approach that relies exclusively on code information, but requires expensive precomputations with limited demonstrated success ( Wang et al. 2023 ; Wen et al. 2024 ) . For the related task of function-level vulnerability detection, recent studies have shown that models exploit superficial dataset correlations rather than learning vulnerability semantics, with simple word-count classifiers matching deep learning models on standard benchmarks ( Risse and Böhme 2024 ; Risse et al. 2025 ) . To enable a systematic and large-scale evaluation of code LM performance for SPD we first address the data fragmentation challenge. Over the past decade, a variety of research groups have collected and labeled VFCs with different target languages, labelling techniques and ground truths, resulting in fragmented datasets that hinder large-scale comparative evaluation. We build a comprehensive framework that unifies access to 20 VFC datasets, enabling systematic evaluation across different labeling strategies, programming languages, and vulnerability classes. To evaluate whether additional semantic context can improve model behavior, we additionally implement a lightweight intra-procedural context enrichment method that identifies semantically relevant code context around modifications. Using this framework, we conduct the first comprehensive evaluation of SPD using code LMs, spanning over 180 experiments across models from 125 M to 14 B parameters, multiple architectures, input representations, and training strategies. Among the fine-tuned models and representations tested, we find no evidence that models acquire transferable security-relevant code understanding. When commit messages are available, they dominate model attention, confirming prior observations at larger scale. When messages are removed, an attribution analysis shows that the actual code changes consistently receive modest attention. Under group-stratified evaluation that separates repositories between training and test sets, performance drops by approximately 17 % compared to random splits, confirming project-specific memorization consistent with findings by Steenhoek et al. 2023 for cross-project vulnerability detection. Temporal splits show an even larger degradation but prove unreliable on aggregated datasets, as a sliding window analysis reveals that performance variation is dominated by shifting project compositions rather than genuine temporal generalization. Neither scaling fine-tuned model capacity from 125 M to 14 B parameters nor enriching diffs with intra-procedural data-flow context changes this picture. Prompt-based generative classification without task-specific training achieves modest improvements over fine-tuned models on the same CVE-confirmed test set, though data contamination from internet-scale pretraining cannot be excluded. At a false positive rate of 0.5 %, all fine-tuned code-only models miss over 93 % of vulnerabilities (PD-S = FNR @ FPR \leq 0.005 ), indicating that none can reliably detect vulnerability-fixing commits under strict false-positive constraints. In summary, we make the following contributions. First, we develop a unified framework consolidating and preprocessing 20 fragmented VFC datasets, enabling systematic comparative evaluation across different labeling strategies and programming languages ( VFCDetective Artifact 2026 ) . 1 1 1 https://github.com/UzL-ITS/vfc_datasets Second, through a comprehensive evaluation we find that fine-tuned code LMs up to 14 B parameters do not acquire transferable security-relevant code understanding under rigorous evaluation for SPD , producing concrete insights on evaluation methodology for aggregated VFC datasets and on the limitations of current approaches. 2. Related Work 2.1. VFC Datasets Security research focusing on SPD or vulnerability detection ( VD ) requires code samples that are annotated with accurate vulnerability information. To approximate the underlying structural distribution of the data, a sufficiently large sample size is required. Yet, obtaining positive ground truth samples is difficult. Over the past decade, an increasingly popular method of obtaining security-annotated samples revolves around VFCs. To obtain code