What Do They Fix? LLM-Aided Categorization of Security Patches for Critical Memory Bugs

Xingyu Li * , Juefei Pu * , Yifan Wu * , Xiaochen Zou * , Shitong Zhu * , Qiushi Wu § , Zheng Zhang * , Joshua Hsu * , Yue Dong * , Zhiyun Qian * , Kangjie Lu † , Trent Jaeger * , Michael De Lucia ‡ , Srikanth V. Krishnamurthy * · 2025

Read What Do They Fix? LLM-Aided Categorization of Security Patches for Critical Memory Bugs by Xingyu Li * , Juefei Pu * , Yifan Wu * , Xiaochen Zou * , Shitong Zhu * , Qiushi Wu § , Zheng Zhang * , Joshua Hsu * , Yue Dong * , Zhiyun Qian * , Kangjie Lu † , Trent Jaeger * , Michael De Lucia ‡ , Srikanth V. Krishnamurthy * (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-22796.txt.

Introduction

Xingyu Li and 13 co-authors sort Linux kernel security patches by the bug they fix, using language models. Downstream maintainers often delay patches, leaving windows of vulnerability. Critical fixes for out-of-bounds access and use-after-free bugs are hard to spot. Some fixes are deliberately silent, CVE records lag or go missing, and kernel CVE rules have shifted. Prior fine-grained classifiers lack coverage and accuracy.

Their pipeline, DUALLM, pairs a large language model with a fine-tuned small one. It reads commit titles, messages, diffs, and code context together. On recent kernel patches it reached high accuracy and found over a hundred candidate critical fixes. Manual checks confirmed most, and the team built proof-of-concept exploits for two bugs.

Catalog record

CREATORXingyu Li * , Juefei Pu * , Yifan Wu * , Xiaochen Zou * , Shitong Zhu * , Qiushi Wu § , Zheng Zhang * , Joshua Hsu * , Yue Dong * , Zhiyun Qian * , Kangjie Lu † , Trent Jaeger * , Michael De Lucia ‡ , Srikanth V. Krishnamurthy *
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE101 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-22796

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

What Do They Fix? LLM-Aided Categorization of Security Patches for Critical Memory Bugs Xingyu Li * , Juefei Pu * , Yifan Wu * , Xiaochen Zou * , Shitong Zhu * , Qiushi Wu § , Zheng Zhang * , Joshua Hsu * , Yue Dong * , Zhiyun Qian * , Kangjie Lu † , Trent Jaeger * , Michael De Lucia ‡ , Srikanth V. Krishnamurthy * Abstract Open-source software projects are foundational to modern software ecosystems, with the Linux kernel standing out as a critical exemplar due to its ubiquity and complexity. Although security patches are continuously integrated into the Linux mainline kernel, downstream maintainers often delay their adoption, creating windows of vulnerability. A key reason for this lag is the difficulty in identifying security-critical patches, particularly those addressing exploitable vulnerabilities such as out-of-bounds (OOB) accesses and use-after-free (UAF) bugs. This challenge is exacerbated by intentionally silent bug fixes, incomplete or missing CVE assignments, delays in CVE issuance, and recent changes to the CVE assignment criteria for the Linux kernel. Prior efforts such as GraphSPD, have proposed binary classifiers to distinguish security versus non-security patches. However, these approaches do not provide fine-grained categorization of vulnerability types, which is essential for prioritizing fixes for high-impact bugs like OOB and UAF. Our work aims to take such coarsely labeled security patches and classify them into fine-grained categories, i.e., OOB, UAF, or non-OOB-UAF types. While fine-grained patch classification approaches exist, they exhibit limitations in both coverage and accuracy. In this work, we identify previously unexplored opportunities to significantly improve fine-grained patch classification. Specifically, by leveraging cues from commit titles/messages and diffs alongside appropriate code context, we develop DualLM , a dual-method pipeline that integrates two approaches based on a Large Language Model (LLM) and a fine-tuned small language model. DualLM achieves 87.4% accuracy and an F1-score of 0.875 , significantly outperforming prior solutions. Notably, DualLM successfully identified 111 of 5,140 recent Linux kernel patches as addressing OOB or UAF vulnerabilities, with 90 true positives confirmed by manual verification (many do not have clear indications in patch descriptions). Moreover, we constructed proof-of-concepts for two identified bugs (one UAF and one OOB), including one developed to conduct a previously unknown control-flow hijack as further evidence of the correctness of the classification. \DeclareCaptionType example[Example][List of Examples] I Introduction Open-source software projects have emerged as cornerstones of modern software development ecosystems. However, incorporating such projects leads to significant security risks. Memory safety vulnerabilities are particularly notable due to their prevalence and severity. However, memory safety vulnerabilities vary significantly in severity . Among them, out-of-bounds (OOB) access and use-after-free (UAF) vulnerabilities (including double-free and invalid-free) are especially perilous, as malicious actors consistently exploit them to achieve privilege escalation and compromise system integrity [ 19 , 6 , 3 , 1 ] . In fact, our analysis of publicly available exploits [ 14 ] since 2020 showed that almost 90% of them exploit OOB or UAF vulnerabilities. Taking the Linux kernel as an example, it has been shown that even when patches addressing such vulnerabilities are promptly integrated into the upstream mainline kernel, downstream kernel maintainers (e.g., Ubuntu) may delay their adoption for weeks or even months [ 44 , 78 ] . This extended lag creates a substantial window of vulnerability in which attackers may exploit bugs to attack unpatched downstreams. Moreover, studies have demonstrated the feasibility of automated exploit generation targeting OOB and UAF bugs [ 27 , 71 , 66 , 73 ] , further underscoring the urgent need for timely patch porting. Prior work (e.g., [ 78 ] ) has documented this issue extensively, which identifies a major cause of this delay: downstream maintainers often lack clear, timely, and fine-grained information about which patches are security-critical, i.e., OOB or UAF bugs [ 78 , 85 ] . The volume of daily commits makes it infeasible to manually audit every patch. Furthermore, CVE-based indicators are insufficient: (1) many security patches are fixed silently without any indications (i.e., perhaps no one understood the security impact); (2) CVE assignments often lag behind the availability of patches by weeks or even months [ 41 ] ; (3) CVEs are incomplete — many exploitable security vulnerabilities are not assigned CVE numbers [ 28 , 85 , 45 , 84 ] . While several tools have been proposed to detect security-related patches (e.g., VulFixMiner [ 81 ] , GraphSPD [ 62 ] ), they generally offer only binary classification (security vs. non-security), which is insufficient for practical prioritization, as not all vulnerabilities are equally impactful and urgent to be ported. To enable downstream maintainers to triage and prioritize critical patches earlier, a more useful analysis involves a fine-grained classification that distinguishes between vulnerability types, particularly to tease out those with high exploitability such as UAF and OOB. Although several approaches have previously been proposed, all exhibit significant limitations in both coverage and accuracy. Limitations of Prior Art. Specifically, state-of-the-art solutions SID [ 70 ] , TreeVul [ 49 ] and CoLeFunDa [ 80 ] offer fine-grained patch classifications. SID [ 70 ] relies on human-defined, hard-coded patterns, but the complexity and variety of patterns in the real world make them difficult to define and accurately capture. Specifically, it supports only a single hard-coded patch pattern for each UAF and OOB vulnerability type, resulting in low coverage, and fails to identify 57% of the relevant patches. TreeVul [ 49 ] and CoLeFunda [ 80 ] eliminate the need for hard-coded patterns by feeding the code diffs (e.g., added and removed lines of code) to a machine-learning model. Unfortunately, TreeVul does not use any code context beyond the diff itself, limiting the model’s ability to appropriately learn patch patterns. While ColeFunda does use code context, its use of the standard slicing technique can introduce bloated code blocks and noise. Furthermore, both approaches suffer from a common limitation: they do not leverage commit messages, missing valuable semantic cues in natural language that can aid classification. 1 1 1 CoLeFunDa has three downstream tasks, what we refer for it in the paper is its CWE classification task. Also, CoLeFunDa is not open-sourced. Our evaluations show that (see § VI ), TreeVul achieves an accuracy of 65.6 % and an F1-score of 0.653, in identifying patches that fix UAF and OOB bugs, indicating ample room for improvement. These limitations motivate the need for a more effective solution that advances the state of the art in vulnerability type classification. Our solution. To tackle the challenge of handling diverse patch patterns, we propose a novel machine-learning-based solution guided by two key insights: First, beyond code diffs in a patch, the commit titles and messages often provide valuable insights into the nature of the bug being fixed (e.g., directly mentioning a “use-after-free” bug being fixed). The advancements in LLMs make them very suitable for extracting such indicators from commit descriptions. Different from CoLeFunDa [ 80 ] and TreeVul [ 49 ] , our approach leverages Large Language Models (LLMs) to effectively extract and utilize these valuable indicators from commit descriptions. While an LLM-only pipeline already outperforms recent approaches such as SID and TreeVul, its accuracy still falls short of our expectations. Second, critical clues are

Read the complete text